".... I really want to start testing and developing methods for scaling and speeding a project of mine, and also just for general knowledge. I can't think of a way to do this without creating a bunch of amazon ec2 instances, or buying more computers and doing this from house. ..."
Think distributed.
Create a useful installable tool that utilises spare cpu cycles that are being under-utilised. Or get some experience with some of the existing systems. Some examples that I can think of are
Another approach is to take a look at seeing how you can create a Beowulf cluster ~ http://www.beowulf.org/showcase/index.html
You don't have to build it (though operacy is worth 10x reading) you can look at the software. If you are still at school or know somebody there you can see if anyone is working on parallel software.
[0] Take a read of this blog on the development of a personalised RSS crawler ~ http://blog.persai.com/ to see the kinds of problems you have to overcome (scaling, data integrety, storage, re-writes)
Go work for google? [0]
".... I really want to start testing and developing methods for scaling and speeding a project of mine, and also just for general knowledge. I can't think of a way to do this without creating a bunch of amazon ec2 instances, or buying more computers and doing this from house. ..."
Think distributed.
Create a useful installable tool that utilises spare cpu cycles that are being under-utilised. Or get some experience with some of the existing systems. Some examples that I can think of are
- SETI ~ http://setiathome.berkeley.edu/
- Electric Sheep ~ http://electricsheep.org/
- PlanetQuest ~http://www.wired.com/science/space/news/2005/03/66757
Another approach is to take a look at seeing how you can create a Beowulf cluster ~ http://www.beowulf.org/showcase/index.html You don't have to build it (though operacy is worth 10x reading) you can look at the software. If you are still at school or know somebody there you can see if anyone is working on parallel software.
[0] Take a read of this blog on the development of a personalised RSS crawler ~ http://blog.persai.com/ to see the kinds of problems you have to overcome (scaling, data integrety, storage, re-writes)