Webcrawler
The first version of a webcrawler, adapted from here.
Save function is currently broken, but it seems to work otherwise.
Needs a lot of cleanup.
While a webcrawler is conceptually very simple, the complexity of the internet makes it so we have a lot of choices in how we actually perform the crawl. Saw some things about selection policy on wikipedia.
Edited by Berg,N. van den (Nathan)