The Uncharted Internet
Doing research on a school I stumbled on an older part of their website that seemed to contradict their current school message (sorry for the lack of details, but I dont want to offend the particular school). Suffice it to say it got me thinking about another way scan the internet.
The traditional way is to follow links. Start at a site, and crawl your way outwards. It's efficient and charts how most people explore the web. However, what if you moved from website to website like a password cracker, trying reasonable combinations of directory name or html file names. It would take a while and the bots may get banned from a few website, but it would give you a picture of the unlinked Internet. I would be really excited to see the results. What do people keep hidden? What seedy back alleys exist that no outsiders are meant to see?
The traditional way is to follow links. Start at a site, and crawl your way outwards. It's efficient and charts how most people explore the web. However, what if you moved from website to website like a password cracker, trying reasonable combinations of directory name or html file names. It would take a while and the bots may get banned from a few website, but it would give you a picture of the unlinked Internet. I would be really excited to see the results. What do people keep hidden? What seedy back alleys exist that no outsiders are meant to see?