Search for "ArchiveBot: The Archive Team Crowdsourced Crawler" on the Internet Archive. archive.org/details/arch...
ArchiveBot: The Archive Team Crowdsourced Crawler
ArchiveBot is an IRC bot designed to automate the archival of smaller websites (e.g. up to a few hundred thousand URLs). You give it a URL to start at, and it grabs all content under that URL, records...
archive.org