If you guys use Bash and Terminal, you can try a little Bash Script I made. It does not randomly index URLs, it visits each URL from a URL list "de novo" and greps their web content in a while loop… If you want to check: GitHub link: https://github.com/mountaineerbr/scripts/blob/master/urlgrep.sh Cheers!
