If you guys use Bash and Terminal, you can try a little Bash Script I made. It does not randomly index URLs, it visits each URL from a URL list "de novo" and greps their web content in a while loop…
If you want to check:
GitHub link: https://github.com/mountaineerbr/scripts/blob/master/urlgrep.sh
Cheers!