looking for web spider type app

[raelmadrid]

I'm looking for a program that I can give a list of URLs to then want it to scan them all and give me a list of URLs that appear on those pages.

Does anyone know of an app that can do this?

[beerman]

I too would be interested in such an app. It would certainly help with something Im working on at the moment.

[ap0c]

Look up "link scraping" i found one here:

http://www.paulmaden.com/tools/scrapey/php-link-scraper

there are quite a few tutorials out ther on how to build them on your preferred language.

[the.insane]

I used to use websnake many years ago