View Single Post
Old 02-11-2004, 11:28 PM   #3
simonced
Green Mole
 
Join Date: Feb 2004
Location: france
Posts: 6
Thanks for your so quick reply

I see, it's a good way.
So, I think my file that lists the pdf can be a php script ?
So, it's easy in fact.

How can I crawl only my pdf listing file in shell command ?
$ php [...]/spider.php http://website/fold1/...../foldx/listingfiles.php
(I don't know how to set the level at one this way...)
May I put a robots.txt in the foldx ?

Where can I get a robots configuration help ?

Thank you very much.
simonced is offline   Reply With Quote