PDA

View Full Version : robots file


xibalba
03-24-2004, 11:14 AM
Spidering in progress...
-----------------------------
SITE : http://example.com/
Exclude paths :
- directory\.bml
Meta Robots = NoIndex, or already indexed : No content indexed
1:http://example.com/(time : 00:00:51)
No link in temporary table
links found : 0
...Was recently indexed
Optimizing tables...
Indexing complete !


how can I force phpdig to ignore the Robots file and index this site anyways?:bang:

Charter
03-25-2004, 12:28 AM
Hi. If the site is yours, then remove the robots.txt file and any HTML no index tags from the site files. Otherwise, it looks like the site owner doesn't want the content indexed. I will not provide info to purposely go against a site owner's wishes, and others should not either.