PhpDig.net

PhpDig.net (http://www.phpdig.net/forum/index.php)
-   Troubleshooting (http://www.phpdig.net/forum/forumdisplay.php?f=22)
-   -   robots file (http://www.phpdig.net/forum/showthread.php?t=718)

xibalba 03-24-2004 10:14 AM

robots file
 
Spidering in progress...
-----------------------------
SITE : http://example.com/
Exclude paths :
- directory\.bml
Meta Robots = NoIndex, or already indexed : No content indexed
1:http://example.com/(time : 00:00:51)
No link in temporary table
links found : 0
...Was recently indexed
Optimizing tables...
Indexing complete !


how can I force phpdig to ignore the Robots file and index this site anyways?:bang:

Charter 03-24-2004 11:28 PM

Hi. If the site is yours, then remove the robots.txt file and any HTML no index tags from the site files. Otherwise, it looks like the site owner doesn't want the content indexed. I will not provide info to purposely go against a site owner's wishes, and others should not either.


All times are GMT -8. The time now is 11:20 PM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.