|
Download -
PHP Reference -
Links -
Demo 1.8.9 RC1 -
API
License - Credits - Copying - Changelog - Documentation |
|
|
03-24-2004, 10:14 AM
|
#1 |
|
Green Mole
Join Date: Mar 2004
Posts: 9
|
robots file
Spidering in progress...
----------------------------- SITE : http://example.com/ Exclude paths : - directory\.bml Meta Robots = NoIndex, or already indexed : No content indexed 1:http://example.com/(time : 00:00:51) No link in temporary table links found : 0 ...Was recently indexed Optimizing tables... Indexing complete ! how can I force phpdig to ignore the Robots file and index this site anyways? 
|
|
|
|
03-24-2004, 11:28 PM
|
#2 |
|
Head Mole
Join Date: May 2003
Posts: 2,539
|
Hi. If the site is yours, then remove the robots.txt file and any HTML no index tags from the site files. Otherwise, it looks like the site owner doesn't want the content indexed. I will not provide info to purposely go against a site owner's wishes, and others should not either.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
|
|
|
 |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| robots.txt seems to be ignored :? | galacticvoyager | Bug Tracker | 1 | 11-12-2005 12:52 PM |
| robots.txt and URL | djavet | How-to Forum | 4 | 01-11-2005 03:19 AM |
| robots.txt comments | edkay | Mod Submissions | 2 | 03-12-2004 12:41 PM |
| robots.txt ignored | roy | Troubleshooting | 3 | 02-20-2004 08:02 PM |
| robots.txt | renehaentjens | Troubleshooting | 3 | 12-05-2003 02:40 PM |
