PhpDig.net - Need Solution Please

Page 2 of 2

Show 40 post(s) from this thread on one page

PhpDig.net (http://www.phpdig.net/forum/index.php)

- Troubleshooting (http://www.phpdig.net/forum/forumdisplay.php?f=22)

- - Need Solution Please (http://www.phpdig.net/forum/showthread.php?t=332)

rwh	12-24-2003 04:42 PM

First thanks for all your help.
Real Web Host I can remove that because I have files and directories that can not be crawled.

On the other it is crawling now, but even though it has a redirect in it there are still directories in there for that domain.
It is not looking at them at all still. it just jumped over that domain and went to the others.
So maybe just have to do those sub directories manually like I did before I guess.

Charter

12-24-2003 05:32 PM

Hi. Add the following to the top of the robots.txt file and then make the code change listed in this thread.

Code:

User-agent: PhpDig

Disallow:

# whatever else below this

This should let PhpDig follow all the links it finds, and then you can go and delete/exclude certain links/directories from the admin panel.

rwh	12-24-2003 05:57 PM

Any way around the other problem with that domain not reading because it gets redirected

Charter

12-24-2003 06:00 PM

Hi. Post fifteen on the first page of this thread should deal with the redirect.

rwh	12-24-2003 06:06 PM

Ok made that change and I put in the main domain it looks like this notice it does not even try to get sub directories under main domain it does not get anything then goes to number 2 which is the redirect domain name so it gets nothing from main domain name.

SITE : http://www.mansfield-tx.gov/
Exclude paths :
- @NONE@
1:http://www.mansfield-tx.gov/
(time : 00:00:00)
Ok for http://www.ci.mansfield.tx.us/ (site_id:49)

No link in temporary table

--------------------------------------------------------------------------------

links found : 1
http://www.mansfield-tx.gov/

--------------------------------------------------------------------------------
SITE : http://www.ci.mansfield.tx.us/
Exclude paths :
- @NONE@
2:http://www.ci.mansfield.tx.us/

it is still running as we speak over 50 minutes now and on number 41.

Charter

12-24-2003 06:20 PM

Hi. PhpDig can't index subdirectories/files if there are no links to such. The only thing PhpDig sees at http://www.mansfield-tx.gov/ is the below so, with the changes made in this thread, the only place PhpDig can go to is http://www.ci.mansfield.tx.us and then follow the links from there.

Code:

<html>

<head>

<meta http-equiv="refresh" content="0;url=http://www.ci.mansfield.tx.us"> 

</head>

</html>

You could setup a temp page with links to the subdirectories/files that you want indexed, and after the index is done, then go to the admin panel, click the site holding the temp page, click the update button, click a blue arrow if needed, and then delete the temp page.

rwh	12-24-2003 07:04 PM

Ok understand that

rwh	12-24-2003 07:26 PM

Just to follow up. I made the temp index.html file itworks getting pages now, for some reason when it got done with domain name it got the same pages using the ip.

Charter

12-26-2003 12:04 PM

Hi. Are you crawling shell or from the browser interface, with FTP on or off? Is there a link somewhere that uses the IP instead of the domain name?

rwh	12-26-2003 12:16 PM

From IE Browser and FTP ON.
Figure there are links with ip in his files, not sure well let him look at them.

All times are GMT -8. The time now is 10:25 AM.

Page 2 of 2

Show 40 post(s) from this thread on one page