PDA

View Full Version : Why Can't I dig this site????


lighthouse
03-11-2004, 07:50 AM
SUrely an easy answer to this... but I can't find it!!!

I'm trying to crawl a website built entirely in asp, but it wont spider past the fron page.

Please help!

Thanks
Gary

lighthouse
03-11-2004, 02:26 PM
http://www.architecturecentre.net


digging results in:

SITE : http://www.architecturecentre.net/
Exclude paths :
- @NONE@
1:http://www.architecturecentre.net/
(time : 00:00:06)
No link in temporary table

--------------------------------------------------------------------------------

links found : 1
http://www.architecturecentre.net/
Optimizing tables...
Indexing complete !



There's loads of links on there!!!!

ANy ideas?

Charter
03-11-2004, 02:50 PM
Hi lighthouse, and welcome to PhpDig.net!

If you uncomment the following line in the robot_functions.php file, you'll see that the site/server sends back a "HTTP/1.1 403 Forbidden" error.

//print $answer."<br>\n";

lighthouse
03-12-2004, 01:05 AM
Hi Charter!!!

Thanks for replying....
So basically this site itself won't let me spider it?...

What would the site admin of this site have to do to allow me access?

Thanks

Charter
03-12-2004, 01:09 AM
>> So basically this site itself won't let me spider it?

Hi, yep that's it. The 403 means forbidden, not allowed, go away.

>> What would the site admin of this site have to do to allow me access?

Really, I don't have much of an idea as it's not my server, but my guess is to allow HEAD requests.