PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Troubleshooting

Reply
 
Thread Tools
Old 03-11-2004, 06:50 AM   #1
lighthouse
Green Mole
 
Join Date: Mar 2004
Posts: 20
Can I spider .asp pages?

SUrely an easy answer to this... but I can't find it!!!

I'm trying to crawl a website built entirely in asp, but it wont spider past the fron page.

Please help!

Thanks
Gary
lighthouse is offline   Reply With Quote
Old 03-11-2004, 01:26 PM   #2
lighthouse
Green Mole
 
Join Date: Mar 2004
Posts: 20
Why Can't I dig this site????

http://www.architecturecentre.net


digging results in:

SITE : http://www.architecturecentre.net/
Exclude paths :
- @NONE@
1:http://www.architecturecentre.net/
(time : 00:00:06)
No link in temporary table

--------------------------------------------------------------------------------

links found : 1
http://www.architecturecentre.net/
Optimizing tables...
Indexing complete !



There's loads of links on there!!!!

ANy ideas?
lighthouse is offline   Reply With Quote
Old 03-11-2004, 01:50 PM   #3
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi lighthouse, and welcome to PhpDig.net!

If you uncomment the following line in the robot_functions.php file, you'll see that the site/server sends back a "HTTP/1.1 403 Forbidden" error.
PHP Code:
//print $answer."<br>\n"; 
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 03-12-2004, 12:05 AM   #4
lighthouse
Green Mole
 
Join Date: Mar 2004
Posts: 20
Hi Charter!!!

Thanks for replying....
So basically this site itself won't let me spider it?...

What would the site admin of this site have to do to allow me access?

Thanks
lighthouse is offline   Reply With Quote
Old 03-12-2004, 12:09 AM   #5
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
>> So basically this site itself won't let me spider it?

Hi, yep that's it. The 403 means forbidden, not allowed, go away.

>> What would the site admin of this site have to do to allow me access?

Really, I don't have much of an idea as it's not my server, but my guess is to allow HEAD requests.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
dig for certain words nmott How-to Forum 1 02-26-2005 08:02 PM
Search Engines and Dig new2dev How-to Forum 1 02-21-2005 01:19 PM
is 'dig 1.8.3 slower than 'dig 1.8.0 JÿGius³ Troubleshooting 1 08-02-2004 07:32 PM
PHP dig not indexing site on clean install... mixonic Troubleshooting 1 06-28-2004 08:15 AM
how to dig only 1 page zaartix Troubleshooting 8 05-11-2004 01:23 AM


All times are GMT -8. The time now is 02:26 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.