PhpDig.net - View Single Post - phpdig seems to guess some urls and spider it

manute · 04-27-2004, 11:29 AM

doesn't anyone have an idea about that? that gives me a stupid lot of duplicate urls, that really sucks.
is there any way that i can tell phpdig to only spider what it gets with a link without "guessing" urls?

04-27-2004, 11:29 AM	#2
manute Orange Mole Join Date: Oct 2003 Location: hamburg, germany Posts: 52	doesn't anyone have an idea about that? that gives me a stupid lot of duplicate urls, that really sucks. is there any way that i can tell phpdig to only spider what it gets with a link without "guessing" urls?