Spidersearch other websites

Swashbuckler · 02-04-2005, 01:09 PM

Just a general query I have that hopefully can be answered. I have constructed a website and want to insert a searchengine that is linked to an external website. As a basic example to have a search engine in mysite.com that would be able to return results from amazon.com.

Is this feasable using Phpdig?

jmitchell · 02-04-2005, 01:17 PM

can you clarify...

do you want to spider amazon.com, or do you want to get a feed from amazon.com and display it using phpdig?

JMitchell

Swashbuckler · 02-04-2005, 01:32 PM

I'm just using amazon.com as an example.

I'm a newbie when it comes to spidering so I would appreciate any help.

In answer to your question I would like the ability to mine data about a product, price, specification, from Amazon.com onto a MySQL database.

Alternatively would it be possible to feed the results from Amazon.com and filter the info (again according to price, spec etc).

Thirdly can you search multiple websites and filter the info mined?

jmitchell · 02-04-2005, 01:43 PM

well, using amazon as an example, amazon provides an xml feed which you can use to get the info you need. You can use PHP's xml functions to parse the xml and display it. There would be no need for a database because you could pull it from amazon each time you need it.

Other sites would be harder because they might not have a feed. You could request it though.

If they don't want to provide a feed/you don't want to ask them, you will have to index the page, and then write some functions to get the data you want, and not the rest of it, which can be tricky.

HTH,

JMitchell

Swashbuckler · 02-04-2005, 02:04 PM

Many thanks Jmitchell for the advice.

Whats the best way to proceed with the two approaches that you specified? e.g. can I use phpdig to that end? Can I use standard php coding etc.

02-04-2005, 01:09 PM	#1
Swashbuckler Green Mole Join Date: Feb 2005 Posts: 3	Spidersearch other websites Just a general query I have that hopefully can be answered. I have constructed a website and want to insert a searchengine that is linked to an external website. As a basic example to have a search engine in mysite.com that would be able to return results from amazon.com. Is this feasable using Phpdig?

02-04-2005, 01:17 PM	#2
jmitchell Orange Mole Join Date: Dec 2004 Location: Tennessee Posts: 60	can you clarify... do you want to spider amazon.com, or do you want to get a feed from amazon.com and display it using phpdig? JMitchell __________________ 60,000 pages indexed!!!!! http://www.sharemylink.com

02-04-2005, 01:43 PM	#4
jmitchell Orange Mole Join Date: Dec 2004 Location: Tennessee Posts: 60	well, using amazon as an example, amazon provides an xml feed which you can use to get the info you need. You can use PHP's xml functions to parse the xml and display it. There would be no need for a database because you could pull it from amazon each time you need it. Other sites would be harder because they might not have a feed. You could request it though. If they don't want to provide a feed/you don't want to ask them, you will have to index the page, and then write some functions to get the data you want, and not the rest of it, which can be tricky. HTH, JMitchell __________________ 60,000 pages indexed!!!!! http://www.sharemylink.com

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Search Engine / Directory Submittion Tools & Websites	Paradoxic	IPs, SEs, & UAs	0	04-10-2005 01:28 PM
Indexing chinese websites	GeminiHB	How-to Forum	1	05-13-2004 04:34 AM

02-04-2005, 01:32 PM	#3
Swashbuckler Green Mole Join Date: Feb 2005 Posts: 3	I'm just using amazon.com as an example. I'm a newbie when it comes to spidering so I would appreciate any help. In answer to your question I would like the ability to mine data about a product, price, specification, from Amazon.com onto a MySQL database. Alternatively would it be possible to feed the results from Amazon.com and filter the info (again according to price, spec etc). Thirdly can you search multiple websites and filter the info mined?

02-04-2005, 02:04 PM	#5
Swashbuckler Green Mole Join Date: Feb 2005 Posts: 3	Many thanks Jmitchell for the advice. Whats the best way to proceed with the two approaches that you specified? e.g. can I use phpdig to that end? Can I use standard php coding etc.