I guess it all depends on what specifically you are trying to accomplish when you talk about customizing a search engine for your website. phpDig is open source, and as such, is fully customizable.
Do you need some recommendations of good PHP/MySQL books so you'll know what you're doing when you try to modify the code? If so, there are some excellent books you can order through sitepoint.com.
Hope this helps. If you're looking for something else, let us know and we'll be glad to help.