13462101:iLLbiLLy said:
Yea I totally understand. Yet there are ways to do it without risking performance degradation or impact to the site. First off, as I mentioned you would put it on a completely separate service. A PaaS like Azure is a good choice because it's cheap and allows for unlimited scaling. Their cloud service and cloud storage is set up perfectly for something like Lucene. NS is fairly large, but still on the small side of what Lucene was meant to handle.
The API would be very minimal:
Id findDocument(String phrase, [options]);
void indexDocument(Document doc);
void deleteDocument(Id documentId);
Simple 2-way SSL or oAuth could be used to secure the service. You could leave findDocument public if desired so it could be called from other clients in the future (native iPhones).
For the initial index, you would take a backup of your database and index directly on server.
Before you ever make a single change to NS you do QA and load testing using something like Soap UI. Only actually hook it up to NS once your sure everything is behaving as expected. Once its up and running, costs would be minor.
It's been a while, but last time I looked at the 3rd party service they were limited and expensive.
We are aware of the available technologies and how to use them, and we even have plans for a new search system in mind. There are just other changes we need to do first.
A proper site-wide search is something we intend to add soon, likely something based on Solr or Sphinx. At the moment our problem isn't so much integrating with the search system itself, it's the fragmented nature of our content storage.
NS always had a Picture section, Video section, Forums, News, Reviews, Members+profiles, etc... Each of these sections should be searchable. If we implemented a site-wide search today, each section would need to separately maintain their data in the search system for additions, changes and removals. It's possible but would be a bit of a mess.
The change I referenced yesterday regarding the "content listing" will simplify this. Until 2-3 years ago, the homepage of NS was separated into modules for each sub-section. So there was the News slider, Video Of The Day, then lists of Blogs, Photos and Reviews. There was no crossover between site sections. When we moved to the block-type layout for the homepage we decided that content is content and they should all be listed together based on member activity. So we began to list all content from the whole site in one big mashed-up list. Today the homepage still does this, though the layout is visually more list-like.
The challenge to make this was to efficiently fetch content from the whole site in a reliable manner. Since the new homepage was an experiment we decided to just fetch content from each section as best we can and merge them in PHP. It's slow but would suffice to test the idea.
2 years later we've decided it was a good move, and needs to go further. Currently I'm working on building an internal "content list" that every site section hooks into. The content listing would maintain info like Rating and Heat for every single piece of content on the whole site. The end goal would be to provide lookup capabilities, like get the "Top Rated Pictures+Viedos+News by Mr.Bishop" or "Get Latest Content from Whole Site", and it would be reliable and extremely fast.
The benefit is obvious, however there are additional features we can build off this. Search is one of them. Since each section will maintain their listing in this new system, it gives us a single point where we can then push that information to a Search system.
