A couple of months ago I started working at Buzzilla, a company developing "cutting edge technologies and revolutionary analysis and research methodologies that combine to create advanced solutions aimed at harnessing the vast opportunities presented by online conversation". Or in short, full-text search and analytics on BigData.
When I started there the existing system was based on a home-brewed solution for distributing a Lucene index across dozens of nodes. Since we are looking to expand way beyond the number of a couple of dozen servers, we really needed to recreate the system using tools better suited for the job, which we could also take to the cloud. And it is much more than just about the search engine.
Refactoring an operational distributed system is really a great challenge. While keeping it operational, you need to replace parts one by one, but you also get the opportunity to experiment with new tools and bleeding edge technologies. It is also a great subject for a series of blog posts, this one being the first of many to come. Among the items we had to tackle on which I'll blog are:
Such a refactoring process could easily turn out to be a disaster. This is the story of a careful planning and a great team working together, which made this a success. We are now in the final stages of migrating to the new system, and we have many more challenges pending after we have gone completely live. This series is going to span months and hopefully have some great content which will spark great discussions.
My name is Itamar Syn-Hershko, and my main interests and expertise are in search engines, databases, distributed systems and architecture.
I'm a software dev with Buzzilla, consultant, speaker and frequent OSS contributor.
More about me here: http://code972.com/about