The story of a massive system refactoring for the cloud: prologue
A couple of months ago I started working at Buzzilla, a company developing "cutting edge technologies and revolutionary analysis and research methodologies that combine to create advanced solutions aimed at harnessing the vast opportunities presented by online conversation". Or in short, full-text search and analytics on BigData.
When I started there the existing system was based on a home-brewed solution for distributing a Lucene index across dozens of nodes. Since we are looking to expand way beyond the number of a couple of dozen servers, we really needed to recreate the system using tools better suited for the job, which we could also take to the cloud. And it is much more than just about the search engine.
Refactoring an operational distributed system is really a great challenge. While keeping it operational, you need to replace parts one by one, but you also get the opportunity to experiment with new tools and bleeding edge technologies. It is also a great subject for a series of blog posts, this one being the first of many to come. Among the items we had to tackle on which I'll blog are:
- Building a distributed and highly available search engine
- Various topics on full-text search: relevance, scoring, multi-lingual search, best practices for analysis and more
- Choosing the right web framework for a web UI
- Migrating from MySQL to NoSQL - and selecting the right NoSQL for the job
- Caching
- Keeping tabs of logs in a distributed environment
- Planning for and recovering from failures and crashes
- CI, deployment, versioning and backups
- Identifying and fixing performance bottlenecks
- Generating and displaying system stats and performance metrics
Such a refactoring process could easily turn out to be a disaster. This is the story of a careful planning and a great team working together, which made this a success. We are now in the final stages of migrating to the new system, and we have many more challenges pending after we have gone completely live. This series is going to span months and hopefully have some great content which will spark great discussions.