Adding Search to Your Applications: Lucene and Elasticsearch on .NET Rocks(Cross-posted from BigData Boutique Blog)
This week I had the pleasure of talking to Carl and Richard of the .NET Rocks fame. We had a great discussion on search engine libraries and search servers - where they fit in the general scheme of things, and when and how one should use them.
You can go and listen to the recording here.
A quick run down of the things we discuss:
- Relational databases like SQL Server have only basic support for full-text search usually, definitely not something you can do a lot of with. And don't even mentioned the horrendous LIKE operator.
- This also holds true for most NoSQL solutions - if they have full-text support, it is usually limited. For example, language support, relevance ranking and overcoming spelling errors - are all hardly supported if at all.
- Search engines libraries like Java Lucene and Lucene.NET provide easy to use API to create and embed search functionality in your applications. And they are completely free and open-source (ASL).
- Search servers like Solr and Elasticsearch extend this power for cloud-based applications, or for scenarios when data can't simply be stored on one server. Both are open source as well.
- Full-text search - that is the ability to search for textual data within an unstructured stream of text - is only the tip of the iceberg with modern search engines. Geo-spatial search, image search and real-time aggregations are just an example of extremely powerful search features one can build on top of search technologies.
- We discussed patterns for using search engines technologies. Generally, they should be thought of as a projection of your data that is volatile and being used as a read-facade.
- Elasticsearch is actually only part of a bigger stack. Logstash and Kibana are two additional parts of what's called the ELK stack - and they let you collect data from various sources and then visualize search results and real-time aggregations produced by Elasticsearch, the search server part of the stack.
- An open call for participation in the Lucene.NET open-source project. Grab the sources from github, and we are waiting to hear from you in our mailing lists!