Apache Hudi is a data Lake technology that has been in use since 2016. Originally built by Uber, Hudi is the first of the three data platforms that we’re going to examine in detail. The series has in-depth reviews of Delta Lake and Apache Iceberg, and will end with a comparison between those three p...

Let’s cut to the chase. Every Elasticsearch and OpenSearch user and administrator, whether on a managed platform or self-hosted, knows this feeling - endlessly hoping the cluster keeps up and doesn’t crash, and dreading the on-call alert in the middle of the night that demands action - which is ofte...

In our introductory post we discussed the typical structure and usual components of a modern data platform.

A very common component of any Data Lake and Data Warehouse implementation is what we often call the “Cold Storage” tier. This is where, or rather how, the vast majority of data is persisted i...

We live in an era of data. Data is in every organization’s strategy, every engineer’s job description, and every CIO’s dreams (or nightmares). Day in, day out, more data collectors and more data generators are being built. The collectors are observability tools and data platforms, and the generators...

Cloud Computing today is accessible by everyone: anyone can launch a EC2 instance on AWS or write entire systems using Serverless technologies without launching even a single VM. The on-going competition between cloud giants AWS, Google Cloud Platform and Microsoft Azure keeps bringing prices down a...

Presto is a state of the art Distributed SQL Query Engine for BigData, enabling efficient querying on cold data and various data sources. With extended SQL language and features like geospatial queries, joins between different data sources (SQL to join data from HDFS, Elasticsearch, and Kafka anyone...

A few weeks ago I gave a talk on Google Campus TLV on deploying and running Elasticsearch on Kubernetes - best practices and various gotchas. The video for the talk is below.

There is also a blog post with most of the technical details here.

On that note, you should definitely check out Elastic's re...

Kubernetes is quickly becoming the de-facto standard for running systems in the cloud and on-premises, and in the last couple of years we at BigData Boutique have had to deploy and support quite a few Elasticsearch clusters on Kubernetes.

Now is probably a good time to reflect on this and have a hig...

I just read the recent announcements from AWS on their fresh "Open Distro" of Elasticsearch. Statements like "Keeping Open Source Open" and "it has created uncertainty about the longevity of the open source project as it is getting less innovation focus" drew my attenti...