Cloud Computing today is accessible by everyone: anyone can launch a EC2 instance on AWS or write entire systems using Serverless technologies without launching even a single VM. The on-going competition between cloud giants AWS, Google Cloud Platform and Microsoft Azure keeps bringing prices down and creating your own full-scale cloud solution is cheaper than ever.

Data Warehouses, however, are by definition not going to be cheap to create and operate. Today, only rich companies or truly data-driven companies can afford to put efforts into launching a real data warehouse. Even fewer are the organizations that run data warehouses in really amazing scales.

It is no surprise then that it is companies like Google, Uber, Netflix and Facebook that are driving innovation in the Data Warehousing space. And by Data Warehousing we mean a solution or platform which let's you run ad-hoc queries on large volumes of data (so called "BigData"); efficient storage of this data; and cataloguing the data in a way that makes it easy to later append or update it.

You shouldn't underestimate th...

Presto is a state of the art Distributed SQL Query Engine for BigData, enabling efficient querying on cold data and various data sources. With extended SQL language and features like geospatial queries, joins between different data sources (SQL to join data from HDFS, Elasticsearch, and Kafka anyone...

A few weeks ago I gave a talk on Google Campus TLV on deploying and running Elasticsearch on Kubernetes - best practices and various gotchas. The video for the talk is below.

There is also a blog post with most of the technical details here.

On that note, you should definitely check out Elastic's re...

Kubernetes is quickly becoming the de-facto standard for running systems in the cloud and on-premises, and in the last couple of years we at BigData Boutique have had to deploy and support quite a few Elasticsearch clusters on Kubernetes.

Now is probably a good time to reflect on this and have a hig...

I just read the recent announcements from AWS on their fresh "Open Distro" of Elasticsearch. Statements like "Keeping Open Source Open" and "it has created uncertainty about the longevity of the open source project as it is getting less innovation focus" drew my attenti...

I just published Easily deploying Presto on AWS with Terraform, but ignored a very important question: AWS offers Athena for SQL over S3, which is essentially a Presto deployment managed by AWS. Why not just use AWS Athena instead of going through the trouble of deploying your own cluster?

I probabl...

In the past year or so we have been involved in creating several data-lakes and data-warehouses - some from scratch and some in a brownfield environment where we had to either switch from traditional SQL clusters or run side-by-side with them.

Our tool of choice for running SQL on top of "BigDa...

Terraform is my go-to tool for repeatable and easy infrastructure deployments. I've previously shared how I deploy Elasticsearch on AWS with Terraform and Packer, and since posting that I used it to deploy many clusters, and it also got picked up by quite a few others.

Our offerings at BigData Bouti...

That thing that happens when your company is scaling so fast you have to replace your infrastructure within weeks or the whole thing crashes. This is what happened to our customer, and this is the story of how we replaced good-ol' SQL with streams and batch processing technologies on AWS.

Using Terr...

Elasticsearch is very widely used today for text and geospatial search, real-time BI dashboards and log analysis. While it is tempting to use a managed Elasticsearch cloud service instead of running your own cluster on your own machines, Amazon's Elasticsearch Service is a bad choice, as bad as it g...