ChaosSearch
A Breakthrough Data Platform for Scalable Log Analytics

Businesses run on data, which is why demand for faster, more efficient data access and analytics is skyrocketing. And why Boston, Massachusetts-based, ChaosSearch is enjoying so much success. The company’s cloud-built data platform seamlessly automates and accelerates the process of gaining actionable insights from raw data while driving massive cost savings. "Our customers have an ever-increasing demand for access to data, but legacy systems don’t scale well and are incredibly complex to manage. We reduce the time, cost, and complexity required to get valuable insights at scale. We enable customers to know better by making their data fully and easily searchable and accessible in their own data lake," says Ed Walsh, the CEO of ChaosSearch.

While the ChaosSearch platform will support various data access tools and use cases over time, today it is most commonly deployed as a massively scalable log analytics solution, allowing customers to ingest and index billions of log files daily, in order to monitor and analyze their IT operations and perform vital security operations.

In the world of log analytics today, the most widely used solution is the Elastic Stack, also known as the “ELK Stack” because it consists of three primary components: the Elasticsearch analytics engine, the Logstash data pipeline, and Kibana for search, queries and visualizations. However, according to Walsh, the vast majority of enterprise customers are not able to analyze the massive amounts of data they generate due to the ELK Stack’s inherent challenges. Managing the environment is extremely time-consuming and expensive. Its underlying database limits the ability of a single cluster to scale, requiring IT to set up and maintain multiple clusters, and to break up, or shard, the database to enable growth. The more data they push into the ELK Stack, the less stable and more costly it becomes. These challenges are exacerbated any time there is an underlying change to the data, or a change in how users want to access the data, which requires a very difficult and time-consuming process of re-loading and re-indexing the entire database. As a result of these challenges, businesses are forced to limit the amount of data they retain and analyze, limiting their ability to extract the vital insights they need from their data.

ChaosSearch takes a radically different approach to log analytics. Its cloud-native data platform connects to a customer’s existing cloud storage, such as Amazon AWS or Google GCP, indexing all of the data within it regardless of the scale. As new data floods in daily, ChaosSearch continually indexes the data in real time without any performance hit to the data ingest speeds. This renders all of the data searchable and ready for use by commonly used analytics tools, leveraging open APIs. Think of it as an ELK stack alternative — delivered as a managed service or deployed via the customer’s Virtual Private Cloud — without the cost, complexity, limitations, and instability of today’s ELK stack.

At the heart of the ChaosSearch platform is the innovative Chaos Refinery, which allows IT to change data schemas and allow for differing views into the underlying data, without the difficult process of re-indexing and re-loading the data.

We enable our clients to know better by making their data fully and easily searchable and accessible. Our mission is to vastly reduce the amount of time, cost, and complexity required to get valuable insights at scale

This unique approach allows different tools and different use cases while accessing the data within the massive underlying data lake. So while ChaosSearch is ideal for log data management today, Walsh explained that the Platform will support other tools and use cases down the road.

Later this year, ChaosSearch will introduce support for SQL, which will enable users to conduct business intelligence analyses using tools like Looker and Tableau. "Our refinery allows clients to give the right people access to the right data at the appropriate time, using the tools they already use," comments Walsh.

Regardless of the use case, with ChaosSearch, customers get instant elasticity, unlimited scale and data retention, and secure data access while saving significant amounts of money. This value proposition is proven time and again by ChaosSearch customers, such as HubSpot. HubSpot had challenges keeping up with the scale and volume of their Cloudflare logs which are a critical part of preventing and mitigating denial of service (DDoS) cyberattacks. The massive daily volume of these logs, which total over 20 TB per day, were extremely difficult and expensive to ingest and use within their sprawling ELK environment. Because of the volume and expense, Hubspot was forced to keep only 5 days of Cloudflare logs, severely limiting their ability to analyze historical data.

When they switched to ChaosSearch, they were immediately able to move to 90 days of log data retention. Since deployment, they’ve solved the instability challenges and eliminated the outages that were once common. Moreover, they’ve saved over $2.8 million and counting just from the single Cloudflare use case. On the strength of their initial success with ChaosSearch, HubSpot quickly added other use cases, keeping all of their data in one easily searchable, unified data lake — all with unlimited scale and as much retention as they want.

Unsurprisingly, ChaosSearch’s pricing is disruptive. For its log analytics managed service, which is priced at 70 cents per indexed gigabyte, customers routinely save up to 80% compared to their legacy systems. Walsh says, “With ChaosSearch, companies get insights at scale while achieving the true promise of cloud data lake economics.”