Following the move of Cognito iQ’s infrastructure to the cloud and the growing needs of both the engineering and support teams at the company, the existing methods and tools used for monitoring and logging became too costly. The manual work involved in extracting critical insights from the data consumed too many valuable resources from the developers.
To replace these methods, Cognito iQ gradually constructed an ELK-based centralized logging and monitoring system. 'ELK' stands for Elasticsearch, Logstash, Kibana. Elasticsearch is a schema-less document database built on top of Apache Lucene™. It features high performance, full-featured information retrieval library (full text search) - and supports real-time data and real-time analysis. Logstash is an open source tool for collecting, parsing, transforming, and storing logs of any type into Elasticsearch for future use. And Kibana is a real-time data analytics interface that allows our staff to query, graph, and alert on any Elasticsearch metrics we choose. ELK was chosen for a number of reasons, but first and foremost, they felt that ELK had the flexibility and robustness to handle the requirements of their architecture and company needs.
As Chris Funderburg, site reliability engineer at Cognito iQ, puts it: “ELK was our choice from day one because it was becoming the industry standard, there was a lot of support online, and it did most of what we needed. We found that we could use it for monitoring, metric display, and reporting as a bonus. It’s given us almost-infinite flexibility in how we monitor for problems because our individual support users can craft queries all day long without developers or senior engineers getting involved.”
Why We Chose Logz.io
The DevOps team at Cognito iQ decided to move to Logz.io after trying other ELK solutions that did not satisfy the needs of the company.
The first Elasticsearch service they tried suffered from performance issues on a technological level because it used an outdated version that did not support the needs of the team.
Cognito iQ then decided to build their own ELK Stack, a task that quickly became too resource-consuming to maintain in the long run. With limited resources and higher priorities, the company decided that they would rather hand out the heavy lifting involved in maintaining a large ELK deployment to Logz.io.
Making the Move to Logz.io
The move from Cognito iQ’s ELK stack to Logz.io was smooth. The existing logging infrastructure consisted of one Logstash instance per server. To integrate with Logz.io, the only engineering that had to be done was configuring Logstash to output to the Logz.io listeners.
Who Uses Logz.io
Cognito iQ ships around 150 GB of log data to Logz.io per day. This data is used not only by the DevOps team for log analysis and troubleshooting issues but also by the company’s support team, which uses Logz.io to analyze the behavior of users and monitor error messages.
“For Cognito iQ, having the ability to share log data with multiple users and across multiple indexes is key to correctly understanding what users are experiencing and — ultimately — how to resolve cases more quickly.”
Instead of spending a huge amount of resources on building and maintaining an ever-growing ELK Stack, the DevOps team at Cognito iQ can now focus on improving the company’s core engineering processes. The support team can also use Logz.io to monitor for and resolve issues before they impact customers.