How We Stretched Elasticsearch to Fit Our Needs
Our DPO, Kris Lahiri once said, “If you still have to SSH to a server, you’ve already lost the battle.” At that time, we were still searching for a way to make terabytes (TB) of log data available to our engineers. Each time engineers submitted access tickets for connecting through SSH (Secure Shell), we’d remember Kris’ quote.Last year, we deployed the open source analytics platform Elasticsearch Logstash Kibana (ELK) and it’s proven to be great. The days of connecting through SSH are behind us and we’re no longer subjected to manually troubleshooting production issues. Every day our system generates around 4TB data for our 100+ Kibana users to query.Engineers around the world build security monitoring procedures with ELK to watch systems produce logs in real-time. ELK works with powerful setups like Security Onion and Wazuh to store data that must be continuously tweaked, reviewed, correlated and visualized.ELK’s low cost set up is easy and its active user community provides professional support that often surpasses what commercial solutions offer.Curious to see how our open source choice compares in the market, we looked into the third-party cost of a similar data visualization platform. It turns out vendors charge millions of dollars per year for such services. Our cost analysis proved Kibana (ELK) to be the most economical solution, providing the same, if not better service at 10% of what vendors charge.We chose to implement this open source platform because it’s not only cost-effective, but also enables us to create a solution tailored to our needs.Here’s what our team thinks:“I encourage every security team to consider Kibana as a way to expand their security toolset. ELK is here to stay, so it's wise to become familiar with it. Security is the last thing in your toolset you want compromised."
–Dawid Bałut
Security Architect | Egnyte
"Kibana helps me analyze specific issues, explore for errors and generate graphs to help understand system efficiency. Searching through days of logs now takes seconds, as opposed to minutes or timing out altogether.”
–Ran Biron
Principal Engineer
“Centralized logging with Kibana makes everyone a detective. Before, only engineers who understood the system architecture could navigate the service graph and debug issues. Kibana removed these barriers and gave everyone, from members of our customer support team to product managers, the ability to debug."
—Kalpesh Patel
Distinguished Engineer
“Searching through gigabytes of text files is an ineffective way to spend an engineer’s time. So, we deployed a centralized logs platform and built a custom pipeline based on Elastic Stack, supported with Apache Kafka. We use Google Cloud to scale with ease and currently process 70,000 messages per second during peak hours. Our logs use up to 140TB of Elasticsearch data every month.”
—Krzysztof “Eloy” Krzyżaniak
Senior DevOPS Engineer
“With little preparation, Elastic treats log entries as structured data and consequently filters by environment, service, request id, timestamp and other attributes with ease. The same can be done using traditional tools like grep, but Elastic is superior for more complex queries with ranges, varying conditions etc. Elastic’s visualization tools and basic out-of-the-box aggregation availability are also very helpful. We use these to monitor error rates, processing rates or similar metrics over some longer period of time with graphs. All in all, a great tool!”
—Tomasz Kaczmarek
Software Engineer - Egnyte Protect
“In the past, we used grep to find specific log information. It was complicated because logs were located across different servers and because of this, we lost a lot of time completing simple activities. With Kibana, checking logs is simple. We can use Lucene query syntax to find exactly what we are looking for in one place.”
—Rafał Tartanus
Senior QA Automation Engineer
“Using Kibana saves me hours of manual work. Its rich UI allows me to search and visualize things in almost any dimension without worrying about underlying complexity. Having a centralized log platform in this highly distributed environment is crucial for a production system and with Kibana, it couldn't be easier.”
—Adam Jędro
Software Engineer - Egnyte Connect
Businesses that want to avoid costly commercial tools should look into how incorporating ELK can provide high ROI for security teams.Our brilliant team of engineers used open source tools to: build a custom pipeline that streams information into Apache Kafka, create a platform to process log data, and deploy Kibana. ELK not only helps us search logs, but also generates visualizations and prepares informational dashboards available to everyone. Congratulations to our DevOps teams and especially to Krzysztof Krzyżaniak (Eloy), for making this a working reality.
