Ax: Query Logs with Confidence

May 17, 2018

Modern log aggregation stacks including ELK (Elasticsearch, Logstash, and Kibana), Google’s Stackdriver, or AWS Cloudwatch are great tools, but their browser-based interfaces are not for everybody. At Egnyte, we rely heavily on logs to monitor our systems and solve customer issues. Therefore, many of our engineers depend on Kibana, the ELK browser-based query tool. In many ways, Kibana is a great tool, but there is tremendous value in the scriptability and simplicity of a command line interface.If logs could be queried easily using a command line tool, we could write Bash scripts that parameterize queries, or further process search results using classic Unix tools and advanced custom scripts. Basic tools like Elktail do this but we found that their features are either limited or tied to one particular log aggregation back end.Therefore, we developed Ax; an open-source command line tool capable of querying multiple log sources, including:

Kibana indexes
Stackdriver logs
Cloudwatch logs
Docker container logs
JSON or plain text log files (any process can be piped into Ax directly)

Ax supports multiple environments, such as an index in Kibana or a group in Cloudwatch, and effortlessly switches between them using a single command line flag. Queries may contain a plain text search, or be filtered by attributes of log messages and time ranges. To reduce output clutter, a subset of return attributes can be selected.Here’s an example of an Ax command:ax --env kibana-prod --where level=DEBUG --select message --select domain "Unsupported characters"This queries the “kibana-prod” environment for the text phrase “Unsupported characters” to debug log messages, returning only the “message” and “domain” attributes.By default, Ax outputs colored plain text, but to further process the output, it can also present results as JSON (--output json or --output pretty-json) or YAML (--output yaml). Ax also supports a --follow (-f) flag that attempts to “tail” logs in quasi-real time (returning newly matched results in quasi-real time). But perhaps the coolest feature is that it offers command line completion for all subcommands, flags and attribute names, simply by hitting Tab. This makes using Ax a great command line experience.We are still early in development, but many of our engineers are already relying on Ax every day. Ax is written in Go, and binaries can be downloaded for both Mac and Linux with a single command. The self-upgrade functionality helps ensure that your Ax install stays up to date. Give Ax a try and see for yourself. We look forward to improving this tool based on your feedback.