Splunk may be the most famous way to make sense of mass quantities of log data, but it’s far from the only player around
Quick! Name a log analysis service. If the first word that popped out of your mouth was “Splunk,” you’re far from alone.
But Splunk’s success has spurred many others to up their log-analysis game, whether open source or commercial. Here are six contenders that have a lot to offer sys admins and devops folks alike.
ELK/Logstash (open source)
Splunk faces heavy competition from the family of projects that use the ELK stack: Elasticsearch for search, Logstash for data collection, and Kibana for data visualization. All are open source.
Elasticsearch, the company that handles commercial development of the stack, provides all the pieces either as cloud services or as free, open source offerings with support subscriptions. They provide the best alternative to Splunk when used together, since Splunk’s strength is in searching and reporting as well as data collection.
Graylog (open source with commercial version)
Graylog made a name for itself last year when it unveiled its 1.0 release. Like the ELK stack, it uses Elasticsearch as its core component, but also relies on the MongoDB data store and the Apache Kafka messaging broker system. The core product is available for free; the enterprise version adds functions like archiving.
Version 2 of Graylog (currently in beta), upgrades to the latest version of Elasticsearch and adds more archiving functionality, new filtering and message-processing options, and the ability to see what’s passing through Graylog with nothing more than a tail -f command.
Sumo Logic (cloud service)
Two years ago, Sumo Logic made the list of 10 big data startups to watchat Network World. Back then, the company’s big selling point was using machine learning to find out what data to get insight into the first place, then analyze it with, well, more insight.
The most recent release of the company’s cloud-native log analysis service focuses on how the service can “natively ingest, index, and analyze structured metrics data and unstructured log data together in real time.” In other words, it lets you take all the different data sources, put them into a central place, keep their native formats, and analyze them side by side.
If you want to get started right now, the free tier lets you ingest up to 500MB per day with seven days of data retention.
Logentries (cloud service)
Logentries‘ log-data collection and analysis service (recently acquired by security firm Rapid7) originally caught my eye when it added logging from Docker containers as a features. Besides collecting data from container systems like Docker and CoreOS, Logentries can slurp up events from Logstash, PagerDuty, and New Relic, as well a take in alerts from notification/discussion systems like Slack and HipChat. Most anything else can manually hook into Logentries with Webhooks and Logentries’ API.
Loggly (cloud service)
Loggly focuses on collecting logs from systems that have a syslog-compatible agent (anything that uses RFC 5424, basically), and making it available for fast searching and analysis via a RESTful API.
This sounds modest, but it covers a lot of territory. For example, data from AWS CloudTrail and New Relic can all be aggregated and analyzed together. Loggly is also appealing to those who don’t want to become analytics experts to get useful results, thanks to its Web-based dashboarding.
Those with modest needs or who simply want a taste test can try out the free tier as long as they ingest only 200MB per day and don’t mind having only seven days of data retention.
Sematext Logsene/SPM (cloud and on-prem)
This outfit also came to my attention by way of its Docker integrations, but it includes far more. Sematext’s Logsene product is ELK as a service: a hosted ELK stack, available either in the cloud or behind the firewall, that works with any log-shipping service. SPM integrates with 40-plus services and apps to generate contextual information about what’s going on inside your organization. A 30-day free trial tier is available.