Splunk as s Service - How We Took a Good Thing and Made It Better

Every project larger than a couple of servers requires support. And, naturally, support requires monitoring to be able to promptly respond to any requests or complaints. An important component of monitoring is log collection - gathering data from various resources and applications making up your system with the ability to search and analyze it later.

Realizing the importance of log collection for virtually every project running in EPAM, the Cloud Team developed a log aggregator as one of the very first services it ever offered. Several years of improvements and enhancements has turned it into a solid and reliable platform with Graylog2 collecting the log data and Elasticsearch performing the functions of a full-text search engine. The combination has proved to be as reliable as a Swiss watch and, on top of that, it is open source. You only need to start a virtual Graylog2 server - and within minutes your first log data is already coming in. Your bill at the end of the month will include only the cost of your virtual machine.

However, we noticed that many projects prefer Splunk as their log aggregation choice when they select the components of their infrastructure. Splunk is a licensed software, still, projects are prepared to purchase Splunk licenses for their log collection functions. Splunk is easy to set up and use, requires no databases to store data, processes data in almost any format - the advantages are clear.

"OK," we thought, "if you can't beat it, join it." And in September, 2016, EPAM Orchestrator release came with Splunk as a Service - your good old Splunk but running on a virtual server in Cloud.

"This is all very interesting," you would say, "but why would I run my Splunk in Cloud when I can have it for free by getting a free license?"

True, Splunk does have a free license if you limit your daily traffic to 500 Mb... but no mechanism to control it. As soon as you exceed your limit, your free license is blocked and you are required to purchase one, as you are no longer eligible for a free license.

This is where running Splunk in Cloud can be a perfect solution. In its current implementation Splunk as a Service is offered in its trial version which is to be immediately changed to free license. However, we have also developed Splunk proxy - an application to stand between your Splunk and your system and to limit the data traffic. Once you have reached your free limit, the traffic is stopped. This way, you can be sure that you will never run over the free limit and breach your free license.

How It Works

Splunk is activated as any other platform service in EPAM Cloud - you send one command in Maestro CLI, and your virtual Splunk server is started with Splunk Enterprise trial version already installed and configured. According to the Splunk policy, you can use the trial version for up to 60 days, however, we recommend changing it to the free license immediately, otherwise, your data will be lost.

Basically, this is already the minimum required Splunk configuration. If you are sure that your data traffic will never exceed 500 Mb on any particular day, you are OK with just one Splunk server which will cover all your log aggregation tasks. However, if there is a risk of any excessive traffic, set up a Splunk proxy to be on the safe side.

A Splunk proxy is another virtual machine which will control your data traffic. Once your Splunk proxy is up, add endpoints specifying Splunk server ports receiving data from certain applications and the traffic quotas for each endpoint. When the daily traffic reaches the quota, Splunk proxy will interrupt data flow to the corresponding port and resume it on the next day when the quota is restored. If you set up the endpoints so that all your applications together generate up to 500 Mb per day, you can rest assured that you will always stay within the scope of your free license.

This solution is already available in EPAM Orchestrator version 2.1.77. We have released it as a beta-version, as we realize how unusual it may seem. This is the first time we release a licensed software as a cloud service. We want it to be tested inside and out. Please let us know if it works for you and, especially, if it does not. And, of course, tell us if any other useful features may be added or if something should be tweaked to make it just perfect.

You can find a detailed description of Splunk activation and usage in the Services Guide. Please send your feedback, ideas and suggestions to EPAM Cloud Consulting Team .