Direct AWS EC2 Usage Monitoring

By Ganna Shargorodska

Basic AWS Usage Monitoring

As you could see in the announcement of the 2.1.93 Orchestrator release, we have implemented support of direct monitoring of AWS EC2 billing. If your project uses AWS and if you wish to obtain billing data directly from Amazon, you need to take the following steps:

  • 1. Create or select the S3 bucket where you would like your billing reports to be stored. This is an optional step, as you can request a dedicated S3 bucket to be created for you together with the service activation.
  • 2. Request direct monitoring of AWS EC2 billing via the Support Portal. If you have no S3 bucket to store billing reports, it will be created for you.

This is the minimum preparation work you need to get done to have AWS EC2 billing reports delivered directly to you. The reports will be stored in the S3 bucket in the CSV format. You can download the files and analyze them locally.

However, there is a solution for advanced analytics of your billing data - Amazon QuickSight. With QuickSight, you will get your business analytics to a new level - as highly visual, graphic and interactive dashboards where you can analyze and process your billing data by multiple criteria.

Below, are the detailed instructions on QuickSight activation and setup for your project.

QuickSight Activation and Setup

To have Amazon QuickSight activated for your project, specify it in the request for direct monitoring of AWS EC2 billing. When the service is activated, use your AWS account to access it and configure it as follows:

  • 1. Click your user name and select Manage QuickSight from the dropdown menu:


  • 2. This will open the Account settings screen. Click Edit AWS permissions to set the QuickSight permissions to access AWS resources.
  • 3. On the next screen, check the Amazon S3 (all buckets) checkbox and click the Choose Permissions link to specify the bucket(s) to which QuickSight must have access. The system will list all S3 buckets found for your project. You can either grant QuickSight access to all your buckets by checking Select all or choose individual buckets.


  • 4. Once you have specified the bucket(s) which QuickSight can access, set the data source - the file(s) to be used for advanced analytics. You can upload a file manually or select one or several files from an S3 bucket.
    Click Manage data on the QuickSight start page, then choose New data set on the Your Data Sets page. Click the S3 icon in the Create a Data Set section.


This will open the New S3 data source window. Specify the data source name (a human-friendly name to serve as the reference to the data source) and then upload a JSON manifest file specifying the path to the CSV file(s) in your bucket. You can use one or several files as a single data source. See Amazon documentation for more information on the supported formats and elements of manifest files. Below is an example of a basic manifest file specifying one file in the S3 bucket to be used as data source:

{
"fileLocations": [
{"URIs": ["https://s3.amazonaws.com/test-bucket/017288127620-aws-billing-detailed-line-items-with-resources-and-tags-2017-04.csv.gz"]}
]
}


  • 5. When the data set has been created, click Visualize to start with your business analytics.


QuickSight Use Cases

QuickSight has numerous analytics options and possibilities. Select the data fields you would like to analyze, set filters, choose the type of visualization best suiting your purposes - and get strong graphical representation of your data.

You can analyze your AWS usage costs by various criteria - for example, break them down by availability zones:

Or filter your resources by tags to compare their usage on a diagram:

You can even vizualize your costs by zones over time to identify trends and make some conslusions on resource usage optimization:

Pricing

Please note that QuickSight is a paid service. As of the day of this article (May 3, 2017), QuickSight is free for the first user in both Standard and Enterprise edition, and is charged $9 for each additional user in the Standard edition and $18 - in the Enterprise edition. See the current pricing information on QuickSight official website.