FTP to Amazon S3

Amazon S3 is a native Amazon service providing scalable cloud storage. Amazon S3 is well-known to the EPAM community as the service to be used for big data storage, transfer and distribution, archiving, backup and many other applications. However, using S3 by native AWS tools requires an Amazon account which may sometimes present a security risk, when the credentials are shared with all project members.

The introduction of the FTP2S3 service in EPAM Cloud is the first step to implementing a new data management service in the Cloud. In short, FTP2S3 allows Cloud users to access and manage S3 buckets via a project FTP server. In this case, no special AWS credentials are necessary for each project member to work with S3 buckets.

In this article, we would like to provide the step-by-step guidelines to setting up the FTP server for your project and to configuring it for S3 bucket management.

Activate your project in Amazon

If you already have S3 buckets created for your project, activate the project in the same region where the buckets are, or close to it. This will increase the response and improve the performance.

Create S3 buckets

This is a task for a project member having an AWS IAM User account. Connect to AWS Management Console and access the S3 Service. If no buckets exist for your project, create them.

Activate the FTP2S3 service for your project

The service is activated in the same way as all other platform services:

or2ms -p project -r aws-region -a -s backup -k key_name

Start the service in the same AWS region in which you have created your S3 buckets. Make sure you use an SSH key, as for AWS regions it is a mandatory option. This command creates a small Ubuntu 14.04_64-bit virtual machine which will become an FTP server for your project.

Mount the buckets

The FTP server will not be able to immediately recognize the S3 buckets you have created. For the buckets to be available via the FTP server, they have to be registered, or mounted, on it:

or2ftp -p project -r aws-region -a mount -b demo-ftp-to-s3-bucket

Always use exactly the same bucket name as is used in S3, otherwise your bucket will not be mounted. In our example, we mount the 'demo-ftp-to-s3-bucket'. When the bucket is mounted properly, it will appear in the list of buckets on your server. Display the list with the following CLI command:

or2ftp -p project -r aws-region -a list

If you need to mount other buckets, repeat the command.

Grant access to users

By default, users will not be able to access the FTP server, unless you grant them access explicitly:

or2ftpa -p project -r aws_region -a grant -e FirstName_LastName@epam.com

The users to which the FTP access has been granted, will be able to read, upload and delete files from the buckets.

Access the FTP server

To connect to the FTP server, use any convenient FTP client with the following details:

Connection DNS or Public IP
User Name FirstName_LastName@epam.com
Password Token generated by Orchestrator at or2access command run and stored in the default.cr file
Port (default) 2121

The login details can be found in the FTP service description returned by the or2dser command:

When you connect to the FTP server, you can access your S3 storage with buckets shown as top-level directories.

Now you are all set and can work with the AWS S3 service without assigning AWS credentials to each project member who needs it. You are getting the same S3 storage functionality but in a more secure way.

Should you have any questions about using the FTP2S3 service or suggestions to its improvement, please contact us at Cloud Consulting.