FTP to AWS S3 Service (FTP2S3)

FTP tp AWS S3 is the first stage of the new data management service implementation. It is represented as FTP access to AWS Simple Storage Service (S3), a service that allows you to store any amounts of your data in Cloud, and share it with your colleagues.

By default, storage management is performed via AWS Management Console and AWS CLI.

Establishing FTP access to AWS S3 storage allows to bring data management to a more common format. In addition, people who have access to S3 via FTP, do not have to have AWS credentials to work with the storage, which increases the project security.

The service is available in AWS regions only.

For more information on AWS S3 service, visit the Amazon website.

Have a Question?

The current page gives the general information on the service and the main workflows. However, while working with the services, our users encounter new questions they need assistance with. The most frequently asked questions on EPAM Cloud Services are gathered on the Cloud Services FAQ page.
Visit the page to check whether we have a ready answer for your question.

Related CLI Commands

The table below provides the list of service-related commands and their descriptions:

Command Short Command Description
or2-manage-service...-s backup -a -k key_name or2ms Activates the service in the specified project and AWS region
or2-manage-ftp...-a [mount, list, unmount] -b bucket_name or2ftp Manages S3 buckets in the specified project and AWS region
or2-ftp-access... -a [grant, list, revoke] -e user_email or2ftpa Manages user access to the FTP server
or2-describe-instances -S backup or2din Displays the details of VMs created during the service activation
or2-describe-services -s backup or2dser Describes the service activated in the specified project and region

Further on this page, you can find more details on the FTP to AWS S3 service manipulation.

Pre-Requisites and Limitations

The service can be described as a medium between users and AWS S3. For that reason, there is a number of pre-requisites to be met for the successful service usage:

  • The project should be activated in AWS and have S3 service in use
  • All bucket management operations - creating, removing and renaming buckets - should be performed by native AWS tools (Management Console and CLI)
  • The user to have the FTP access has to have the %maestro_cli%lib/default.cr file, created by their or2access command call

FTP2S3 Servcie Activation and Manipulation

To athe service, use the or2-manage-service (or2ms) command with the -a/--activate flag and the -s/--service-name parameter with backup value. The -k key_name parameter is also necessary, as for any instance in AWS:

or2ms -p project -r aws-region -a -s backup -k key_name

By default, FTP servers are SMALL-shaped VMs with Ubuntu14.04_64-bit operating system.

It is recommended to start the service in the region, in which the project S3 buckets are stored.

As soon as the service is activated, its data can be retrieved using the or2-describe-services (or2dser) command:

or2dser -p project -r region -s backup

You can find the details on the VMs created within the service activation, by calling the or2-describe-instances (or2din) command with -S backup parameter:

or2din -p project -r region -S backup

To deactivate the service, run the or2ms command with the -d/--deactivate flag:

or2ms -p project -r aws-region -d -s backup

User Access Management

By default, users cannot access the FTP server. For each user, the access should be granted explicitly. User access management is performed by means of the or2-ftp-access (or2ftpa) command:

  • To grant access to a user, run the command with the -a grant action:
or2ftpa -p project -r aws_region -a grant -e epam_user@epam.com
  • To see the list of the users to whom access is granted, run the command with the -a list action:
or2ftpa -p project -r aws_region -a list
  • To prohibit access to project S3 via FTP for a user to whom the access was granted previously, use the -a revoke action:
or2ftpa -p project -r aws_region -a revoke -e epam_user@epam.com

The users who have access to FTP can read, upload and delete from the buckets.

S3 Buckets Management

By default, the service does not provide access to any S3 buckets. They need to be registered, or mounted, on the FTP server. If necessary, the bucket can be deactivated on the server, so that FTP access to it gets restricted. These manipulations are performed with the or2-manage-ftp (or2ftp) command.

The command references bucket names that must correspond to the AWS naming standards and be the same as the name specified for the target AWS S3 bucket:

  • To register a new S3 bucket, run the command with the -a mount action. The bucket name must correspond to the AWS naming standards and be the same as the name specified for the target AWS S3 bucket:
or2ftp -p project -r aws_region -a mount -b demo-ftp-to-s3-bucket

To add several buckets, repeat the command.

  • To view the list of the buckets registered on the FTP server, run the command with the -a list action:
or2ftp -p project -r aws_region -a list
  • To restrict FTP access to an S3 bucket, use the -a unmount parameter:
or2ftp -p project -r aws_region -a unmount -b demo-ftp-to-s3-bucket

Please note that this action does not block or restrict the bucket itself. It will still be available by native AWS tools.

Accessing the FTP Server

The users can access the FTP server by any FTP client convenient for them. The following login details should be specified:

Parameter Value
Connection DNS or Public IP
User Name epam_user@epam.com
Password Token generated by Orchestrator at or2access command run and stored in the default.cr file
Port (default) 2121

You can find the server DNS and Port in the webUiURL column of the or2-describe-services (or2dser) command response:

When connected, the user can start working with the S3 storage. The buckets will be given as top-level directories:

Within each bucket, the users can create, change, and delete the necessary files.

Pricing

The price of the service consists of the price for the FTP Server instance (SMALL, Ubuntu14.04_64-bit), the price of the S3 service used, and the price for additional services, such as data transfer.

Each of these points depends on AWS pricing policy in each specific region.

EPAM Cloud does not apply any additional costs.

References

More information on the FTP2S3 service can be found in the EPAM Cloud Services Guide. For detailed description of the Maestro CLI commands used to manage the service, refer to the Maestro CLI User Guide.