How to migrate SAP BTP – Amazon Web Services(AWS) S3 Bucket files from one system to other system

SAP BTP Object Store as a cloud storage for Store and manage the blobs/objects on SAP BTP ,which involves creation, upload, download, and deletion. This service is specific to the IaaS layer such as Azure Blob Storage, Amazon Web Services(AWS), and Google Cloud Platform.

In this blog post I would like to focus on AWS S3 Bucket

In AWS S3, bucket is a container for objects. An object is a file and any metadata that describes that file.

To store an object in Amazon S3, you create a bucket and then upload the object to the bucket. When the object is in the bucket, you can open it, download it, and move it. When you no longer need an object or a bucket, you can clean up your resources.

I assume you already configured Object Store Instance in your BTP CF Space to use AWS Storage Service

Option 1 : Either Create Object Store Instance in your BTP CF space either using cockpit or CLI

Configure Object Store to use Amazon Simple Storage Service

Option 2 :  In MTA Yaml  add this Object Store resource as a dependency. When application deployed the AWS Object Store service instance will be created automatically.

In this blog post I would like share “How to migrate AWS S3 Bucket files from one system to other

Documents are hosted within a S3 Bucket on AWS. To Migrate objects from one S3 bucket to another S3 Bucket or Other Storage follow these options:

1) Using AWS Command Line Interface (AWS CLI) for platform (Mac/ Linux/ Windows)

2)  Programmatically

3) Migration software

   What is AWS CLI  you can read here https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html

Steps

  1. Install the AWS CLI [Instructions here]
  2. Verify Installation by executing below command from command prompt /Terminal
  • aws –version

3. configure the AWS CLI

  • aws configure

Enter your access keys (access key ID and secret access key).

Press Enter to skip the default Region and default output options.

  For more information about Amazon S3 Region parameters, see AWS service endpoints.

“credentials”: {

“access_key_id”: “< Access Key ID of the AWS IAM technical user created>”,

“bucket”: “<  Name of the AWS S3 bucket provisioned>”,

“host”: “<region_specific_s3_endpoint>”,

“region”: “< Region in which the AWS S3 bucket is provisioned>”,

“secret_access_key”: “< Secret Key ID of the AWS IAM technical user created>”,

“uri”: “s3://<some_access_key_id>:<some_secret_access_key>@<region_specific_s3_endpoint>/<some_bucket_name>”,

“username”: AWS IAM user name”

}

  1. Download files from S3 Bucket to Local Folder of your machine
  • C:\Yourfolder>aws s3 sync s3://EXAMPLE-BUCKET-SOURCE/ .
  1. Upload from Local directory to BTP AWS S3bucket

#1) As you downloaded to files to your local machine then you can copy/migrate to any system as per your requirement you can write utility program

#2) If you have requirement to upload or migrate files to other BTP space or sub-account AWS S3 bucket, here you need to configure other bucket details just like step -3, in new command prompt (i.e open new session form command prompt )

  • C:\Yourfolder>aws configure
  • C:\Yourfolder>aws s3 sync . s3://EXAMPLE-BUCKET-TARGET

Here source can be bucket name or local folder (.)

Note : You can also try to Copy the objects between the source and target buckets by running the following sync command using the AWS CLI:

aws s3 sync s3://DOC-EXAMPLE-BUCKET-SOURCE s3://DOC-EXAMPLE-BUCKET-TARGET

Note: Update the sync command to include your source and target bucket names.

The sync command uses the CopyObject APIs to copy objects between S3 buckets.

6) To List (Display File details on CLI) use below command

  • aws s3 ls s3://your-bucket-id

 

More details : https://aws.amazon.com/premiumsupport/knowledge-center/move-objects-s3-bucket/

Recommendations from AWS as below

Source: https://aws.amazon.com/premiumsupport/knowledge-center/move-objects-s3-bucket/

Limitations  : Using the aws s3 ls or aws s3 sync commands (Option # AWS CLI) on large buckets (with 10 million objects or more) can be expensive, resulting in a timeout. If you encounter timeouts because of a large bucket, then consider using Amazon CloudWatch metrics to calculate the size and number of objects in a bucket. Also, consider using S3 Batch Operations to copy the objects.

This option is suitable If you have many objects in your S3 bucket (more than 10 million objects), then consider using S3 Batch Operations. You can use S3 Batch Operations to automate the copy process. You can use the following JAVA sample code snippets as a reference to Download an object from S3 Bucket

https://help.sap.com/viewer/2ee77ef7ea4648f9ab2c54ee3aef0a29/Cloud/en-US/32517ae707c44ad48f635ea6bcbe271a.html

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html

For more information, see the documentation on the Amazon Web Services Web site.

You can choose your comfortable programming language of AWS SDK & connect S3 Bucket ,write code to download files from source S3 Bucket & copy to Destination as per your requirement