The popularity and adoption of object storage continues to grow, with more and more applications and databases using it or having an option to utilize it.

In this post, we’ll show you how to replicate object data from an on-premises Pure Storage® FlashBlade® Object Store bucket to an Oracle Cloud Infrastructure (OCI) Object Storage private bucket.

Potential Use Cases

There are several reasons why you may want to replicate on-premises data to a cloud object store, including:

  • Data protection: Replicating Oracle RMAN backups to cloud object storage
  • Data archiving: Replicating on-premises Oracle RMAN backups to cloud archive object storage
  • Database cloud migration: Transferring on-premises data to cloud object storage for migration using the following, for example:
    • SQL Developer Data Import Wizard
    • SQL Developer Migration Workbench
    • Oracle Data Pump
    • Database links
    • DBMS_CLOUD package
    • External tables
  • Data warehouse and data lake workloads: Transferring on-premises data for use with these solutions
  • Data mobility: Replicating data for the creation of cloud development and test environments

Benefits of Object Storage

Using Oracle Object Storage as a staging area for data loading or data access has many benefits. These include:

  • Simplifying architectures, as well as ELT/ETL, on object storage
  • Providing the option to query in place from ADM
  • Having object storage that is independent from the data processing engines
  • Reducing storage costs

Setting Up Oracle Cloud Infrastructure (OCI)

To get started, if you don’t already have an OCI account, you may want to consider creating one. The OCI Always Free Services provide several free services, including 10 GB Standard Object Storage and 10 GB Infrequent Access Object Storage.

Create the OCI Bucket

Log in to Oracle Cloud Infrastructure. From the dashboard, select Object Storage, or use the hamburger menu to navigate to Storage > Object Storage > Buckets.

Select your required Compartment, and click Create Bucket.

flashblade

Provide a Bucket Name, and confirm storage options. For this blog post, we’ll use the standard default storage tier. OCI also provides an archive storage tier for long-term storage, which may be of interest as it’s one-tenth of the cost of standard object storage.

Please note: You need to enable Object Versioning, as this is required for FlashBlade replication.

oracle cloud infrastructure

Note: Bucket names must be unique. If you’ve used the name in another compartment, you’ll get an error message.

By default, the bucket visibility is private. You can change the visibility by clicking on the three dots (also known as a snowman) on the right-hand side and selecting Public.

Customer Secret Key

Before we can replicate to the Oracle Cloud Infrastructure private bucket from the on-premises Pure Storage FlashBlade, we need to provide authentication.

To do this from the OCI dashboard, click on your profile icon in the upper right and navigate to User Settings > Customer Secret Keys. Click Generate Secret Key and provide a descriptive name.

flashblade

Enter the secret key Name.

flashblade

Once generated, you’ll need to select Copy. This will be the aws_secret_access_key we’ll use later.

flashblade

Now, click on the newly created Secret Name to obtain the aws_access_key and save it for later.

Determine Object Storage Namespace Name

From the Oracle Cloud Infrastructure dashboard, click on your profile icon in the upper right and navigate to Tenancy.

The Object Storage Namespace and default S3 compartments should be visible.

oracle cloud infrastructure

Alternatively, you can use the OCI command line interface, for example:
% oci os ns get
{
"data": "<Object Storage Namespace>"
}

Test Oracle Cloud Infrastructure Object Storage URL

To test access to the OCI Object Storage Bucket, we can use the AWS s3api command line interface.

Update your local ~/.aws/credentials file with a profile and query the OCI Object Storage URL, for example:

aws s3api list-buckets --query "Buckets[].Name" --endpoint-url https://<Object Storage Namespace>.compat.objectstorage.uk-london-1.oraclecloud.com  --profile oci

[

    "FlashBlade"

]

The OCI Bucket URL format is https://<Object Storage Namespace>.compat.objectstorage.<region>.oraclecloud.com

A full list of OCI Object Storage endpoints is available in the OCI Documentation.

Pure Storage FlashBlade

Now let’s look at the FlashBlade side.

Create S3 Target Connection

  • Log in to FlashBlade and navigate to Storage > S3 Target Connections.
  • Click the + to the right of Connect S3 Target.
  • Enter the Name and Address.

flashblade

Add Remote S3 Credentials

  • Navigate to Protection > Object Replication > Remote Credentials.
  • Click the + to the right of Create Remote Credentials.
  • Select the Oracle Cloud Infrastructure Target from the drop-down menu for Remote Array or Target.
  • Enter a Name for the connection, for example: OCI-UK-London1
  • Paste the OCI Access Key ID and Secret Key we created earlier.
  • Click Create.

oracle cloud infrastructure

Set Up Bucket Replication to OCI Object Storage

Navigate to Protection > Object Replication > Bucket Replica Link.

Select the Local Bucket Name, Remote Array or Target, and Remote Credential. Then, provide the Oracle Cloud Infrastructure Remote Bucket name.

From FlashBlade, we can now see that we’ve established a bucket replica link.

flashblade

Test Replication

To test the replication link, upload a test file to the FlashBlade bucket, for example:

aws s3api put-object --bucket oracle-bucket --key VagrantAPEX.mp4 --body /home/oracle/osbws/VagrantAPEX.mp4 --endpoint-url ${ENDPOINT} --profile default

flashblade

Oracle Cloud Infrastructure Object Storage

Returning to OCI, navigate to OCI Object Storage > Bucket Details and confirm replication.

oracle cloud infrastructure

Summary

In this post, we covered some of the use cases and reasons why you may want to replicate on-premises data to the cloud. We also demonstrated how to replicate an on-premises Pure Storage FlashBlade Object Store bucket to an Oracle Cloud Infrastructure (OCI) Object Storage private bucket