The popularity and adoption of object storage continues to grow, with more and more applications and databases using it or having an option to utilize it.
In this post, we’ll show you how to replicate object data from an on-premises Pure Storage® FlashBlade® Object Store bucket to an Oracle Cloud Infrastructure (OCI) Object Storage private bucket.
Potential Use Cases
There are several reasons why you may want to replicate on-premises data to a cloud object store, including:
- Data protection: Replicating Oracle RMAN backups to cloud object storage
- Data archiving: Replicating on-premises Oracle RMAN backups to cloud archive object storage
- Database cloud migration: Transferring on-premises data to cloud object storage for migration using the following, for example:
- SQL Developer Data Import Wizard
- SQL Developer Migration Workbench
- Oracle Data Pump
- Database links
- DBMS_CLOUD package
- External tables
- Data warehouse and data lake workloads: Transferring on-premises data for use with these solutions
- Data mobility: Replicating data for the creation of cloud development and test environments
Benefits of Object Storage
Using Oracle Object Storage as a staging area for data loading or data access has many benefits. These include:
- Simplifying architectures, as well as ELT/ETL, on object storage
- Providing the option to query in place from ADM
- Having object storage that is independent from the data processing engines
- Reducing storage costs
Setting Up Oracle Cloud Infrastructure (OCI)
To get started, if you don’t already have an OCI account, you may want to consider creating one. The OCI Always Free Services provide several free services, including 10 GB Standard Object Storage and 10 GB Infrequent Access Object Storage.
Create the OCI Bucket
Log in to Oracle Cloud Infrastructure. From the dashboard, select Object Storage, or use the hamburger menu to navigate to Storage > Object Storage > Buckets.
Select your required Compartment, and click Create Bucket.
Provide a Bucket Name, and confirm storage options. For this blog post, we’ll use the standard default storage tier. OCI also provides an archive storage tier for long-term storage, which may be of interest as it’s one-tenth of the cost of standard object storage.
Please note: You need to enable Object Versioning, as this is required for FlashBlade replication.
Note: Bucket names must be unique. If you’ve used the name in another compartment, you’ll get an error message.
By default, the bucket visibility is private. You can change the visibility by clicking on the three dots (also known as a snowman) on the right-hand side and selecting Public.
Customer Secret Key
Before we can replicate to the Oracle Cloud Infrastructure private bucket from the on-premises Pure Storage FlashBlade, we need to provide authentication.
To do this from the OCI dashboard, click on your profile icon in the upper right and navigate to User Settings > Customer Secret Keys. Click Generate Secret Key and provide a descriptive name.
Enter the secret key Name.
Once generated, you’ll need to select Copy. This will be the aws_secret_access_key we’ll use later.
Now, click on the newly created Secret Name to obtain the aws_access_key and save it for later.
Determine Object Storage Namespace Name
From the Oracle Cloud Infrastructure dashboard, click on your profile icon in the upper right and navigate to Tenancy.
The Object Storage Namespace and default S3 compartments should be visible.
Alternatively, you can use the OCI command line interface, for example:
% oci os ns get
{
"data": "<Object Storage Namespace>"
}
Test Oracle Cloud Infrastructure Object Storage URL
To test access to the OCI Object Storage Bucket, we can use the AWS s3api command line interface.
Update your local ~/.aws/credentials file with a profile and query the OCI Object Storage URL, for example:
aws s3api list-buckets --query "Buckets[].Name" --endpoint-url https://<Object Storage Namespace>.compat.objectstorage.uk-london-1.oraclecloud.com --profile oci
[
"FlashBlade"
]
The OCI Bucket URL format is https://<Object Storage Namespace>.compat.objectstorage.<region>.oraclecloud.com
A full list of OCI Object Storage endpoints is available in the OCI Documentation.
Pure Storage FlashBlade
Now let’s look at the FlashBlade side.
Create S3 Target Connection
- Log in to FlashBlade and navigate to Storage > S3 Target Connections.
- Click the + to the right of Connect S3 Target.
- Enter the Name and Address.
Add Remote S3 Credentials
- Navigate to Protection > Object Replication > Remote Credentials.
- Click the + to the right of Create Remote Credentials.
- Select the Oracle Cloud Infrastructure Target from the drop-down menu for Remote Array or Target.
- Enter a Name for the connection, for example: OCI-UK-London1
- Paste the OCI Access Key ID and Secret Key we created earlier.
- Click Create.
Set Up Bucket Replication to OCI Object Storage
Navigate to Protection > Object Replication > Bucket Replica Link.
Select the Local Bucket Name, Remote Array or Target, and Remote Credential. Then, provide the Oracle Cloud Infrastructure Remote Bucket name.
From FlashBlade, we can now see that we’ve established a bucket replica link.
Test Replication
To test the replication link, upload a test file to the FlashBlade bucket, for example:
aws s3api put-object --bucket oracle-bucket --key VagrantAPEX.mp4 --body /home/oracle/osbws/VagrantAPEX.mp4 --endpoint-url ${ENDPOINT} --profile default
Oracle Cloud Infrastructure Object Storage
Returning to OCI, navigate to OCI Object Storage > Bucket Details and confirm replication.
Summary
In this post, we covered some of the use cases and reasons why you may want to replicate on-premises data to the cloud. We also demonstrated how to replicate an on-premises Pure Storage FlashBlade Object Store bucket to an Oracle Cloud Infrastructure (OCI) Object Storage private bucket
