Migrating Data from AWS S3 to OCI Object Storage Using rclone (Step-by-Step)

-

 Migrating Data from AWS S3 to OCI Object Storage Using rclone (Step-by-Step)

Overview

In real-world enterprise environments, multi-cloud adoption is common. Organizations often need to migrate or synchronize data from AWS S3 to Oracle Cloud Infrastructure (OCI) Object Storage for reasons such as cost optimization, performance, compliance, or OCI-native service integration.

In this post, I demonstrate a secure, reproducible, and automation-friendly method to migrate data from AWS S3 to OCI Object Storage using rclone.

⚠️ This demonstration is recreated in a personal OCI tenancy using test data to avoid exposure of any customer or confidential information.

Oracle Products Used

  • Oracle Cloud Infrastructure
  • OCI Object Storage
  • OCI IAM (Customer Secret Keys)
  • OCI Regions & Namespace
  • OCI Compute / Cloud Shell (execution environment)

Additional Products Used

Amazon AWS Account

AWS S3 Object Storage


Why rclone?

rclone is an open-source tool that supports 40+ cloud storage providers, including AWS S3 and OCI Object Storage.

Key benefits:

  • CLI-based (automation friendly)
  • Supports encryption, checksum validation
  • Resume-safe for large datasets
  • Ideal for multi-cloud migrations


High-Level Architecture

Flow:

AWS S3 Bucket  →  rclone  →  OCI Object Storage Bucket

  • rclone authenticates to AWS using IAM credentials
  • rclone authenticates to OCI using Customer Secret Keys
  • Data is copied or synchronized securely over HTTPS

Prerequisites

AWS Side

  • An AWS S3 bucket with test data
  • IAM user with:
    • s3:ListBucket
    • s3:GetObject

OCI Side

  • OCI tenancy (Free Tier is sufficient)
  • Object Storage bucket
  • Customer Secret Key created for OCI user

Tools

  • rclone (installed on Linux / OCI Compute / Cloud Shell)

·         Step 1: Install rclone

·         On OCI Linux instance or Cloud Shell:

·         curl https://rclone.org/install.sh | sudo bash

·         Verify installation:

·         rclone version

·         Step 2: Configure AWS S3 in rclone

·         Run:

·         rclone config

n) New remote

name> aws-s3

Storage> s3

Provider> AWS

Access Key ID> <AWS_ACCESS_KEY>

Secret Access Key> <AWS_SECRET_KEY>

Region> ap-south-1

rclone ls aws-s3:my-test-bucket

Step 3: Configure OCI Object Storage in rclone

OCI Object Storage uses S3-compatible API.

rclone config

n) New remote

name> oci-os

Storage> s3

Provider> Other

Endpoint> https://<namespace>.compat.objectstorage.<region>.oraclecloud.com

Access Key ID> <OCI_ACCESS_KEY>

Secret Access Key> <OCI_SECRET_KEY>

Region> us-ashburn-1

Important Notes

  • Access/Secret keys are OCI Customer Secret Keys
  • Namespace is unique per tenancy

·         Step 4: Validate Both Ends

·         List AWS bucket:

·         rclone ls aws-s3:my-test-bucket

·         List OCI bucket:

·         rclone ls oci-os:oci-target-bucket

 Step 5: Perform the Migration

One-Time Copy

rclone copy aws-s3:my-test-bucket oci-os:oci-target-bucket \

  --progress \

  --transfers=8 \

  --checkers=8

Continuous Sync (Incremental)

rclone sync aws-s3:my-test-bucket oci-os:oci-target-bucket \

  --progress \

  --checksum

Step 6: Validate Data Integrity

rclone check aws-s3:my-test-bucket oci-os:oci-target-bucket

This ensures:

  • No missing files
  • No corrupted objects

Performance & Cost Considerations

  • OCI Object Storage ingress is free
  • AWS S3 egress costs apply
  • Parallel transfers improve throughput
  • Use --bwlimit if throttling is required

Common Issues & Troubleshooting

Issue

Resolution

Access denied

Verify IAM permissions

Slow transfer

Increase --transfers

Namespace error

Double-check OCI namespace

TLS errors

Ensure correct endpoint

Security Best Practices

  • Use least-privilege IAM policies
  • Rotate Customer Secret Keys
  • Avoid storing credentials in scripts
  • Use .rclone.conf file permissions (600)

Lessons Learned

  • OCI Object Storage S3-compatibility simplifies migrations
  • rclone is reliable for enterprise-scale transfers
  • Automation enables repeatable multi-cloud strategies

Use Cases

  • AWS → OCI workload migration
  • Hybrid backup strategy
  • Cost-optimized archival
  • DR copy of critical datasets

Conclusion

This demonstration shows how Oracle Cloud Infrastructure Object Storage can seamlessly integrate into multi-cloud architectures using open-source tooling like rclone.

The approach is secure, scalable, and automation-friendly, making it suitable for both proof-of-concept and enterprise-grade deployments.

References

  • rclone documentation
  • OCI Object Storage documentation
  • AWS S3 documentation

·         About the Author

·         Debapriya Biswas
Oracle ACE Apprentice | Sr. Consultant – Cloud Technologies

·  

 

Comments

Post a Comment

Popular posts from this blog

Access Oracle OCI Object Storage through GUI Client

-
Image
 As we already know that there are several GUI clients available for Amazon AWS S3 Object storage like - S3 Browser (https://s3browser.com/), CloudBerry Explorer (https://www.msp360.com/explorer/windows/amazon-s3.aspx), CyberDuck (https://cyberduck.io/) etc. Using these tools we can easily connect to the Object Storage service, upload & download "objects" i.e files and folders and also do some advance management like Bucket Policies, Replication, Versioning, lifecycle policies etc. Now these tools can also be used with Oracle OCI Object storage, but there are some prerequisites to get the proper credentials. You need to get following information before you can connect to you OCI Object storage - Object Storage Namespace: REST Endpoint: Access Key: Secret Key ID: How to get these details - Login to your OCI Account, head over to - Hamburger Icon on left hand top corner -> Storage -> Under Object Storage & Archive -> Buckets. It will list all the buckets in thi...
Read more

Instance OS Baseline Configuration Runbook

-
Overview & Scope This runbook defines the standard baseline configuration procedure for new OCI compute instances across five supported operating system families. It is intended for use by Cloud SRE and operations personnel during initial instance provisioning. Objectives Each OS configuration procedure achieves the following baseline state: •        Set system timezone to Asia/Kolkata (IST, UTC+05:30) •        Apply all available OS security and feature updates •        Enable and configure the host-based firewall to permit SSH (Linux) or RDP (Windows) ingress exclusively from the VCN CIDR block •        Enable EPEL repository where applicable (Oracle Linux, Rocky Linux, Alma Linux) •        Create a named administrative user account separate from the default OCI-provisioned user (opc / Administrator) •   ...
Read more

Accessing OCI Compute Instances Using VNC Console (Instance Console Connection)

-
Image
  Overview In Oracle Cloud Infrastructure (OCI), administrators occasionally face scenarios where SSH access to a compute instance is unavailable —for example, due to network misconfiguration, firewall issues, or OS-level boot problems. To handle such situations, OCI provides Instance Console Connection , which enables VNC-based graphical console access and serial console access to compute instances. In this post, I demonstrate how to access an OCI Compute instance using the VNC console , explain when to use it, and highlight best practices for secure and effective troubleshooting. ⚠️ This demonstration is performed in a personal OCI tenancy using test instances. No customer environments or confidential information are exposed. Oracle Products Used Oracle Cloud Infrastructure OCI Compute Instance Console Connection OCI IAM Virtual Cloud Network (VCN) What Is Instance Console Connection? Instance Console Connection provides ou...
Read more