S3 Cross-Account Replication — Resolving Access Denied

Understand object ownership across AWS accounts, reproduce Access Denied, and resolve it using ownership controls and permission overrides. · 30 min

Open Table of contents

0. Prerequisites
1. Building the lab environment
- 1.1. Create the source bucket
- 1.2. Create the destination bucket
2. Configuring replication permissions
- 2.1. Create the IAM role for replication
- 2.2. Register the destination bucket policy
3. Registering the replication rule and reproducing Access Denied
Compare the first 4 characters of Owner.ID with the source account (crr-source) canonical user ID on your record sheet.
4. Applying the ownership override
5. Checking the results
6. Cleaning up resources

0. Prerequisites

Here is what you need.

Item	Details
AWS account	1, used as the management account. The two lab accounts are created in section 0.1
Email	2 different addresses, one for each of the two lab accounts
Browser	1, move between accounts by switching roles
Regions	Source `ap-northeast-2`, destination `us-east-1`
Local tools	AWS CLI v2, a text editor

Note In every command that follows, put your own bucket prefix in place of [your prefix], and the account numbers of the accounts created in section 0.1 in place of [source account 12 digits] and [destination account 12 digits].

0.1. Create an organization and two lab accounts

Type Organizations in the search bar and click AWS Organizations in the results.
If you have no organization yet, click Create an organization.
In the left menu, click AWS accounts.
Click Add an AWS account at the top right.
Select [Create an AWS account].
In the AWS account name field, enter crr-source.
In the Email address of the account’s owner field, enter [your Gmail local part][email protected].
Leave the IAM role name field at its default value, OrganizationAccountAccessRole.
Click Create AWS account at the bottom right.
Repeat the same procedure to create the crr-dest account. Change the email to use +crr-dest.

Note Account creation is handled asynchronously, so it can take a few minutes. Once it finishes, check the 12-digit numbers of the two accounts (crr-source, crr-dest) in the AWS accounts list.

0.2. Create a lab IAM user in the management account

Type IAM in the AWS Console search bar and click IAM in the results.
In the left menu, click Users.
Click Create user at the top right.
In the User name field, enter crr-lab-admin.
Click the Provide user access to the AWS Management Console checkbox.
Select [I want to create an IAM user].
Under Console password, select [Custom password] and enter a password.
Clear the Users must create a new password at next sign-in checkbox and click Next.
Under Permissions options, select [Attach policies directly].
In the policy list, search for AdministratorAccess, click the checkbox to its left, and click Next.
Click Create user.
In the user list, click crr-lab-admin.
Click the Security credentials tab.
Under Access keys, click Create access key.
Under Use case, select [Command line interface (CLI)], click the confirmation checkbox below it, then click Next.
Click Create access key.
Record the Access key and the Secret access key, then click Done.

⚠️Warning The secret access key can only be seen on this screen. Once you close the window, you cannot see it again. If you missed it, delete that access key and create a new one.

0.3. Prepare console role switching

In the account menu at the top right, click Add Sessions.
On the sign-in screen, select [IAM user].
In the Account ID field, enter the 12 digits of the management account.
Enter crr-lab-admin in the IAM user name field and the password you set in section 0.2 in the Password field, then click Sign in.
In the account menu at the top right, click Switch role.
In the Account field, enter [source account 12 digits]. (Check the 12 digits of crr-source in AWS Organizations.)
In the IAM role name field, enter OrganizationAccountAccessRole.
In the Display name field, enter crr-source and pick a color.
Click Switch Role.
In the account menu, click Back to crr-lab-admin.
Register [destination account 12 digits] the same way, under the name crr-dest.

Note From here on, Working session: source account means you have switched to crr-source, Working session: destination account means you have switched to crr-dest, and Working session: management account means you have left the switched role and returned to crr-lab-admin. Switching takes a single click from the history in the account menu.

0.4. Register CLI profiles

Check the AWS CLI version

aws --version

Create the management account CLI profile

aws configure --profile crr-lab-admin

Values for the interactive prompts

Item	Value
AWS Access Key ID	`[management account access key ID]`
AWS Secret Access Key	`[management account secret access key]`
Default region name	`ap-northeast-2`
Default output format	`json`

Register the source account profile

aws configure set role_arn arn:aws:iam::[source account 12 digits]:role/OrganizationAccountAccessRole --profile crr-source-admin
aws configure set source_profile crr-lab-admin --profile crr-source-admin
aws configure set region ap-northeast-2 --profile crr-source-admin

Register the destination account profile

aws configure set role_arn arn:aws:iam::[destination account 12 digits]:role/OrganizationAccountAccessRole --profile crr-dest-admin
aws configure set source_profile crr-lab-admin --profile crr-dest-admin
aws configure set region us-east-1 --profile crr-dest-admin

Check the list of registered CLI profiles

aws configure list-profiles

Note You are good if crr-lab-admin, crr-source-admin, and crr-dest-admin appear in the list. Any other names are profiles you were already using, so they differ from person to person.

0.5. Verify the account values

Check which accounts the two profiles point to

aws sts get-caller-identity --profile crr-source-admin
aws sts get-caller-identity --profile crr-dest-admin

Check that the Account value from both commands matches the account numbers on your record sheet. Check the canonical user ID of both accounts

aws s3api list-buckets --query Owner --profile crr-source-admin
aws s3api list-buckets --query Owner --profile crr-dest-admin

Note the first 4 characters of the ID from both commands on your record sheet.

1. Building the lab environment

1.1. Create the source bucket

In the region selector at the top right, select [Asia Pacific (Seoul) ap-northeast-2].
Type S3 in the search bar and click S3 in the results.
In the left menu, click General purpose buckets.
Click Create bucket at the top right.
In the Bucket name field, enter [your prefix]-crr-source. e.g. dev-playbuilder-crr-source
Under Bucket Versioning, select [Enable].
Leave everything else at its default value and click Create bucket at the very bottom. Check the versioning status of the source bucket

aws s3api get-bucket-versioning --bucket [your prefix]-crr-source --profile crr-source-admin

1.2. Create the destination bucket

In the region selector at the top right, select [US East (N. Virginia) us-east-1].
Type S3 in the search bar and click S3 in the results.
In the left menu, click General purpose buckets.
Click Create bucket at the top right.
In the Bucket name field, enter [your prefix]-crr-dest.
Under Object Ownership, select [ACLs enabled].
From the options that appear below, select [Object writer].
Click the confirmation checkbox.
Under Bucket Versioning, select [Enable].
Leave everything else at its default value and click Create bucket at the very bottom.

Warning You must select [ACLs enabled] and [Object writer] under Object Ownership. If you leave it at the default [ACLs disabled], the result in section 3.3 will be different. Check versioning and Object Ownership on the destination bucket

aws s3api get-bucket-versioning --bucket [your prefix]-crr-dest --profile crr-dest-admin
aws s3api get-bucket-ownership-controls --bucket [your prefix]-crr-dest --profile crr-dest-admin

2. Configuring replication permissions

2.1. Create the IAM role for replication

Create the trust policy file

cat > crr-trust-policy.json << 'EOF'
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF

Create the permissions policy file for replication

cat > crr-role-policy.json << 'EOF'
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": "arn:aws:s3:::[your prefix]-crr-source/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetReplicationConfiguration"
            ],
            "Resource": "arn:aws:s3:::[your prefix]-crr-source"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags"
            ],
            "Resource": "arn:aws:s3:::[your prefix]-crr-dest/*"
        }
    ]
}
EOF

Note Before creating the file above, replace [your prefix] with your own bucket name. You can also create the file in an editor and save it. Create the replication IAM role and attach the permissions policy

aws iam create-role --role-name crr-replication-role --assume-role-policy-document file://crr-trust-policy.json --profile crr-source-admin
aws iam put-role-policy --role-name crr-replication-role --policy-name crr-replication-policy --policy-document file://crr-role-policy.json --profile crr-source-admin

Write down the Arn value from the first command on your record sheet.

2.2. Register the destination bucket policy

In the General purpose buckets list, click [your prefix]-crr-dest.
Click the Permissions tab.
Click Edit to the right of Bucket policy.
Paste the content below into the Policy editor.
Replace [source account 12 digits] and [your prefix] with your own values.
Click Save changes at the very bottom. Destination bucket policy — no ownership override

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowReplicationFromSourceAccount",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::[source account 12 digits]:role/crr-replication-role"
            },
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": "arn:aws:s3:::[your prefix]-crr-dest/*"
        }
    ]
}

3. Registering the replication rule and reproducing Access Denied

3.1. Register the replication rule

In the General purpose buckets list, click [your prefix]-crr-source.
Click the Management tab.
Under Replication rules, click Create replication rule.
In the Replication rule name field, enter crr-ownership-lab.
Under Status, select [Enabled].
Under Choose a rule scope, select [Apply to all objects in the bucket].
Under Destination, select [Specify a bucket in another account].
In the Account ID field, enter [destination account 12 digits].
In the Bucket name field, enter [your prefix]-crr-dest.
Leave the Change object ownership to destination bucket owner checkbox unselected.
Under IAM role, select [Choose from existing IAM roles] and choose crr-replication-role.
Leave everything else at its default value and click Save at the very bottom.
If you are asked whether to replicate existing objects, select [No, do not replicate existing objects]. Check the replication rule registered on the source bucket

aws s3api get-bucket-replication --bucket [your prefix]-crr-source --profile crr-source-admin

3.2. Upload a file and check replication

Create and upload the first lab file

echo "before override" > before-override.txt
aws s3 cp before-override.txt s3://[your prefix]-crr-source/before-override.txt --profile crr-source-admin

Check the replication status of the uploaded object

aws s3api head-object --bucket [your prefix]-crr-source --key before-override.txt --profile crr-source-admin

Note If ReplicationStatus is PENDING, run the same command again after a moment. Once it shows COMPLETED, move on to the next section.

3.3. Try reading from the destination account

List the replicated objects from the destination account

aws s3api list-objects-v2 --bucket [your prefix]-crr-dest --profile crr-dest-admin

Try downloading the replicated object from the destination account

aws s3api get-object --bucket [your prefix]-crr-dest --key before-override.txt ./dest-before.txt --profile crr-dest-admin

Warning The second command in this section is the step where you confirm the AccessDenied error.

3.4. Check the object owner

Check the owner of the replicated object

aws s3api list-objects-v2 --bucket [your prefix]-crr-dest --fetch-owner --profile crr-dest-admin

Compare the first 4 characters of `Owner.ID` with the source account (crr-source) canonical user ID on your record sheet.

4. Applying the ownership override

4.1. Add the permission to the destination bucket policy

In the General purpose buckets list, click [your prefix]-crr-dest.
Click the Permissions tab.
Click Edit to the right of Bucket policy.
Replace the contents of the Policy editor with the following.
Replace [source account 12 digits] and [your prefix] with your own values.
Click Save changes at the very bottom. Destination bucket policy — with ownership override

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowReplicationFromSourceAccount",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::[source account 12 digits]:role/crr-replication-role"
            },
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:GetObjectVersionTagging",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Resource": "arn:aws:s3:::[your prefix]-crr-dest/*"
        }
    ]
}

4.2. Add the permission to the replication role policy

Create the permissions policy file that includes the ownership override

cat > crr-role-policy-override.json << 'EOF'
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging"
            ],
            "Resource": "arn:aws:s3:::[your prefix]-crr-source/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetReplicationConfiguration"
            ],
            "Resource": "arn:aws:s3:::[your prefix]-crr-source"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Resource": "arn:aws:s3:::[your prefix]-crr-dest/*"
        }
    ]
}
EOF

Replace the permissions policy on the replication role

aws iam put-role-policy --role-name crr-replication-role --policy-name crr-replication-policy --policy-document file://crr-role-policy-override.json --profile crr-source-admin

Check the replaced permissions policy

aws iam get-role-policy --role-name crr-replication-role --policy-name crr-replication-policy --query "PolicyDocument.Statement[2]" --profile crr-source-admin

4.3. Turn on the ownership override in the replication rule

In the General purpose buckets list, click [your prefix]-crr-source.
Click the Management tab.
In the Replication rules list, click the radio button to the left of crr-ownership-lab.
Click Edit rule at the top right.
Under Destination, click the Change object ownership to destination bucket owner checkbox.
Check that [destination account 12 digits] is filled in the Destination bucket owner account ID field.
Click Save at the very bottom. Check that the ownership override made it into the replication rule

aws s3api get-bucket-replication --bucket [your prefix]-crr-source --profile crr-source-admin

5. Checking the results

5.1. Upload a new file and replicate it

Create and upload the second lab file

echo "after override" > after-override.txt
aws s3 cp after-override.txt s3://[your prefix]-crr-source/after-override.txt --profile crr-source-admin

Check the replication status of the second object

aws s3api head-object --bucket [your prefix]-crr-source --key after-override.txt --profile crr-source-admin

Note If ReplicationStatus is PENDING, run the same command again after a moment. Once it shows COMPLETED, move on to the next section.

5.2. Compare owners and read the object from the destination account

Compare the owners of the two objects from the destination account

aws s3api list-objects-v2 --bucket [your prefix]-crr-dest --fetch-owner --profile crr-dest-admin

Compare the first 4 characters of each object’s Owner.ID with the two canonical user IDs on your record sheet. Download the second object from the destination account

aws s3api get-object --bucket [your prefix]-crr-dest --key after-override.txt ./dest-after.txt --profile crr-dest-admin

5.3. Check the state of the earlier object

Retry downloading the first object from the destination account

aws s3api get-object --bucket [your prefix]-crr-dest --key before-override.txt ./dest-before.txt --profile crr-dest-admin

Warning This command confirms the same result as in section 3.3.

6. Cleaning up resources

6.1. Delete the replication rule and the IAM role

Delete the replication rule and the replication IAM role

aws s3api delete-bucket-replication --bucket [your prefix]-crr-source --profile crr-source-admin
aws iam delete-role-policy --role-name crr-replication-role --policy-name crr-replication-policy --profile crr-source-admin
aws iam delete-role --role-name crr-replication-role --profile crr-source-admin

6.2. Delete every object version in the buckets

Delete every object version in the source bucket

aws s3api delete-objects --bucket [your prefix]-crr-source --delete "$(aws s3api list-object-versions --bucket [your prefix]-crr-source --output json --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' --profile crr-source-admin)" --profile crr-source-admin

Delete every object version in the destination bucket

aws s3api delete-objects --bucket [your prefix]-crr-dest --delete "$(aws s3api list-object-versions --bucket [your prefix]-crr-dest --output json --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}' --profile crr-dest-admin)" --profile crr-dest-admin

Note If the Deleted array from the two commands contains before-override.txt and after-override.txt respectively, the buckets are empty.

6.3. Delete the buckets

Delete the two buckets

aws s3api delete-bucket --bucket [your prefix]-crr-source --profile crr-source-admin
aws s3api delete-bucket --bucket [your prefix]-crr-dest --profile crr-dest-admin

6.4. Verify the deletion

Check whether any lab buckets remain in either account

aws s3api list-buckets --query "Buckets[?starts_with(Name, '[your prefix]-crr')].Name" --profile crr-source-admin
aws s3api list-buckets --query "Buckets[?starts_with(Name, '[your prefix]-crr')].Name" --profile crr-dest-admin

Check whether the replication IAM role remains

aws iam list-roles --query "Roles[?RoleName=='crr-replication-role'].RoleName" --profile crr-source-admin

Note If all three commands return an empty array, the cleanup is done. If a name is still listed, go back to section 6.1 or 6.2 and delete just that resource again. Delete the lab files created locally

rm -f crr-trust-policy.json crr-role-policy.json crr-role-policy-override.json before-override.txt after-override.txt dest-after.txt

6.5. Delete the CLI profiles and the management account user

Open the ~/.aws/credentials file in an editor.
Delete the block that starts with [crr-lab-admin].
Open the ~/.aws/config file in an editor.
Delete each of the blocks that start with [profile crr-lab-admin], [profile crr-source-admin], and [profile crr-dest-admin].
Save both files. Check the remaining CLI profiles

aws configure list-profiles

Note The cleanup is done when crr-lab-admin, crr-source-admin, and crr-dest-admin are no longer in the list. Sign in to the management account as root.

In your browser, click the management account root tab.
Type IAM in the search bar and click IAM in the results.
In the left menu, click Users.
Click the checkbox to the left of crr-lab-admin.
Click Delete at the top right.
Enter crr-lab-admin in the confirmation field and click Delete.

Note This is the only step done in the console. An IAM user cannot delete itself with its own access key. There are no users to delete in the two lab accounts.

6.6. Clean up the lab accounts

If you plan to use the lab accounts in the next chapter, skip this section. If you completed everything through section 6.4, no resources remain in the accounts. Sign in to the management account as root.

Type Organizations in the search bar and click AWS Organizations in the results.
In the left menu, click AWS accounts.
Click the checkbox to the left of crr-source.
Click Close at the top right.
Read the notice, enter the 12 digits of the source account in the confirmation field, then click Close account.
Repeat the same procedure to close the crr-dest account.

Warning A closed account stays in the Suspended state for 90 days and is then permanently terminated. After 90 days it cannot be undone. If you plan to create an account with the same email again, skip this section.

That is the end of this lab. Nice work.