s3cmd returns: An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: Unknown

This document (000019598) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 6

Situation

Customer created user "rgw-user" in ceph dashboard/Object Gateway/Users. He copied the access_key and secret_key and tried to utilize the s3cmd/aws-cli command to connect to the radosgw.

Upon configuring s3 connection configuration file with the copied keys, the connection does not allow him to create a bucket per the standard mechanism. It provides an error stating "An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: Unknown."

Customer verified the user with radosgw-admin command.

mon-1:~ # radosgw-admin sync status --rgw-zone=mt-local --rgw-zonegroup=mt
realm 72b0ef15-4035-46b0-8b59-da800c1461d4 (research) zonegroup cbb0ba91-d7fa-43e7-8950-46708eded129 (mt) zone dcebd321-b0e1-45fe-828f-d727bdd993f5 (mt-local)
metadata sync syncing
full sync: 0/64 shards incremental
sync: 64/64 shards metadata is behind on 16 shards behind shards: [0,1,5,12,16,17,23,28,32,33,37,44,48,49,53,60]
oldest incremental change not applied: 2020-03-05 12:39:19.0.473331s

mon-1:~ # s3cmd -c .s3cfg_201_rgw-user mb s3://rgw-bucket
An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: Unknown

Resolution

After restarting the RGWs in the primary zone, this no longer gives the error reported.

mon-1:~ # radosgw-admin sync status --rgw-zone=bd-local --rgw-zonegroup=bd
realm 72b0ef15-4035-46b0-8b59-da800c1461d4 (research)
zonegroup a9a33ee3-a6be-4562-a988-0a8a1696b663 (bd)
zone ae7a85e5-6963-49af-9876-c25b8547c0cc (bd-local)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master

mon-1:~ # s3cmd -c .s3cfg_201_rgw-user mb s3://rgw-bucket
Bucket 's3://rgw-bucket/' created
mon-1:~ # s3cmd -c .s3cfg_201_rjhall ls
2020-03-20 22:46 s3://rgw-bucket

Cause

This issue was caused by RGW synchronization stalled.
Since the accounts were not replicated, they stayed in a not accessible/created state.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.