403 Forbidden: "system:unauthenticated" error message when using a cluster-scoped Rancher API Token on a different cluster

A Rancher v2.x instance, managing multiple clusters, and a cluster-scoped Rancher API token

When you try to access a Rancher-managed Kubernetes cluster using kubectl, the command fails with a 403 Forbidden error. The error message indicates an authentication failure from User "system:unauthenticated":

User \\\"system:unauthenticated\\\" cannot get resource \\\"clusters\\\" in API group \\\"management.cattle.io\\\" at the cluster scope

This typically happens when you use a kubeconfig file downloaded from one cluster (Cluster A) to access a different cluster (Cluster B), where Cluster A has the Authorized Cluster Endpoint (ACE) enabled.

This error is caused by attempting to use a cluster-scoped Rancher API token to query the Kubernetes API endpoint of a different cluster. In order to mitigate this, use a cluster-scoped Rancher API token that is scoped to the correct cluster, if you are connecting to the Authorized Cluster Endpoint; or alternatively, if you are not connecting to the Authorized Cluster Endpoint, you can use a non-scoped token.

This most regularly occurs, where a user downloads a kubeconfig file from Rancher, for a cluster with Authorized Cluster Endpoint enabled, and then attempts to use the token from that kubeconfig to connect to the Kubernetes API endpoint of a different cluster.

There is a difference between the token used in a Rancher-generated kubeconfig for clusters with and without Authorized Cluster Endpoint enabled. Here's a breakdown of the two kubeconfig types:

Authorized Cluster Endpoint Disabled (Default)

• API Server: The kubeconfig points to the Rancher-proxied Kubernetes API endpoint of the cluster.

• Authentication: The token is a non-scoped Rancher API token. Rancher acts as a proxy, authenticating the request and then forwarding it to Kubernetes API endpoint of the cluster.

• Portability: You can use the non-scoped token from this kubeconfig to connect to the Rancher-proxied Kubernetes API endpoint of any cluster on which you are granted permissions.

Authorized Cluster Endpoint Enabled

• API Server: The kubeconfig contains the Authorized Cluster Endpoint address, in addition to the Rancher-proxied Kubernetes API endpoint of the cluster.

• Authentication: The token is a cluster-scoped API token created specifically for your user on that individual cluster. This token is propagated to the downstream cluster, to enable its use when accessing the Authorized Cluster Endpoint, which bypasses the Rancher-proxied endpoint.

• Portability: The token in this kubeconfig is valid only for the Kubernetes API server of the cluster it was generated for. Using it against any other cluster will result in an system:unauthenticated error.