Inability to attach/detach vSphere CNS block volumes

This document (000021286) is provided subject to the disclaimer at the end of this document.

Environment

  • Rancher 2.6 / 2.7
  • RKE1/RKE2
  • Kubernetes v1.19+
  • vSphere 6.7 U3+ or vSphere 7.0+
  • Vsphere cloud provider:
    • Vsphere CPI: rancher-vsphere-cpi:100.3.0+up1.2.1+
    • Vsphere CSI: rancher-vsphere-csi:100.3.0+up2.5.1-rancher1+

Situation

 

Inability to detach/attach CNS block volumes:

Customers can create CNS block volumes in the RKE1/RKE2 cluster using the Vsphere CSI.

However, when scaling down a workload (deployment,statefulset), the block volume does not get detached automatically from the nodes. Scaling up the workload, the following error appears in the cluster events:
rpc error: code = Internal desc = queryVolume failed for volumeID: "5db7cc3c-62b9-427d-823b-87729fcef771" with err=ServerFaultCode: NoPermission 

 

Resolution

This error indicates the user is missing the permission "Cns.Searchable" at the root vCenter level and Datastore level.

To grant the user account the Cns.Searchable permission in vSphere, see the following documentation:
https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials

 

Cause

The user account within vSphere must be granted the following permissions:
https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.security.doc/GUID-C04F1605-D158-4B65-810F-6F5B109BCDEC.html

 
 
 

Additional Information

vSphere CNS Block volumes:

Cloud Native Storage (CNS) integrates vSphere and Kubernetes and offers capabilities to create and manage container volumes in vSphere environment. CNS consists of the two components, CNS component in vCenter Server and a vSphere volume driver in Kubernetes, called vSphere Container Storage Plug-in.

vSphere Cloud Provider Interface (CPI):

Is responsible for running all the platform-specific control loops that were previously run in core Kubernetes components like the KCM and the kubelet, but have been moved out-of-tree to allow cloud and infrastructure providers to implement integrations that can be developed, built, and released independent of Kubernetes core

vSphere Container Storage Interface (CSI):

It is a specification designed to enable persistent storage volume management on Container Orchestrators (COs) such as Kubernetes. The specification allows storage systems to integrate with containerized workloads running on Kubernetes. Using CSI, storage providers, such as VMware, can write and deploy plugins for storage systems in Kubernetes without a need to modify any core Kubernetes code.

CSI allows volume plugins to be installed on Kubernetes clusters as extensions. Once a CSI-compatible volume driver is deployed on a Kubernetes cluster, users can use the CSI to provision, attach, mount, and format the volumes exposed by the CSI driver.

The CSI driver for vSphere is csi.vsphere.vmware.com.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000021286
  • Creation Date: 04-Dec-2023
  • Modified Date:04-Dec-2023
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center