iSCSI Gateway fails to start due to a timeout error when reading the "gateway.conf" object

This document (000019731) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 6

Situation

When the iSCSI Gateway service "rbd-target-api.service" is starting up, it encounters a timeout error when reading the "gateway.conf" object. The following type of error messages are captured in "/var/log/rbd-target-api/rbd-target-api.log" of the iSCSI Node:
 
  DEBUG [common.py:139:_open_ioctx()] - (_open_ioctx) Opening connection to iscsi-images pool
  DEBUG [common.py:146:_open_ioctx()] - (_open_ioctx) connection opened
  DEBUG [common.py:139:_open_ioctx()] - (_open_ioctx) Opening connection to iscsi-images pool
  DEBUG [common.py:146:_open_ioctx()] - (_open_ioctx) connection opened
  DEBUG [common.py:442:lock()] - config.lock attempting to acquire lock on gateway.conf
  DEBUG [common.py:450:lock()] - (Config.lock) waiting for excl lock on gateway.conf object
   ...
  DEBUG [common.py:450:lock()] - (Config.lock) waiting for excl lock on gateway.conf object
  ERROR [common.py:458:lock()] - (Config.lock) Timed out (30s) waiting for excl lock on gateway.conf object
  ERROR [rbd-target-api:2935:<module>()] - Timed out (30s) waiting for excl lock on gateway.conf object
  CRITICAL [rbd-target-api:2750:halt()] - Unable to open/read the configuration object

In the above example, the pool name used by the iSCSI gateway is "iscsi-images", note however that the pool name can be different in other iSCSI deployments.

Resolution

The "gateway.conf" object stored in the iSCSI pool is locked, and the lock must be broken by performing the below example steps (replace the "iscsi-images" pool with the appropriate pool name):

1. Get the lock information of the "gateway.conf" object from the iSCSI pool by running on the Admin Node:

     # rados -p iscsi-images lock list gateway.conf
     {"objname":"gateway.conf","locks":[{"name":"lock"}]}


     # rados -p iscsi-images lock info gateway.conf lock
     {"name":"lock","type":"exclusive","tag":"","lockers":[{"name":"client.4711","cookie":"config","description":"","expiration":"0.000000","addr":"192.168.202.32:0/21039112"}


2. Stop the iSCSI Gateway service on the iSCSI Node(s):
 
# systemctl stop rbd-target-api.service
     
3. Using the lock information of the "gateway.conf" object printed from step 1, break the lock for the "gateway.conf" object on the iSCSI pool:
 
# rados -p iscsi-images lock break gateway.conf lock client.4711 --lock-cookie config

4. Start the iSCSI Gateway service:

     # systemctl start rbd-target-api.service

5. If the iSCSI Gateway can now read the "gateway.conf" object, the "/var/log/rbd-target-api/rbd-target-api.log" should contain for example the following messages:

     DEBUG [common.py:139:_open_ioctx()] - (_open_ioctx) Opening connection to iscsi-images pool
     DEBUG [common.py:146:_open_ioctx()] - (_open_ioctx) connection opened
     DEBUG [common.py:430:init_config()] - (init_config) using pre existing config object
     DEBUG [common.py:139:_open_ioctx()] - (_open_ioctx) Opening connection to iscsi-images pool
     DEBUG [common.py:146:_open_ioctx()] - (_open_ioctx) connection opened
     DEBUG [common.py:118:_read_config_object()] - _read_config_object reading the config object

Cause

The "gateway.conf" object stored on the iSCSI pool is locked.

Status

Reported to Engineering

Additional Information

The pool storing the "gateway.conf" object is configured in the "/etc/ceph/iscsi-gateway.cfg" file located on the designated iSCSI node(s).

 

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000019731
  • Creation Date: 16-Oct-2020
  • Modified Date:16-Oct-2020
    • SUSE Enterprise Storage

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback@suse.com

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

Join Our Community

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.


SUSE Customer Support Quick Reference Guide SUSE Technical Support Handbook Update Advisories
Support FAQ

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.

Go to Customer Center