After adding services to cluster, dashboard shows error 404 when accessing object gateway

This document (000019596) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 6

Situation

After adding services to cluster (IGW, ISCSI, and NFS) dashboard gets error 500 when accessing previously working portion of the dashboard with Error 500 pop-up alert and Error 404 in iframe (see attached picture).

Confirmed manager settings with the following:

mgr-1:~ # ceph dashboard get-rgw-api-host rgw-1.foo.com
mgr-1:~ # ceph dashboard get-rgw-api-port 80
mgr-1:~ # ceph dashboard get-rgw-api-user-id admin
mgr-1:~ # ceph dashboard set-rgw-api-ssl-verify False Option RGW_API_SSL_VERIFY updated
mgr-1:~ # ceph dashboard get-rgw-api-access-key --removed--
mgr-1:~ # ceph dashboard get-rgw-api-secret-key --removed--

Potential issues could be:
- Expired, or missing certificate.
- rgw configured for wrong port.
- Incorrect DNS configuration for rgw.

Log File Records the following:

2020-03-16 09:52:04.934 7f74918bee40  0 ceph version 14.2.5-382-g8881d33957 (8881d33957b54b101eae9c7627b351af10e87ee8) nautilus (stable), process ceph-mgr, pid 1341
2020-03-16 09:52:04.934 7f74918bee40  0 pidfile_write: ignore empty --pid-file
2020-03-16 09:52:05.134 7f74918bee40  1 mgr[py] Loading python module 'ansible'
2020-03-16 09:52:06.597 7f74918bee40  1 mgr[py] Loading python module 'balancer'
2020-03-16 09:52:06.957 7f74918bee40  1 mgr[py] Loading python module 'crash'
2020-03-16 09:52:07.509 7f74918bee40  1 mgr[py] Loading python module 'dashboard'
2020-03-16 09:52:10.491 7f74918bee40  1 mgr[py] Loading python module 'deepsea'
2020-03-16 09:52:11.711 7f74918bee40  1 mgr[py] Loading python module devicehealth'
2020-03-16 09:52:12.075 7f74918bee40  1 mgr[py] Loading python module 'diskprediction_local'
2020-03-16 09:52:12.387 7f74918bee40  1 mgr[py] Loading python module 'influx'
2020-03-16 09:52:13.983 7f74918bee40  1 mgr[py] Loading python module 'insights'
2020-03-16 09:52:14.287 7f74918bee40  1 mgr[py] Loading python module 'iostat'
2020-03-16 09:52:14.615 7f74918bee40  1 mgr[py] Loading python module 'localpool'
2020-03-16 09:52:14.899 7f74918bee40  1 mgr[py] Loading python module 'orchestrator_cli'
2020-03-16 09:52:15.727 7f74918bee40  1 mgr[py] Loading python module 'pg_autoscaler'
2020-03-16 09:52:16.215 7f74918bee40  1 mgr[py] Loading python module 'progress'
2020-03-16 09:52:16.675 7f74918bee40  1 mgr[py] Loading python module 'prometheus'
2020-03-16 09:52:18.655 7f74918bee40  1 mgr[py] Loading python module 'rbd_support'
2020-03-16 09:52:18.859 7f74918bee40  1 mgr[py] Loading python module 'restful'
2020-03-16 09:52:19.835 7f74918bee40  1 mgr[py] Loading python module 'rook'
2020-03-16 09:52:23.295 7f74918bee40  1 mgr[py] Loading python module 'selftest'
2020-03-16 09:52:23.387 7f74918bee40  1 mgr[py] Loading python module 'status'
2020-03-16 09:52:23.519 7f74918bee40  1 mgr[py] Loading python module 'telegraf'
2020-03-16 09:52:23.643 7f74918bee40  1 mgr[py] Loading python module 'telemetry'
2020-03-16 09:52:24.287 7f74918bee40  1 mgr[py] Loading python module 'test_orchestrator'
2020-03-16 09:52:24.387 7f74918bee40  1 mgr[py] Loading python module 'volumes'
2020-03-16 09:52:24.499 7f74918bee40  1 mgr[py] Loading python module 'zabbix'
2020-03-16 09:52:24.567 7f747decb700  1 mgr load Constructed class from module: dashboard
2020-03-16 09:52:24.567 7f747decb700  1 mgr load Constructed class from module: prometheus
2020-03-16 09:52:24.571 7f747d6ca700  0 ms_deliver_dispatch: unhandled message 0x55d0d7468a00 mon_map magic: 0 v1 from mon.1 v2:192.168.6.72:3300/0
2020-03-16 09:52:24.703 7f7466298700  0 mgr[dashboard] [16/Mar/2020:09:52:24] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cheroot/server.py", line 1317, in start
    self.tick()
  File "/usr/lib/python3.6/site-packages/cheroot/server.py", line 1385, in tick
    s, ssl_env = self.ssl_adapter.wrap(s)
  File "/usr/lib/python3.6/site-packages/cheroot/ssl/builtin.py", line 75, in wrap
    server_side=True)
  File "/usr/lib64/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/usr/lib64/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib64/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

Resolution

Review rgw configuration in:
https://documentation.suse.com/ses/6/single-html/ses-admin/#ceph-rgw-https

The specific section is:
https://documentation.suse.com/ses/6/single-html/ses-admin/#ogw-ssl-advanced

also see:
https://documentation.suse.com/ses/6/single-html/ses-admin/#ds-custom-cephconf

Cause

Potential issues could be:
- Expired, or missing certificate.
- rgw configured for wrong port.
- Incorrect DNS configuration for rgw.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.