SUSE Multi-Linux Manager Server migration fails due to unknown (or lost) password for CA

SUSE Multi-Linux Manager 5.1 Server
SUSE Multi-Linux Manager 5.0 Server

SUSE Multi-Linux Manager Server 5.1 migration from SUSE Manager Server 4.3 or SUSE Multi-Linux Manager Server 5.0 fails due to lost (or unknown) password for CA used on top of the source system. Excerpt of an error message during such a failed migration:

zerolog: could not write event: cannot decode event: invalid character '<' after object key:value pair
Error: cannot configure db container: Cannot generate database certificate: CA validation failed!

When the CA password of the source system is lost (or forgotten), a new certificate chain for the database container can be created and the migration then run using for example:

a) SUSE Multi-Linux Manager Server 5.0 => SUSE Multi-Linux Manager Server 5.1

mgradm upgrade podman --ssl-db-ca-root /path/to/ca.crt --ssl-db-cert /path/to/server.crt --ssl-db-key /path/to/server.key

b) SUSE Manager Server 4.3 =>  SUSE Multi-Linux Manager Server 5.1

mgradm migrate podman fqdn.of.source.system --ssl-db-ca-root /path/to/ca.crt --ssl-db-cert /path/to/server.crt --ssl-db-key /path/to/server.key

When the CA certificate on the source system is password protected and the password is not properly "typed" during the SUSE Multi-Linux Manager Server migration, the database certificate (for the database container) can not be correctly signed by the "default" CA and the migration fails as mentioned above.

SUSE Multi-Linux Manager Server 5.1 is using 2 containers, uyuni-server and a database container. By default, during the migration the CA password is requested so the database certificate can be signed by the existing CA (from the source system).
 
The database certificate can be signed by a different CA (than the server certificate) which allows to create a new certificate chain for the database and use such a chain during the migration (when the password of the existing CA is lost/forgotten). 

A new database certificate chain can be created using for example the following steps (adjust the details to match the local environments criteria):

Tested on top of openSUSE Leap 15.6

a) Create CA cert and the key:

mkdir -p /root/pki && cd /root/pki

openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -keyout ca.key -out ca.crt -subj "/C=CZ/ST=Prague/L=Prague/O=SomeOrg/OU=IT/CN=cn root CA" -addext "subjectKeyIdentifier=hash"

=> set a new password for the new root CA when prompted. Following files are created:

system:~/pki # ll
total 8
-rw------- 1 root root 2041 Aug 12 15:19 ca.crt
-rw------- 1 root root 3434 Aug 12 15:19 ca.key

b) Create database cert and database key. IMPORTANT: the "common name" of the database cert MUST match the container host FQDN and the "subject alternative names" for the database cert MUST be: "db" and "reportdb"

openssl req -new -newkey rsa:4096 -nodes -keyout server.key -out server.csr -subj "/CN=coolhostname.even.cooler.domain" -addext "keyUsage=digitalSignature,keyEncipherment" -addext "extendedKeyUsage=serverAuth" -addext "subjectAltName=DNS:coolhostname.even.cooler.domain,DNS:db,DNS:reportdb"

c) Sign the database certificate with the CA created in a):

system:~/pki # cat > v3-server.ext <<'EOF'
basicConstraints=CA:FALSE
keyUsage=critical,digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
subjectAltName=DNS:coolhostname.even.cooler.domain,DNS:db,DNS:reportdb
authorityKeyIdentifier=keyid,issuer
subjectKeyIdentifier=hash
EOF

system:~/pki # ll
total 20
-rw------- 1 root root 2041 Aug 12 15:19 ca.crt
-rw------- 1 root root 3434 Aug 12 15:19 ca.key
-rw------- 1 root root 1752 Aug 12 15:22 server.csr
-rw------- 1 root root 3272 Aug 12 15:22 server.key
-rw------- 1 root root  235 Aug 12 15:22 v3-server.ext

and sign it:

system:~/pki # openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 825 -sha256 -extfile v3-server.ext
 
system:~/pki # ll
total 28
-rw------- 1 root root 2041 Aug 12 15:19 ca.crt
-rw------- 1 root root 3434 Aug 12 15:19 ca.key
-rw------- 1 root root   41 Aug 12 15:23 ca.srl
-rw------- 1 root root 2069 Aug 12 15:23 server.crt
-rw------- 1 root root 1752 Aug 12 15:22 server.csr
-rw------- 1 root root 3272 Aug 12 15:22 server.key
-rw------- 1 root root  235 Aug 12 15:22 v3-server.ext

d) Move the needed files to container host system e.g. using following command:

system:~/pki # for i in ca.crt server.crt server.key;do scp $i root@coolhostname.even.cooler.domain:~/;done

and run the migration as mentioned above.

 Migration documentation can be found here and here. Detail about the SSL configuration can be found at the following link.