Best practices to troubleshoot (transparent) proxy setup with connectivity issues

This document (7017561) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 12

Situation

It regularly happens that servers can not connect/reach the SUSE update/registration locations, like "secure-www.novell.com" or "updates.suse.com".

As an example, while trying to register a server via "suse_register" the following error message shows up:

error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Closing connection #0
ERROR: Peer certificate cannot be authenticated with known CA certificates: (60)
(2)
ERROR: Peer certificate cannot be authenticated with known CA certificates: (60)
(2)

^ Please note that this is just an example of a possible error message. Other issues can result in different error messages!

The troubleshooting becomes tricky if a PROXY is involved in the setup, specially if it is a transparent PROXY.

Resolution

This TID does provide some best practices on how to troubleshoot such a scenario.

As a default: All the SUSE certificates are signed by the same CA (Certificate Authority), so the related hash value is the same, no matter if you connect to:

SLES 11 : secure-www.novell.com
SLES 12 : updates.suse.com

The following strace command will let you identify this hash value:

# strace -e trace=stat curl "https://updates.suse.com"
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=817, ...}) = 0
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
stat("/etc/ssl/certs//81b9768f.0", {st_mode=S_IFREG|0644, st_size=1409, ...}) = 0
stat("/etc/ssl/certs//81b9768f.1", 0x7ffc4aa0c3a0) = -1 ENOENT (No such file or directory)
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"

# strace -e trace=stat curl "https://secure-www.novell.com"
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=817, ...}) = 0
stat("/dev/urandom", {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 9), ...}) = 0
stat("/etc/ssl/certs//81b9768f.0", {st_mode=S_IFREG|0644, st_size=1409, ...}) = 0
stat("/etc/ssl/certs//81b9768f.1", 0x7ffcb5f61fc0) = -1 ENOENT (No such file or directory)
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

In the above example the hash value is "81b9768f". As a next step it makes sense to check if that certificate is available on the server itself.

Certificates are stored in /etc/ssl/certs
simply run "ls -lsa /etc/ssl/certs | grep <hash-value>", so in this example:
# ls -lsa /etc/ssl/certs | grep 81b9768f
0 lrwxrwxrwx 1 root root 38 Jul 6 2015 81b9768f.0 -> DigiCert_High_Assurance_EV_Root_CA.pem

To test if a (transparent) proxy is part of the setup execute:

SLES 11:
openssl s_client -connect secure-www.novell.com:443 -showcerts -servername secure-www.novell.com
SLES 12:
openssl s_client -connect updates.suse.com:443 -showcerts -servername updates.suse.com

If the above command does not return the following certificate chain, you are behind a (transparent) proxy:

# openssl s_client -connect secure-www.novell.com:443 -showcerts -servername secure-www.novell.com
CONNECTED(00000003)
depth=2 /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
verify return:1
depth=1 /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
verify return:1

If the output is different here and there is a connectivity issue, most probably the Certificate signed by the Proxy CA is not available on the server. This can be tested with the hash test from above.
To make this Certificate available retrieve a ".pem" certificate file from the responsible proxy admin, copy it to /etc/ssl/certs and run the command "c_rehash".

SUSE is not responsible for the PROXY setup in customer environments and can only provide guidance in such a scenario.

Additional Information

A more detailed trace can be pulled by running the following command:

strace -s 256 -o /tmp/curl.trace curl "https://secure-www.novell.com"
strace -s 256 -o /tmp/curl.trace curl "https://updates.suse.com"

To check a certificate itself (Issue, Validity, etc.) run the following command:

openssl x509 -in /etc/ssl/certs/DigiCert_High_Assurance_EV_Root_CA.pem -text -noout
Example output:
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            02:ac:5c:26:6a:0b:40:9b:8f:0b:79:f2:ae:46:25:77
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert High Assurance EV Root CA
        Validity
            Not Before: Nov 10 00:00:00 2006 GMT
            Not After : Nov 10 00:00:00 2031 GMT
        Subject: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert High Assurance EV Root CA
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    00:c6:cc:e5:73:e6:fb:d4:bb:e5:2d:2d:32:a6:df:
                    e5:81:3f:c9:cd:25:49:b6:71:2a:c3:d5:94:34:67:
                    a2:0a:1c:b0:5f:69:a6:40:b1:c4:b7:b2:8f:d0:98:
                    a4:a9:41:59:3a:d3:dc:94:d6:3c:db:74:38:a4:4a:
                    cc:4d:25:82:f7:4a:a5:53:12:38:ee:f3:49:6d:71:
                    91:7e:63:b6:ab:a6:5f:c3:a4:84:f8:4f:62:51:be:
                    f8:c5:ec:db:38:92:e3:06:e5:08:91:0c:c4:28:41:
                    55:fb:cb:5a:89:15:7e:71:e8:35:bf:4d:72:09:3d:
                    be:3a:38:50:5b:77:31:1b:8d:b3:c7:24:45:9a:a7:
                    ac:6d:00:14:5a:04:b7:ba:13:eb:51:0a:98:41:41:
                    22:4e:65:61:87:81:41:50:a6:79:5c:89:de:19:4a:
                    57:d5:2e:e6:5d:1c:53:2c:7e:98:cd:1a:06:16:a4:
                    68:73:d0:34:04:13:5c:a1:71:d3:5a:7c:55:db:5e:
                    64:e1:37:87:30:56:04:e5:11:b4:29:80:12:f1:79:
                    39:88:a2:02:11:7c:27:66:b7:88:b7:78:f2:ca:0a:
                    a8:38:ab:0a:64:c2:bf:66:5d:95:84:c1:a1:25:1e:
                    87:5d:1a:50:0b:20:12:cc:41:bb:6e:0b:51:38:b8:
                    4b:cb
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                B1:3E:C3:69:03:F8:BF:47:01:D4:98:26:1A:08:02:EF:63:64:2B:C3
            X509v3 Authority Key Identifier:
                keyid:B1:3E:C3:69:03:F8:BF:47:01:D4:98:26:1A:08:02:EF:63:64:2B:C3

    Signature Algorithm: sha1WithRSAEncryption

It is also possible to display only the hash for a Certificate by running the following command:

openssl x509 -in /etc/ssl/certs/DigiCert_High_Assurance_EV_Root_CA.pem -subject_hash -noout

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.