My Favorites

Close

Please to see your favorites.


DSfW: Kerberos fails to start

This document (7011515) is provided subject to the disclaimer at the end of this document.

Environment

Open Enterprise Server 11 SP2 (OES11 SP2)
Open Enterprise Server 11 SP1 (OES11 SP1)
Open Enterprise Server 11 (OES11)
Domain Services for Windows
DSfW

Situation

Troubleshoot Kerberos
Kerberos fails to start
xad-krb5kdc fails to start
xadsd fails to start

Resolution

Missing uniquedomainid attribute on key objects
  Do a ldapsearch for object with out uniquedomainid replace <domain name> with the name of the domain.  Export the LDAPCONF before doing the ldap search (7003070).

export LDAPCONF=/etc/opt/novell/xad/openldap/ldap.conf
ldapsearch -Y EXTERNAL -LLL -Q -b <domain name> -s sub '(!(uniquedomainid=*))' dn

example:
ldapsearch -Y EXTERNAL -LLL -Q -b dc=novell,dc=com -s sub '(!(uniquedomainid=*))' dn
Missing uniquedomainid attribute on key objects
If an error "ldap_sasl_interactive_bind_s: Can't contact LDAP server (-1)"  is returned proceed to Missing ldap interfaces on the ldap server object.
Objects that are missing the uniquedomainid should have it added. Use a ldif to add the attribute.
Some key objects to check for the uniquedomainid attribute are: krbtgt, domain controller object, domain controller container and the container mapped to the domain.  If these 4 objects have a uniquedomainid then usually kerberos will start.  Once all DSfW services are running the domain can be re-samified which should add the uniquedomainid attribute to any object in the domain missing this key attribute.  See TID 7009851.

Missing ldap interfaces on the ldap server object
The ldap interfaces are missing from the LDAP Server object for the DSfW server.
This usually is a result of the ldap Group and ldap server objects were deleted and recreated.
The interfaces on the server objects and the mappings on the group object are unique on DSfW servers and need to be recreated with the correct settings.
See TID 7010319 on how to correctly recreate these objects.  

Corrupt or missing libraries
Check that the libraries are in place and the symbolic links to the libraries.

Here is a list of key xad, ldap, and kerberos libraries with symbolic links as of the November 2012 Maintenance patch

/opt/novell/xad/lib64/krb5/plugins/kdb
-rwxr-xr-x 1 root root 74896 Nov 27 14:31 kdb_xad.so*

/opt/novell/xad/lib64/sasl2
-rwxr-xr-x 1 root root 31888 Aug  3 06:31 libgssapiv2.so*

:/opt/novell/xad/lib64/gss
lrwxrwxrwx 1 root root     25 Dec 14 12:40 libmech_netlogon.so.1 -> libmech_netlogon.so.1.0.0*
-rwxr-xr-x 1 root root 759960 Nov 27 14:31 libmech_netlogon.so.1.0.0*
-rwxr-xr-x 1 root root 123144 Aug  3 06:54 mech_krb5.so*
-rwxr-xr-x 1 root root 797528 Nov 27 14:31 mech_ntlm.so*
-rwxr-xr-x 1 root root  48272 Aug  3 06:54 mech_spnego.so*

/opt/novell/xad/lib64/nds-modules
-rwxr-xr-x 1 root root  14848 Nov 27 14:31 libanr-plugin.so*
-rwxr-xr-x 1 root root  35808 Nov 27 14:31 libcrossref-plugin.so*
-rwxr-xr-x 1 root root  23232 Nov 27 14:31 libdsearch-plugin.so*
-rwxr-xr-x 1 root root 263560 Nov 27 14:31 libnad-plugin.so*
-rwxr-xr-x 1 root root  31664 Nov 27 14:31 libnetlogon-plugin.so*
-rwxr-xr-x 1 root root  36256 Nov 27 14:31 libntacl-plugin.so*
-rwxr-xr-x 1 root root  32176 Nov 27 14:31 libsubschema-plugin.so*
-rwxr-xr-x 1 root root  15040 Nov 27 14:31 libtokengroups-plugin.so*
-rwxr-xr-x 1 root root   6448 Nov 27 14:31 libwhoami-plugin.so*

/opt/novell/xad/lib64
lrwxrwxrwx  1 root root      21 Dec 14 12:40 libadmpasswd.so.2 -> libadmpasswd.so.2.0.0*
-rwxr-xr-x  1 root root    6312 Nov 27 14:31 libadmpasswd.so.2.0.0*
-rwxr-xr-x  1 root root   32368 Nov 27 14:31 libauth_gss.so*
-rwxr-xr-x  1 root root   23248 Nov 27 14:31 libauth_netlogon.so*
lrwxrwxrwx  1 root root      17 Oct  8 10:49 libcom_err.so.3 -> libcom_err.so.3.0*
-rwxr-xr-x  1 root root   23136 Aug  3 06:54 libcom_err.so.3.0*
lrwxrwxrwx  1 root root      23 Dec 14 12:39 libdcepthreads.so.1 -> libdcepthreads.so.1.0.0*
-rwxr-xr-x  1 root root   35784 Nov 26 09:15 libdcepthreads.so.1.0.0*
lrwxrwxrwx  1 root root      18 Dec 14 12:39 libdcerpc.so.1 -> libdcerpc.so.1.0.2*
-rwxr-xr-x  1 root root  450496 Nov 26 09:15 libdcerpc.so.1.0.2*
lrwxrwxrwx  1 root root      16 Oct  8 10:49 libdes425.so.3 -> libdes425.so.3.0*
-rwxr-xr-x  1 root root   14688 Aug  3 06:54 libdes425.so.3.0*
lrwxrwxrwx  1 root root      19 Dec 14 12:40 libdssetup.so.1 -> libdssetup.so.1.0.0*
-rwxr-xr-x  1 root root   11200 Nov 27 14:31 libdssetup.so.1.0.0*
lrwxrwxrwx  1 root root      16 Oct  8 10:49 libgssapi.so.1 -> libgssapi.so.1.0*
-rwxr-xr-x  1 root root   80936 Aug  3 06:54 libgssapi.so.1.0*
lrwxrwxrwx  1 root root      21 Oct  8 10:49 libgssapi_krb5.so.2 -> libgssapi_krb5.so.2.2*
-rwxr-xr-x  1 root root   10512 Aug  3 06:54 libgssapi_krb5.so.2.2*
lrwxrwxrwx  1 root root      16 Oct  8 10:49 libgssrpc.so.4 -> libgssrpc.so.4.0*
-rwxr-xr-x  1 root root  115240 Aug  3 06:54 libgssrpc.so.4.0*
lrwxrwxrwx  1 root root      18 Oct  8 10:49 libk5crypto.so.3 -> libk5crypto.so.3.1*
-rwxr-xr-x  1 root root  854704 Aug  3 06:54 libk5crypto.so.3.1*
lrwxrwxrwx  1 root root      19 Oct  8 10:49 libkadm5clnt.so.5 -> libkadm5clnt.so.5.1*
-rwxr-xr-x  1 root root   81296 Aug  3 06:54 libkadm5clnt.so.5.1*
lrwxrwxrwx  1 root root      18 Oct  8 10:49 libkadm5srv.so.5 -> libkadm5srv.so.5.1*
-rwxr-xr-x  1 root root  106352 Aug  3 06:54 libkadm5srv.so.5.1*
lrwxrwxrwx  1 root root      14 Oct  8 10:49 libkdb5.so.4 -> libkdb5.so.4.0*
-rwxr-xr-x  1 root root   44104 Aug  3 06:54 libkdb5.so.4.0*
lrwxrwxrwx  1 root root      14 Oct  8 10:49 libkrb5.so.3 -> libkrb5.so.3.3*
-rwxr-xr-x  1 root root  606736 Aug  3 06:54 libkrb5.so.3.3*
lrwxrwxrwx  1 root root      21 Oct  8 10:49 libkrb5support.so.0 -> libkrb5support.so.0.1*
-rwxr-xr-x  1 root root   31720 Aug  3 06:54 libkrb5support.so.0.1*
lrwxrwxrwx  1 root root      18 Dec 14 12:40 liblsarpc.so.0 -> liblsarpc.so.0.0.0*
-rwxr-xr-x  1 root root  159184 Nov 27 14:31 liblsarpc.so.0.0.0*
-rwxr-xr-x  1 root root   19336 Nov 26 09:15 libnaf_ip.so*
-rwxr-xr-x  1 root root   19304 Nov 26 09:15 libnaf_np.so*
lrwxrwxrwx  1 root root      18 Dec 14 12:40 libnetapi.so.1 -> libnetapi.so.1.0.0*
-rwxr-xr-x  1 root root  105528 Nov 27 14:31 libnetapi.so.1.0.0*
lrwxrwxrwx  1 root root      17 Dec 14 12:40 libntacl.so.2 -> libntacl.so.2.0.0*
-rwxr-xr-x  1 root root   48904 Nov 27 14:31 libntacl.so.2.0.0*
lrwxrwxrwx  1 root root      19 Dec 14 12:40 libntdsapi.so.1 -> libntdsapi.so.1.0.0*
-rwxr-xr-x  1 root root  125808 Nov 27 14:31 libntdsapi.so.1.0.0*
-rwxr-xr-x  1 root root  169936 Nov 26 09:15 libprot_ncacn.so*
-rwxr-xr-x  1 root root  161760 Nov 26 09:15 libprot_ncadg.so*
-rwxr-xr-x  1 root root    6232 Nov 26 09:15 libpthread_ext.so*
-rwxr-xr-x  1 root root    6232 Nov 26 09:15 libpthread_ext.so.0*
-rwxr-xr-x  1 root root   10456 Nov 27 14:31 libw32time.so*
lrwxrwxrwx  1 root root      19 Dec 14 12:40 libwinbase.so.2 -> libwinbase.so.2.0.0*
-rwxr-xr-x  1 root root   82304 Nov 27 14:31 libwinbase.so.2.0.0*
lrwxrwxrwx  1 root root      18 Dec 14 12:40 libxaddrs.so.4 -> libxaddrs.so.4.0.0*
-rwxr-xr-x  1 root root  160672 Nov 27 14:31 libxaddrs.so.4.0.0*
lrwxrwxrwx  1 root root      18 Dec 14 12:40 libxaddsa.so.2 -> libxaddsa.so.2.0.0*
-rwxr-xr-x  1 root root   74512 Nov 27 14:31 libxaddsa.so.2.0.0*
lrwxrwxrwx  1 root root      18 Dec 14 12:40 libxaddsr.so.0 -> libxaddsr.so.0.0.0*
-rwxr-xr-x  1 root root   15488 Nov 27 14:31 libxaddsr.so.0.0.0*
lrwxrwxrwx  1 root root      18 Dec 14 12:40 libxadkcc.so.2 -> libxadkcc.so.2.0.0*
-rwxr-xr-x  1 root root   14928 Nov 27 14:31 libxadkcc.so.2.0.0*
lrwxrwxrwx  1 root root      18 Dec 14 12:40 libxadlsa.so.0 -> libxadlsa.so.0.0.0*
-rwxr-xr-x  1 root root  235584 Nov 27 14:31 libxadlsa.so.0.0.0*
lrwxrwxrwx  1 root root      18 Dec 14 12:40 libxadnds.so.2 -> libxadnds.so.2.0.0*
-rwxr-xr-x  1 root root   45032 Nov 27 14:31 libxadnds.so.2.0.0*
lrwxrwxrwx  1 root root      17 Dec 14 12:40 libxadnl.so.1 -> libxadnl.so.1.0.0*
-rwxr-xr-x  1 root root  206368 Nov 27 14:31 libxadnl.so.1.0.0*
lrwxrwxrwx  1 root root      19 Dec 14 12:40 libxadprov.so.2 -> libxadprov.so.2.0.0*
-rwxr-xr-x  1 root root   62144 Nov 27 14:31 libxadprov.so.2.0.0*
lrwxrwxrwx  1 root root      18 Dec 14 12:40 libxadsam.so.1 -> libxadsam.so.1.0.0*
-rwxr-xr-x  1 root root  207392 Nov 27 14:31 libxadsam.so.1.0.0*
lrwxrwxrwx  1 root root      18 Dec 14 12:40 libxadsdk.so.2 -> libxadsdk.so.2.0.0*
-rwxr-xr-x  1 root root 1316552 Nov 27 14:31 libxadsdk.so.2.0.0*


Mis-configured or missing kdc.conf

The kdc.conf is located at: /etc/opt/novell/xad/krb5.conf
Verify that a symbolic link exists at /etc/krb5.conf to /etc/opt/novell/xad/krb5.conf

sample kdc.conf
In this example the DSfW server's name is servername and the domain name is dsfw.lan
#ident $Id: krb5.conf,v 1.50 2006/03/05 02:58:10 lukeh Exp $
#
# Kerberos configuration for Domain Services for Windows
#

[libdefaults]
        default_keytab_name = /var/opt/novell/xad/ds/krb5kdc/krb5.keytab
        default_realm = DSFW.LAN
        permitted_enctypes = arcfour-hmac-md5 des-cbc-md5 des-cbc-crc
        default_tkt_enctypes = arcfour-hmac-md5
        default_tgs_enctypes = arcfour-hmac-md5
        noaddresses = true
        case_sensitive = false
        verify_ap_req_try_host_spn = true
        dns_lookup_kdc = true
        rdns = true

[appdefaults]
        pam = {
                krb4_convert = false
                validate = true
                ignore_unknown_upn = true
                keytab = /var/opt/novell/xad/ds/krb5kdc/krb5.keytab
        }

[logging]
        kdc = FILE:/var/opt/novell/xad/log/kdc.log
        admin_server = FILE:/var/opt/novell/xad/log/adm.log
        kpasswdd = FILE:/var/opt/novell/xad/log/kpasswdd.log
        kcm = FILE:/var/opt/novell/xad/log/kcm.log

[kcm]
        system_ccache = {
                principal = SERVERNAME$
                spn_aliases = host/servername.dsfw.lan
                spn_aliases = cifs/servername.dsfw.lan
                spn_aliases = HTTP/servername.dsfw.lan
                spn_aliases = ldap/servername.dsfw.lan
                spn_aliases = DNS/servername.dsfw.lan
                spn_aliases = HOST/SERVERNAME
        }

[dbmodules]
        XAD = {
                db_module_dir = /opt/novell/xad/lib/krb5/plugins/kdb
                db_library = kdb_xad
        }

[realms]
        DSFW.LAN = {
                kdc = server.dsfw.lan
#               kpasswd_server = server.dsfw.lan
                database_module = XAD
        }

[domain_realm]
        .servername.dsfw = DSFW.LAN
        servername.dsfw = DSFW.LAN


0 Byte stream file for sasLoginServerMethogdLinux64 

Taking a LDAP/NMAS trace reports a -1693 error.  See TID 7009590 to resolve this issue.
See TID 7009602 for taking eDirectory LDAP/NMAS traces.


Poorly Configured Network

When kerberos start it must contact the security container to access the NMAS methods in order to properly load the methods.  The IPCEXTERNAL must be active and loaded for kerberos to start.
To verify the IPCEXTERNAL method is running do:
lsof -p `pgrep ndsd` |grep -i ipc

This command should return the following:
ndsd    3452 root  mem    REG                8,2   149416   571515 /var/opt/novell/eDirectory/data/nmas-methods/IPCLCMLIN_X64.SO
ndsd    3452 root  mem    REG                8,2   153544   571514 /var/opt/novell/eDirectory/data/nmas-methods/IPCLSMLIN_X64.SO

If IPCLSMLIN is not returned, the IPCEXTERNAL method is not loaded.  It could be that the method and sequence has been set to inactive or that the DSfW server is unable to access the security container to update the reference and streamfiles to the needed NMAS methods.
Use iManager to verify the method is active.

Add a replica of root to the DSfW server or if the security container is partitioned, add a replica of the security partition.
Usually adding a replica that containers the security container is not needed unless there is a slow wan link, packet loss, traffic loop, packet congestion, packet collisions, router/switch configuration issues, misconfigured nic, or bad hardward (router/switch/nic)

If kerberos fails to start on reboot and 5 minutes later kerberos starts, the cause is most likely a poorly configured network.

This is usually seen on ADC servers where Spanning Tree is enabled on the router or switch.
When kerberos on the ADC starts, it attempts to resolve the short name (wins) for the domain.  Configuring WINS on the PDC and adding wins server = <ip address of wins server> to the ADC can help.  See TID 7012934 for more information on configuring WINS.

Taking a packet trace will most likely show a TCP Retransmission or TCP Dup ACK for every other packet.  Ping for NDS, Start Update Replica, Resolve Name will all have TCP Retransmission and TCP Dup ACK.

The router or switch will have Spanning Tree enabled.  Most vendors have a proprietory method to handle spanning tree issues.  Before disabling Spanning Tree, enable the vendors method like Shortest Patch Bridging or Fabric Path.  Then disable Spanning Tree.  Restarting the server should allow the DSfW domain's short name to be found quickly and kerberos to start.

Cause

If kerberos fails to start it is usually caused by one of the following reasons:

Missing uniquedomainid attribute on key objects
Missing ldap interfaces on the ldap server object
Corrupt or missing libraries
Mis-configured or missing kdc.conf

Additional Information

A video presentation is available demonstrating how to add the correct ldap interface here.

For troubleshooting take LDAP/NMAS traces (TID 7009602 )and a packet trace (TID 7001152).

Disclaimer

This Support Knowledgebase provides a valuable tool for NetIQ/Novell/SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:7011515
  • Creation Date:12-DEC-12
  • Modified Date:02-APR-14
    • NovellOpen Enterprise Server
    • SUSESUSE Linux Enterprise Server
    • NetIQeDirectory

Did this document solve your problem? Provide Feedback