Authentication Fails in a Mixed Windows and UNIX Environment

Applies To: Windows Server 2003, Windows Server 2003 R2, Windows Server 2003 with SP1, Windows Server 2003 with SP2

The logon from a computer running Windows NT 4.0 in a mixed Windows and UNIX environment fails and the Active Directory domain controller returns an Access denied error.

In an environment where a trust exists between a Kerberos realm and an Active Directory domain, authentication data can come from one of two sources, either a UNIX Key Distribution Center (KDC) or the Active Directory domain controller.

If the authentication data is coming from a UNIX KDC, Windows users have account mappings set up for them to map their UNIX user account to a Windows user account. Normally, the password on the Windows account does not matter, because all authentications are done by the UNIX KDC.

Cause

The size of a ticket is too large to be transmitted reliably by means of the User Datagram Protocol (UDP). In a Windows environment, this message is purely informational. A computer running a Windows operating system will automatically try TCP if UDP fails.

In a mixed UNIX and Windows authentication environment, many errors can be attributed to the UNIX KDC not running the latest MIT distribution of the Kerberos protocol. One of the errors that can result is "0x34 - KRB_ERR_RESPONSE_TOO_BIG: Response too big for UDP, retry with TCP."

Solution

If this error occurs in a mixed operating system environment, consult the Vendor specific documentation to upgrade the UNIX KDCs to the latest MIT distribution of the Kerberos protocol, which supports TCP connections if UDP fails.

Cause

The "0x19 - KDC_ERR_PREAUTH_REQUIRED: Additional pre-authentication required" error often occurs in UNIX interoperability scenarios. MIT Kerberos clients do not request pre-authentication when they send a KRB_AS_REQ message. If pre-authentication is required (the default), Windows operating systems send this error. Most MIT-Kerberos clients respond to this error by giving the pre-authentication, in which case the error can be ignored, but some clients might not respond in this way.

Solution

Set the Do not require Kerberos pre-authentication flag on the user's account. Alternatively, consider upgrading to the most recent MIT reference distribution of Kerberos authentication.

To set the Do not require Kerberos pre-authentication flag on the user's account using Active Directory Users and Computers:

  1. Open Active Directory Users and Computers.

  2. In the console tree, click Users. (Or, click the folder that contains the user account.)

  3. Right-click the user account, and then click Properties.

  4. On the Account tab, scroll through the Account options and select the Do not require Kerberos pre-authentication checkbox, and then click OK.

Note

To perform this procedure, you must be a member of the Account Operators group, Domain Admins group, or the Enterprise Admins group in Active Directory, or you must have been delegated an assignment of administrative responsibility to a user, computer, group, or organization.

Cause

When the KDC, server, or client receives a packet for which it does not it have a key of the appropriate encryption type, this can cause the "0xF - KDC_ERR_SUMTYPE_NOSUPP: KDC has no support for checksum type" error. The result is that the computer is unable to decrypt the ticket. This error is common in UNIX interoperability scenarios when a new account is created. The new account might have an incompatible key associated with it.

Solution

Change the password on the account to re-create the key, which should eliminate the error.

Cause

The UNIX KDC attempts to use 3DES to encrypt its tickets which can cause the "0xE - KDC_ERR_ETYPE_NOTSUPP: KDC has no support for encryption type" error.

Solution

Consult the vendor-specific documentation to configure the UNIX KDC to use another encryption type, such as DES or RC4. Windows operating systems do not support 3DES.

Cause

The target server does not support the encryption type used by the KDC in some UNIX and Windows interoperability scenarios.

Solution

Consult the vendor-specific documentation to configure the target server to support the RFC standard encryption type RC4 or contact the vendor for assistance.

Cause

The Windows NT 4.0 operating system does not support Kerberos authentication. Therefore, if there are Windows NT–based computers on the network running services, any authentications involving these computers occur by using NTLM and these authentications are conducted by the domain controller. In this scenario, the password on the domain controller must match the password stored on the UNIX KDC. The passwords must match because the fallback to NTLM occurs transparently. If the passwords do not match, the domain controller returns an Access denied error because the user has provided a password that does not match the one stored on the user's Active Directory account.

Solution

Consult the vendor-specific documentation to reset the password on the account that the user’s UNIX principal is mapped to in order to match the password stored on the UNIX KDC.