IPSEC-ESP authentication problem on firewall policy

dario lazzarini 1 Reputation point
2022-05-17T09:47:42.13+00:00

Hello
I have a Windows 10 client, in Windows Server 2016/2019 domain (Domain A) that needs to connect in RDP (with IPSEC Policy) on a Windows Server 2008 R2 DC in other domain (Domain B).
A one-way trust relationship exists between the domains.
A firewall policy was created on DC 2008R2 that forces kerberos (user + computer) authentication (IPSEC ESP) to allow access to only authorized computers and users.
Both authorized users and clients expressed as domain groups are included in the policy.
Both the client and the DC target have the same Security Rules enabled.
Server and client are on the same network.

When I connect from the windows 10 client I get an error: "An IPsec main mode negotiation failed. ERROR ID 4653"

Local Endpoint:
Local Principal Name: -
Network Address: 10.0.0.99
Keying Module Port: 500

Remote Endpoint:
Principal Name: -
Network Address: 10.0.0.12
Keying Module Port: 500

Additional Information:
Keying Module Name: AuthIP
Authentication Method: Unknown authentication
Role: Responder
Impersonation State: Not enabled
Main Mode Filter ID: 76654

Failure Information:
Failure Point: Remote computer
Failure Reason: IKE authentication credentials are unacceptable
State: Sent second (KE) payload
Initiator Cookie: 560a805cd9eb9131
Responder Cookie: 1d0925bef6305e21

I cannot find any information or solution for this error. Can you give me some tips?

Windows 10 Network
Windows 10 Network
Windows 10: A Microsoft operating system that runs on personal computers and tablets.Network: A group of devices that communicate either wirelessly or via a physical connection.
2,270 questions
Windows Server Infrastructure
Windows Server Infrastructure
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.Infrastructure: A Microsoft solution area focused on providing organizations with a cloud solution that supports their real-world needs and meets evolving regulatory requirements.
513 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Limitless Technology 39,351 Reputation points
    2022-05-18T07:42:03.4+00:00

    Hi there,

    Events generated by the Authenticated Internet Protocol (AuthIP) and the Internet Key Exchange protocol (IKE) during Main Mode negotiations are audited by the codes which fall under the Audit IPsec Main Mode subcategory. Event 4653 falls under this category.

    Here is a troubleshooting link for your reference.IPSec Troubleshooting http://technet.microsoft.com/en-us/library/cc783041(v=WS.10).aspx

    Troubleshooting VPN over IPsec http://technet.microsoft.com/en-us/library/bb794765.aspx

    The below thread discusses the same issue and you can try out some troubleshooting steps from this and see if that helps you to sort the Issue.

    Audit Failure - Event ID 4653 - An IPsec main mode negotiation failed https://social.technet.microsoft.com/Forums/en-US/5c5befdb-db83-42e9-8ede-85d049806303/audit-failure-event-id-4653-an-ipsec-main-mode-negotiation-failed?forum=winserversecurity

    Audit IPsec Main Mode https://learn.microsoft.com/en-us/windows/security/threat-protection/auditing/audit-ipsec-main-mode

    ---------------------------------------------------------------------------------------------------------------------------------------

    --If the reply is helpful, please Upvote and Accept it as an answer–

    0 comments