L2TP VPN fails with error 787
This article provides help to fix the error 787 that occurs when a L2TP VPN connection to a Remote Access server fails.
Original product version: Windows Server 2012 R2
Original KB number: 2855053
A L2TP VPN connection to a Windows Server 2012 Remote Access server fails with error 787 "The L2TP connection attempt failed because the security layer could not authenticate the remote computer."
The Server is configured for as well VPN connections as DirectAccess and has at least two valid certificates. One certificate for IPHTTPS and one for L2TP. Both certificates have at least the Server Authentication EKU, for example: • Server Authentication (18.104.22.168.22.214.171.124.1) • Client Authentication (126.96.36.199.188.8.131.52.2) optionally also • IP security IKE intermediate (184.108.40.206.220.127.116.11.2)
One of the certificates is a wildcard certificate. The certificates might also be from different Certificate Authorities.
The IPsec SA establishment for the L2TP connection fails because the server uses the wildcard certificate and/or a certificate from a different Certificate Authority as the computer certificate configured on the clients. Routing and Remote Access (RRAS) is choosing the first certificate it can find in the computer certificate store. For L2TP, different from SSTP or IPHTTPS or any other manual configured IPsec rule, you rely on the RRAS built in mechanism for choosing a certificate. There is no way to influence this.
There are two possible solutions:
Use a single certificate for IPHTTPS and L2TP.
Use a manually configured L2TP IPsec policy on the RRAS server (it is not needed on the clients) and disable the automatically configured IPsec policy.
- Path: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Rasman\Parameters
- Value Name: ProhibitIpSec
- Data Type: REG_DWORD
- Value: 1
Then add an IPsec policy manually - This is an L2TP Rule:
Rule Name: L2TP Manual Rule
Description: L2TP Manual Rule
Profiles: Private, Public
Auth1CAName: DC=com, DC=contoso, DC=corp, CN=corp-DC1-CA
MainModeSecMethods: DHGroup2-AES128-SHA256, DHGroup2-AES128-SHA1, DHGroup2-3DES-SHA1
Rule source: Local Setting