Cloud sync troubleshooting
Cloud sync has many different dependencies and interactions, which can give rise to various problems. This article helps you troubleshoot these problems. It introduces the typical areas for you to focus on, how to gather additional information, and the various techniques you can use to track down problems.
When you troubleshoot agent problems, you verify that the agent was installed correctly, and that it communicates with Azure Active Directory (Azure AD). In particular, some of the first things that you want to verify with the agent are:
- Is it installed?
- Is the agent running locally?
- Is the agent in the portal?
- Is the agent marked as healthy?
You can verify these items in the Azure portal and on the local server that's running the agent.
Azure portal agent verification
To verify that Azure detects the agent, and that the agent is healthy, follow these steps:
Sign in to the Azure portal.
On the left, select Azure Active Directory > Azure AD Connect. In the center, select Manage sync.
On the Azure AD Connect cloud sync screen, select Review all agents.
On the On-premises provisioning agents screen, you see the agents you've installed. Verify that the agent in question is there. If all is well, you will see the active (green) status for the agent.
Verify the required open ports
Verify that the Azure AD Connect provisioning agent is able to communicate successfully with Azure datacenters. If there's a firewall in the path, make sure that the following ports to outbound traffic are open:
|Port number||How it's used|
|80||Downloading certificate revocation lists (CRLs), while validating the TLS/SSL certificate.|
|443||Handling all outbound communication with the Application Proxy service.|
If your firewall enforces traffic according to originating users, also open ports 80 and 443 for traffic from Windows services that run as a network service.
Allow access to URLs
Allow access to the following URLs:
|URL||Port||How it's used|
||443/HTTPS||Communication between the connector and the Application Proxy cloud service.|
||80/HTTP||The connector uses these URLs to verify certificates.|
||443/HTTPS||The connector uses these URLs during the registration process.|
||80/HTTP||The connector uses this URL during the registration process.|
You can allow connections to
*.servicebus.windows.net, and other of the preceding URLs, if your firewall or proxy lets you configure access rules based on domain suffixes. If not, you need to allow access to the Azure IP ranges and service tags - public cloud. The IP ranges are updated each week.
Avoid all forms of inline inspection and termination on outbound TLS communications between Azure AD Application Proxy connectors and Azure AD Application Proxy cloud services.
DNS name resolution for Azure AD Application Proxy endpoints
Public DNS records for Azure AD Application Proxy endpoints are chained CNAME records, pointing to an A record. This ensures fault tolerance and flexibility. It’s guaranteed that the Azure AD Application Proxy connector always accesses host names with the domain suffixes
However, during the name resolution, the CNAME records might contain DNS records with different host names and suffixes. Due to this, you must ensure that the device can resolve all the records in the chain, and allows connection to the resolved IP addresses. Because the DNS records in the chain might be changed from time to time, we can't provide you with any list DNS records.
On the local server
To verify that the agent is running, follow these steps:
On the server with the agent installed, open Services. Do this by going to Start > Run > Services.msc.
Under Services, make sure Microsoft Azure AD Connect Agent Updater and Microsoft Azure AD Connect Provisioning Agent are there. Also confirm that their status is Running.
Common agent installation problems
The following sections describe some common agent installation problems, and typical resolutions of those problems.
Agent failed to start
You might receive an error message that states:
Service 'Microsoft Azure AD Connect Provisioning Agent' failed to start. Verify that you have sufficient privileges to start the system services.
This problem is typically caused by a group policy. The policy prevented permissions from being applied to the local NT Service sign-in account created by the installer (
NT SERVICE\AADConnectProvisioningAgent). These permissions are required to start the service.
To resolve this problem, follow these steps:
Sign in to the server with an administrator account.
Open Services by going to Start > Run > Services.msc.
Under Services, double-click Microsoft Azure AD Connect Provisioning Agent.
On the Log On tab, change This account to a domain admin. Then restart the service.
Agent times out or certificate isn't valid
You might get the following error message when you attempt to register the agent.
This problem is usually caused by the agent being unable to connect to the hybrid identity service. To resolve this problem, configure an outbound proxy.
The provisioning agent supports the use of an outbound proxy. You can configure it by editing the following agent .config file: C:\Program Files\Microsoft Azure AD Connect Provisioning Agent\AADConnectProvisioningAgent.exe.config.
Add the following lines into it, toward the end of the file, just before the closing
</configuration> tag. Replace the variables
[proxy-port] with your proxy server name and port values.
<system.net> <defaultProxy enabled="true" useDefaultCredentials="true"> <proxy usesystemdefault="true" proxyaddress="http://[proxy-server]:[proxy-port]" bypassonlocal="true" /> </defaultProxy> </system.net>
Agent registration fails with security error
You might get an error message when you install the cloud provisioning agent. This problem is typically caused by the agent being unable to run the PowerShell registration scripts, due to local PowerShell execution policies.
To resolve this problem, change the PowerShell execution policies on the server. You need to have machine and user policies set as
RemoteSigned. If they're set as
Unrestricted, you'll see this error. For more information, see PowerShell execution policies.
By default, the agent emits minimal error messages and stack trace information. You can find these trace logs in the following folder: C:\ProgramData\Microsoft\Azure AD Connect Provisioning Agent\Trace.
To gather additional details for troubleshooting agent-related problems, follow these steps.
- Install the AADCloudSyncTools PowerShell module.
- Use the
Export-AADCloudSyncToolsLogsPowerShell cmdlet to capture the information. You can use the following options to fine-tune your data collection.
SkipVerboseTraceto only export current logs without capturing verbose logs (default = false).
TracingDurationMinsto specify a different capture duration (default = 3 minutes).
OutputPathto specify a different output path (default = user’s Documents folder).
Object synchronization problems
In the Azure portal, you can use provisioning logs to help track down and troubleshoot object synchronization problems. To view the logs, select Logs.
Provisioning logs provide a wealth of information on the state of the objects being synchronized between your on-premises Active Directory environment and Azure.
You can filter the view to focus on specific problems, such as dates. Double-click an individual event to see additional information.
This information provides detailed steps and where the synchronization problem is occurring. In this way, you can pinpoint the exact spot of the problem.
Provisioning quarantined problems
Cloud sync monitors the health of your configuration, and places unhealthy objects in a quarantine state. If most or all of the calls made against the target system consistently fail because of an error (for example, invalid admin credentials), the sync job is marked as in quarantine.
By selecting the status, you can see additional information about the quarantine. You can also obtain the error code and message.
Right-clicking on the status will bring up additional options to:
- View the provisioning logs.
- View the agents.
- Clear the quarantine.
Resolve a quarantine
There are two different ways to resolve a quarantine. You can clear the quarantine, or you can restart the provisioning job.
Clear the quarantine
To clear the watermark and run a delta sync on the provisioning job after you have verified it, simply right-click on the status and select Clear quarantine.
You should see a notice that the quarantine is clearing.
Then you should see the status on your agent as healthy.
Restart the provisioning job
Use the Azure portal to restart the provisioning job. On the agent configuration page, select Restart sync.
Alternatively, you can use Microsoft Graph to restart the provisioning job. You have full control over what you restart. You can choose to clear:
- Escrows, to restart the escrow counter that accrues toward quarantine status.
- Quarantine, to remove the application from quarantine.
Use the following request:
Repair the cloud sync service account
If you need to repair the cloud sync service account, you can use the
From a PowerShell session with administrative privileges, type, or copy and paste, the following:
Enter your Azure AD global admin credentials.
Type, or copy and paste, the following:
After this completes, it should say that the account was repaired successfully.
To enable and use password writeback with cloud sync, keep the following in mind:
- If you need to update the gMSA permissions, it might take an hour or more for these permissions to replicate to all the objects in your directory. If you don't assign these permissions, writeback can appear to be configured correctly, but users might encounter errors when they update their on-premises passwords from the cloud. Permissions must be applied to This object and all descendant objects for Unexpire Password to appear.
- If passwords for some user accounts aren't written back to the on-premises directory, make sure that inheritance isn't disabled for the account in the on-premises Active Directory Domain Services (AD DS) environment. Write permissions for passwords must be applied to descendant objects for the feature to work correctly.
- Password policies in the on-premises AD DS environment might prevent password resets from being correctly processed. If you're testing this feature and want to reset passwords for users more than once per day, the group policy for the minimum password age must be set to 0. You can find this setting in the following location: Computer Configuration > Policies > Windows Settings > Security Settings > Account Policies, within gpmc.msc.
- If you update the group policy, wait for the updated policy to replicate, or use the
- For passwords to be changed immediately, the minimum password age must be set to 0. However, if users adhere to the on-premises policies, and the minimum password age is set to a value greater than 0, password writeback doesn't work after the on-premises policies are evaluated.
- If you update the group policy, wait for the updated policy to replicate, or use the
Submit and view feedback for