Troubleshoot registry performance
This article helps you troubleshoot problems you might encounter with the performance of an Azure container registry.
May include one or more of the following:
- Pull or push images with the Docker CLI takes longer than expected
- Deployment of images to a service such as Azure Kubernetes Service takes longer than expected
- You're not able to complete a large number of concurrent pull or push operations in the expected time
- You see an HTTP 429 error similar to
Too many requests
- Pull or push operations in a geo-replicated registry take longer than expected, or push fails with error
Error writing blobor
Error writing manifest
- Your network connection speed may slow registry operations - solution
- Image layer compression or extraction may be slow on the client - solution
- You're reaching a configured limit in your registry service tier or environment - solution
- Your geo-replicated registry has replicas in nearby regions - solution
- You're pulling from a geographically distant registry replica - solution
Check expected network speed
Check your image size against the maximum supported size and the supported download or upload bandwidth for your registry service tier. If your registry is in the Basic or Standard tier, consider upgrading to improve performance.
For image deployment to other services, check the regions where the registry and target are located. Consider locating the registry and the deployment target in the same or network-close regions to improve performance.
- Azure Container Registry service tiers
- Container registry FAQ
- Performance and scalability targets for Azure Blob Storage
Check client hardware
The disk type and CPU on the docker client can affect the speed of extracting or compressing image layers on the client as part of pull or push operations. For example, layer extraction on a hard disk drive will take longer than on a solid-state disk. Compare pull operations for comparable images from your Azure container registry and a public registry such as Docker Hub.
Review configured limits
If you're concurrently pushing or pulling multiple or many multi-layered images to your registry, review the supported ReadOps and WriteOps limits for the registry service tier. If your registry is in the Basic or Standard tier, consider upgrading to increase the limits. Check also with your networking provider about network throttling that may occur with many concurrent operations.
Review your Docker daemon configuration for the maximum concurrent uploads or downloads for each push or pull operation on the client. Configure higher limits if needed.
Because each image layer requires a separate registry read or write operation, check the number of layers in your images. Consider strategies to reduce the number of image layers.
Configure geo-replicated registry
A Docker client that pushes an image to a geo-replicated registry might not push all image layers and its manifest to a single replicated region. This situation may occur because Azure Traffic Manager routes registry requests to the network-closest replicated registry. If the registry has two nearby replication regions, image layers and the manifest could be distributed to the two sites, and the push operation fails when the manifest is validated.
To optimize DNS resolution to the closest replica when pushing images, configure a geo-replicated registry in the same Azure regions as the source of the push operations, or the closest region when working outside of Azure.
To troubleshoot operations with a geo-replicated registry, you can also temporarily disable Traffic Manager routing to one or more replications.
Configure DNS for geo-replicated registry
If pull operations from a geo-replicated registry appear slow, the DNS configuration on the client might resolve to a geographically distant DNS server. In this case, Traffic Manager might be routing requests to a replica that is network-close to the DNS server but distant from the client. Run a tool such as
dig (on Linux) to determine the replica that Traffic Manager routes registry requests to. For example:
A potential solution is to configure a closer DNS server.
- Geo-replication in Azure Container Registry
- Troubleshoot push operations with geo-replicated registries
- Temporarily disable routing to replication
- Traffic Manager FAQs
If collection of resource logs is enabled in the registry, review the ContainterRegistryRepositoryEvents log. This log stores information for operations such as push or pull events. Query the log for repository-level operation failures.
- Logs for diagnostic evaluation and auditing
- Container registry FAQ
- Best practices for Azure Container Registry
If you don't resolve your problem here, see the following options.