Management operations on Event Grid fails with "Internal Error"

Question

We are using EventGrid and have seen an increase in errors the last week.

The errors look like this

{
	"id": "https://management.azure.com/subscriptions/xxx/providers/Microsoft.EventGrid/locations/westeurope/operationsStatus/617D6122-1E6C-41B2-9F91-1037F356173E?api-version=2022-06-15",
	"name": "617d6122-1e6c-41b2-9f91-1037f356173e",
	"status": "Failed",
	"error": {
		"code": "Internal error",
		"message": "The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 8b473daf-4d32-43fa-9971-3c67d2ab0a8b:8/28/2023 12:21:01 PM (UTC) to our forums for assistance or raise a support ticket ."
	}
}

More details about the issue:

• We have seen the error across multiple different EventGrids and spread out thoughout the day. Examples of other failures:

"9bfc5bf0-80de-4ce4-a4da-cb24f77a2960:8/28/2023 7:03:28 AM (UTC)"

"5898ee3b-4c14-4226-9fcd-83bfc8801ae3:8/25/2023 8:18:14 AM (UTC)"

• We see the error randomly, so its not failing every time, but often enough to get our attention.
• We see the error when creating, updating or deleting event grid subscriptions using the Azure.ResourceManager.EventGrid NuGet package.

Any help to resolve the issue or get an idea of how to mitigate it would be highly appreciated.

Regards

/Kristoffer

Answer 1

Hi Kristoffer Steen Pagels Thanks for your patience, this was caused by Event Grid being throttled by Cosmos DB. We increased our capacity to mitigate this issue. could you please verify and let me know if this is still happening for you.

Feel free to add a comment and if the above answer helped with your question, please "Accept as answer". That would really help others in the community with similar interests. Thanks again for posting it in Q&A.