Working with large Azure resource data sets
Azure Resource Graph is designed for working with and getting information about resources in your Azure environment. Resource Graph makes getting this data fast, even when querying thousands of records. Resource Graph has several options for working with these large data sets.
Data set result size
By default, Resource Graph limits any query to returning only 100 records. This control protects both the user and the service from unintentional queries that would result in large data sets. This event most often happens as a customer is experimenting with queries to find and filter resources in the way that suits their particular needs. This control is different than using the top or limit Azure Data Explorer language operators to limit the results.
The default limit can be overridden through all methods of interacting with Resource Graph. The following examples show how to change the data set size limit to 200:
az graph query -q "project name | order by name asc" --first 200 --output table
Search-AzGraph -Query "project name | order by name asc" -First 200
In the REST API, the control is $top and is part of QueryRequestOptions.
The control that is most restrictive will win. For example, if your query uses the top or limit operators and would result in more records than First, the maximum records returned would be equal to First. Likewise, if top or limit is smaller than First, the record set returned would be the smaller value configured by top or limit.
First currently has a maximum allowed value of 5000.
The next option for working with large data sets is the Skip control. This control allows your query to jump over or skip the defined number of records before returning the results. Skip is useful for queries that sort results in a meaningful way where the intent is to get at records somewhere in the middle of the result set. If the results needed are at the end of the returned data set, it's more efficient to use a different sort configuration and retrieve the results from the top of the data set instead.
The following examples show how to skip the first 10 records a query would result in, instead starting the returned result set with the 11th record:
az graph query -q "project name | order by name asc" --skip 10 --output table
Search-AzGraph -Query "project name | order by name asc" -Skip 10
In the REST API, the control is $skip and is part of QueryRequestOptions.
When it's necessary to break a result set into smaller sets of records for processing or because a result set would exceed the maximum allowed value of 5000 returned records, use paging. The REST API QueryResponse provides values to indicate of a results set has been broken up: resultTruncated and $skipToken. resultTruncated is a boolean value that informs the consumer if there are additional records not returned in the response. This condition can also be identified when the count property is less than the totalRecords property. totalRecords defines how many records that match the query.
When resultTruncated is true, the $skipToken property is set in the response. This value is used with the same query and subscription values to get the next set of records that matched the query.
The query must project the id field in order for pagination to work. If it is missing from the query, the REST API response will not include the $skipToken.
For an example, see Next page query in the REST API docs.
We'd love to hear your thoughts. Choose the type you'd like to provide:
Our feedback system is built on GitHub Issues. Read more on our blog.