How to analyze the cost of each container within the storage account?

Question

question

CaioLopes-2540 asked May 6, '22 SumanthMarigowda-MSFT commented May 19, '22

How to analyze the cost of each container within the storage account?

Hello, I have a need to calculate the cost of each container within a storage account instance, I have been facing this problem for some time. I tried to create a Python script that would traverse all the blobs and calculate the sizes, but it became a completely inefficient process in terms of performance, and I still couldn't get the costs for read and write operations.

Is there any way to get the cost values for each container, including operations and storage values?

I would be very grateful if someone could help me.

azure-storage-accounts azure-blob-storage azure-cost-management

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2022-05-09T09:55:14.42Z

SumanthMarigowda-MSFT answered May 9, '22 SumanthMarigowda-MSFT commented May 19, '22

@CaioLopes-2540 Firstly, apologies for the delay in responding here! Welcome to Microsoft Q&A Forum, Thank you for posting you query here!

You can sure do it via Python or PowerShell but you could also consider leveraging Blob Inventory. Blob inventory will periodically scan your Storage (can be scoped to container or even folder) and calculate total object size and number of objects, amongst other things. The result of this is delivered to another container in the form of CSV or Parquet file.

The best option would be analyze how much data is stored per container (Azure Storage explorer is one option) and use it as alternative to understand usage per container.

Inventory job fails to complete for hierarchical namespace enabled accounts
The inventory job might not complete within 2 days for an account with hundreds of millions of blobs and hierarchical namespace enabled. If this happens, no inventory file is created. If a job does not complete successfully, check subsequent jobs to see if they complete before contacting support. The performance of a job can vary, so if a job doesn't complete, it's possible that subsequent jobs will.

To count the bytes in a storage account you need to iterate over each object in the storage account. Blob inventory does this for you using some tricks on the backend, but you can also call the list blob api to get your own list (i.e. writing code). Both suffer the same problem on larger accounts it is hard to list all the objects in the account. If you write your own code you can use your knowledge of the account to scale out the listing operation and build robust retry logic because listing a large account will be a significant number of calls to the api.

Azure Blob Storage pricing: https://azure.microsoft.com/en-in/pricing/details/storage/blobs/

Understand the full billing model for Azure Blob Storage

Public preview: Multitasking in the cost analysis preview

If the issue is still not rectified, for more specialized assistance on this kindly contact Azure Billing support, it's free, and it's the best choice for you: https://azure.microsoft.com/en-in/support/create-ticket/, https://azure.microsoft.com/en-in/support/options/

Please let us know if you have any further queries. I’m happy to assist you further.

Please do not forget to and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

screenshot-2021-12-10-121802.png (1.4 KiB)

· 6

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

CaiodeOliveiraLopes-6441 · May 09 at 12:04 PM

@Sumarigo-MSFT Thank you very much for your answer, I will try to use the solutions you have indicated, but they still do not meet my need to calculate the cost of the transactions performed in the storage account. Is there a solution for this?

0 ·

SumanthMarigowda-MSFT CaiodeOliveiraLopes-6441 · May 10 at 05:26 AM

@CaiodeOliveiraLopes-6441 You can try the below mentioned suggestion.

Count the size and tier of the objects in the storage account (using blob inventory) and then prorate the transaction cost based on how many GBs each container has.
Parse the storage logs to get an idea of the transactions that each container in the account has had. NOTE: this will yield more accuracy than option #1, however it will not be perfect.
Split the workload into multiple storage accounts. While you will have to deal with the complexity of multiple storage accounts, it will yield the most accurate results.

If the account is small this can work, but as the size of the account grows this gets harder. Moreover, the act of counting will also drive additional transactions. My recommendation is to always evaluate blob inventory first. If that doesn’t work for you and you need to count yourself and you have a large storage account (by # of files), then I would look at multithreading the calls to list blobs using the prefix option and scaling your scan out over many different threads or kindly contact Azure Billing support, it's free, and it's the best choice for your scenario: https://azure.microsoft.com/en-in/support/create-ticket/, https://azure.microsoft.com/en-in/support/options/

Please do not forget to and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

1 ·

screenshot-2021-12-10-121802.png (1.4 KiB)

CaiodeOliveiraLopes-6441 · May 11 at 05:26 PM

@Sumarigo-MSFT

Thanks again for the help. But I still have a doubt, how can I capture the number of operations in a container?

0 ·

SumanthMarigowda-MSFT CaiodeOliveiraLopes-6441 · May 12 at 04:36 AM

@CaiodeOliveiraLopes-6441
Collection and routing: https://docs.microsoft.com/en-us/azure/storage/blobs/monitor-blob-storage?tabs=azure-portal

Please do not forget to and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

1 ·

screenshot-2021-12-10-121802.png (1.4 KiB)

CaiodeOliveiraLopes-6441 SumanthMarigowda-MSFT · May 18 at 07:41 PM

@Sumarigo-MSFT The problem is that in this solution you mentioned, it is not possible to identify the containers that refer to the transactions.

0 ·

Show more comments

question