question

ReneLhde-7598 avatar image
0 Votes"
ReneLhde-7598 asked ·

Long term archiving of Azure Activity Logs

Hi,

I have a compliance and audit scenario, where I need to archive Azure Activity Logs (Categories; Administrative, Security and Policy ...at a minimum).
The archive will extend and take me past the build-in 90 days retention in Azure (I will probably need a 5 year retention). I don't think this a unique scenario, yet I can't seem find any information on how other Azure customers or Microsoft accomplish' this - in an Enterprise setup, on a global scale.

Here is what I have tried...

  1. Azure Log Analytics: Using LA as an archive is extremely expensive in my scenario!

  2. Azure Storage (hot): Automatic logging to Azure storage is just plain expensive

  3. Azure Storage (hot -> archive) lifecycle management: Not possible, due to the LA origin is an append blob vs the archive block blob target

  4. Ship all to local storage: Abandoned due to high traffic egress costs

...so I am moving on to my next try which is a variation over 2.)

  • This is going to be a solution were I deploy a small VM in each region, where I have Azure resources running.

  • The VM will contain a script that, downloads the last 90 days append blobs and convert the Activity Logs to block blob and write them to Azure Storage (archive) in the same region as the source storage account, and deletes the source append blobs.

  • An Azure Automation script will start the VM every 90 days and execute the script and shut it down again once the task is complete

This will get my archive on to the most cost efficient storage, while at the same time making sure that any data transport is not egressing.

Okay, so that is what I am going to do - does it sound reasonable? ...but more importantly; is there really no other Azure using enterprise in the World, that has a practice I can copy? ..Microsoft?

Kr, René

azure-monitorazure-storage-accounts
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

olufemiaMSFT avatar image
0 Votes"
olufemiaMSFT answered ·

Hello @ReneLhde-7598, Your proposal sounds reasonable in theory. From my experience, a 5 year retention requirement is on the higher end of the spectrum but I'm sure there are other Azure-based enterprises with similar needs. Typically, the longer the retention period, the higher the rentention costs.

I recommend engaging AzCommunity@microsoft.com, include your subscription ID and link to this thread (for context) and we will gladly assist you offline to explore other feasible options.

Some useful refrerences:

  1. Archive via Storage
    63782-send-to-storage.png

  2. Manage rentention cost by data type
    63832-manage-cost.png

Hope this helps.

Cheers.




send-to-storage.png (70.2 KiB)
manage-cost.png (58.7 KiB)
· 3 ·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @olufemiaMSFT ,

Thank you for your answer - and yes, your suggestions are what I meant by 2) and partly why I discarded 1).
I can definitely send a hyperlink - referencing this thread - to the Microsoft-alias you suggest, but would it not be better to have this conversation in public - I do believe this is a very general question!?

I am honestly puzzled and partly worried, as to why I am not seeing any references to this (very particular, but very general workload) scenario anywhere on the internet. What is the implication of this (lacking) observation? ...my peers simply archive on Azure Storage as you suggest, and "swallow" the cost ...or... my peers don't archive Azure Activity Logs (and are happy with 90 days)? ...other scenarios I am missing?

Kr, René

0 Votes 0 ·

Sincere Apologies for the delayed response, @ReneLhde-7598 - While I agree the question is general, the reality (from experience) is there's typically very customer-specific requirements or environmental or architectural nuances that require deeper review and may influence the final solution. Your above question remains in the public domain where others can provide feedback. In parallel, we have the option to investigate internally and share an update 1:1.
Also note, as a practice we always try to close the discussion loop on public forums when necessary so there's no risk of abandoning this thread if you decide to take the conversation offline.

1 Vote 1 ·

Thanks and no worries. I have had two observations since my last entry. The cost has gone down and of course my employer is pleased with this. I am trying to figure out why and here is what we are noticing:

  1. From a long compliance perspective - we have been "more selective" on Azure Activity Log categories. We have scaled down to 'Administrative', 'Policy', 'Autoscale' and 'Security' - other catagories look to be overhead in our setup

  2. Cost analytic forecast in the Azure Portal sometimes hits way off the mark. I am assuming (and can confirm) that the cost forecast needs more than a week of data to get a yearly forecast. In fact the first yearly forecast based on a week data, was way off.

Kr, René

0 Votes 0 ·