question

bhushangawale avatar image
0 Votes"
bhushangawale asked SankarAzad-8590 published

ADLS Gen2 and Lifecycle Management with Nested Directories

Hello, I am currently exploring lifecycle management policies for ADLS Gen 2 / Hierarchical namespace enabled storage accounts and have a query on it.

For us, there is a root container and different processes are creating content in it by creating their own folders. E.g. rootcontainer/process-1/subfolder-1 and rootcontainer/process-2/subfolder-1/inner-folder-1 and so on.. Note that, It is not a fixed structure but folder hierarchy will be defined by business processes and can be quite deep and names of these folders and subfolders can be different too.

In terms of lifecycle management rules, what we want is - there should be a single rule that takes of deleting / processing all blobs after say 30 days that are stored under these different folders inside the rootcontainer i.e. irrespective of nested folders.
Is it possible since wildcards are not supported in blob prefix while defining the rule? OR the only option is to create a lifecycle management policy targeting each folder?

The reason of defining rule at root container level makes sense is because we do not have control over business processes creating their content inside the rootcontainer and so will not be able to create different rules targeting each folder and subfolder does not sound practical and there may be a possibility to hit the max rules allowed threshold in future.

94130-rule.jpg

Any details or potential workaround or suggestion will be appreciated.


azure-storage-accountsazure-data-lake-storage
rule.jpg (36.6 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @bhushangawale,

Thanks for reaching out and providing details info about the requirement. We are reaching out to internal team to get additional info about this requirement and will get back to you as soon as we have an update from the team.

We appreciate your patience.

0 Votes 0 ·
bhushangawale avatar image
0 Votes"
bhushangawale answered SankarAzad-8590 published

Thanks for your reply. I have tried doing a poc of it and yes though the wildcards are not supported but there is no need to define policy per nested folder to delete it. You can define a policy scoping at a root container level and define your rule, same rule gets applied to all folders within the container so it is inherently recursive.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

After deleting the files will the policy also delete the nested folders.

0 Votes 0 ·
SumanthMarigowda-MSFT avatar image
0 Votes"
SumanthMarigowda-MSFT answered

@bhushangawale, Firstly, apologies for the delay in responding here and any inconvenience this issue may have caused.

Today you would need to define a policy with a rule per nested folder (or path) given as you point out, we don’t support wild cards. The possibility of more flexible conditional statements is something we are considering for a new capability we call Storage Tasks.

Hope this helps!

Kindly let us know if the above helps or you need further assistance on this issue.


Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.