question

grajee avatar image
0 Votes"
grajee asked HimanshuSinha-MSFT commented

Data Lake Organization Pattern

What are the ways in which a data lake can be organized? This is in reference to the Zones and access.

First Option:
RawZone (Container)\Retail\Full
RawZone (Container)\Retail\Incr

StageZone (Container)\Retail\Full
StageZone (Container)\Retail\Incr
CuratedZone (Container)\Retail\Global

RawZone (Container)\Wholesale\Full
RawZone (Container)\ Wholesale\Incr
RawZone (Container)\ Wholesale\ITD

StageZone (Container)\Wholesale\Full
StageZone (Container)\Wholesale\Incr

CuratedZone (Container)\WholeSale\Global


Second Option:
Retail(Container)\RawZone\Full
Retail(Container)\RawZone\Incr
Retail(Container)\RawZone\ITD
Retail(Container)\StageZone\Full
Retail(Container)\GoldZone\Global


Wholesale(Container)\RawZone\Full
Wholesale(Container)\RawZone\Incr
Wholesale(Container)\RawZone\ITD
Wholesale(Container)\GoldZone\Global

SubjectAreas are– Retail, Wholesale, Online, Lease … Rentals.

In the first option, data files will be organized Starting with the Zones
In the second option, data files will be organized Starting with the subject area


The second option is very specific to a SubjectArea and the access rights can be easily assigned to specific groups but the issue is there are many RawZone as many subject areas (same applies to other zones as well). In the first approach, applying RBAC at the Zone level results in the rights getting inherited to the folders resulting in users not belonging to Retail getting access to WholeSale which means we have to use ACLs (?) to remove the unwanted permissions.

Thanks,
grajee

azure-data-lake-storage
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @grajee ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

Hello @grajee ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered

Hello @grajee ,
Thanks for the ask and using the Microsoft Q&A platform .
Let me acknowledge that your ask was very descriptive , and I understand the ask here is to use RBAC or ACL ? Since you are looking for granular permission I could suggest you to go with the ACL . After looking at the folder structure it appears that you will have to manually set the ACL which can be a challenge for so many folders .Not sure if you have already gone through the doc here , sharing it just in case .
.
Thanks
Himanshu
Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.