question

arkiboys avatar image
0 Votes"
arkiboys asked HimanshuSinha-MSFT answered

Synapse design

Hello,
I would like to get your thoughts regarding our following design approach.
1- We have build synapse pipelines
2- Pipelines transfer data from on-prem sql server to Azure blob storage called "Stage1" as .parquet files stored in daily folders
3- Stage2 in the blob storage is populated with full data as a one off load and stored as .parquet files
4- Stage2 is populated daily only with the upserts as .parquet. These are the Delta files and we have set the inline dataset type as Delta
5- In SQL Scripts folder we have scripts to create views, i.e. view1, view2, etc. to read the .Parquet files in Stage2
6- The SQL Scripts are created in the serverless Built-in database and NOT dedicated
7- There is no Workspace being used.
8- Power BI uses the connectionstring to the Built-in database to read the views (Reading .Parquet files) created above.


Q1-What do you think of this design?
Q2- In-particular where power BI reads the views to present data

Thank you

azure-synapse-analytics
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Correction:
In point no 7- I should have said there is no Gen2 used. Instead using blob storage. Apologies
Thank you

0 Votes 0 ·

Correction:
In point no 7- I should have said there is no Gen2 used. Instead using blob storage. Apologies
Thank you

0 Votes 0 ·

1 Answer

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered

Hello @arkiboys ,

Thanks for the ask and using the Microsoft Q&A platform .
The design looks fine to me . The volume of the data matters a lot and we have not talked about the same anywhere .
.
Thanks
Himanshu

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.