Handle 3 Billion records

Praveen Naregal 1 Reputation point
2021-09-17T11:18:20.233+00:00

We have a view built in Synapse Data warehouse which pulls out data around 3 Billion.
Need to understand how can we handle this data and also if there is other solution on storing this 3 Billion data in Azure.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,395 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee
    2021-09-20T11:01:21.237+00:00

    Hello @Praveen Naregal ,

    Welcome to the Microsoft Q&A platform.

    ‎Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob Storage.

    Azure Data Lake Storage Gen2 supports individual file sizes as high as 190.7 TiB and most of the hard limits for performance have been removed.

    Azure Synapse removes the traditional technology barriers between using SQL and Spark together. You can seamlessly mix and match based on your needs and expertise.

    • Tables defined on files in the data lake are seamlessly consumed by either Spark or Hive.
    • SQL and Spark can directly explore and analyze Parquet, CSV, TSV, Delta Lake, and JSON files stored in the data lake.
    • Fast, scalable data loading between SQL and Spark databases

    133567-image.png

    For more details, refer to Introduction to Azure Data Lake Storage Gen2.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators