synapse table partition on single letter i.e Y or N?
In synapse, table reached over 5 billions rows, contain various history of version records, and every record row has status flag of Y or N. Y is active record and N is oldest . Almost 80% rows are history record and only 20% row are active. Majority of…
How to query a 3rd party Azure Data Lake (2nd Gen) via Shared Access Siganture (SAS URL) in Azure Synapse?
I have been granted read only access to a Vendor's azure data lake and given a SAS connect string. How can I query this data via Azure Synapse and store the results in my own data lake? fyi- The vendor's data is stored in JSON file format in this…
Azure data lake 2 parallel load to multiple files and use delta lake
Hi Guys, We have multiple shipment files coming in from multiple clients as below Shipment_2020111801 Shipment_2020111802 Shipment_2020111803 Shipment_2020111804 Shipment_2020111805 Shipment_2020111806 Shipment_2020111807 Each file is…
Azure synapse : How to check transaction log size usage ?
I have few questions for Azure synapse transaction :. How to check the current transaction log size used by Azure synapse db ? How to check the transaction log size used by each query ? How to truncate the transaction log ? or shrink ?
Azure data warehouse vs snowflake
Hello, We would like to move to the cloud, ideally azure and use power bi... We have a lot of ssis packages at present and possibily need to convert to cloud... I have to give a presentation about microsoft azure (synapse, ADF, etc.) vs snowflake …
SQL Pools missing in the data flows source settings
Hi, I have created a couple dedicated sql pools within the synapse workspace with a view to load data from one poll to the other. However, the sql pools doesn't show in the source type (inline). Can you help please? I wonder if I need to add an extension…
External Database Scoped Credential via SAS
Is it possible to create a data base scoped credential in synapse for Azure Blob Storage in SYnapse? Tried this scenario : WITH IDENTITY = 'SHARED ACCESS SIGNATURE', SECRET = '<your SAS secret without the preleading ?>'; CLOSE MASTER KEY; -- only…
Querying flat files (parquet, csv) with different columns
Hi, I'm trying to Query a set of files that are stored in ADL Gen2. Those files have different columns between them like: File 1: Id, Name, LastName File 2: Id, Name, LastName, Phone How could I create an external table without conflicts? …
Visualization/plots/graphs executed successfully on PySpark in Azure Synapse but not showing up! Please help!!
Visualization/plots/graphs executed successfully on PySpark in Azure Synapse but not showing up! Please help!!
Can you connect to an Azure Synapse Analytics Spark cluster through SSMS?
Hi All, I'm new to Azure Synapse Analytics and would like to use the Spark Cluster feature. My team are SSMS SQL users and would like to query the data on a Spark Cluster through SSMS. I know the Serverless SQL endpoint link can be used was…
How to query 3rd party Azure DataLake Gen2 and only store the results
First, what I am trying to do is I want to query and aggregate raw JSON files stored in a 3rd party's Azure Data Lake (Gen2) and store those aggregates in my own data lake or relation db. I do not want to physically copy all of those raw JSON files…
temp table getting dropped in synapse in the same session.
Hi All, I am trying to create temp tables for some analysis on the synapse analytics. While I am able to operate the data from temp table in the same execution but I am unable to access temp table individually in the same session. Just wondering if…
Azure Synapse resource group concept
Please correct me if I am wrong Suppose I have two users in smallrc and two users in xlargerc If both users in smallrc run a query at the same time, user B will need to wait user A to complete before executing , because Slots used by smallrc is…
Azure synapse : is it possible to check each query DWU usage ?
Are there any DMV in Azure synapse that can check each query DWU usage ?
Best options to pull data from SaaS API to Azure Synapse
What are the best options for ingesting data from SaaS Application to Synapse through API with Authentication? We have a couple of SaaS Applications such as NetSuite, EyeShare & SAP SaaS Applications, and planning to ingest the data to Synapse but…
Are composite primary keys supported in Azure Synapse external tables?
Hi. Are composite primary keys in Azure Synapse external tables supported? If yes, can you kindly share the syntax for creating the composite PK? Thank you. Hans Kiamzon
SQL SERVER Source table create point in table of 70 gb table
Hi Guys, We want to create snapshot of an table everyday in azure synapse on sQldw, what would be best approach for daily load of 70GB table from on premise sql,oracle or sap hana, we want load the data in mill seconds max in minutes. Hint -Similar…
Synapse with git integration - publish issues with spark job
I'm having issues with git integration in Synapse. They concern spark job definition in pipelines. When creating new job, the execution policy is saved, commited to git. It is present on collaborative branch after completing pull request. …
No results when querying using Openrowset from ADLS Gen2.
No results when querying using Openrowset from ADLS Gen2. I used to able to query earlier and for the past two weeks it is showing me No results. Any change in permissions? I did follow all the permissions as in below URL. …
Azure synapse : partition a table that has no partition
Is it possible to partition a table (which current not partitioned) , without recreate the table ?