Error when using pandas to write data to ADLS in Azure Synapse Analytics
Hi Mr/Mrs, I want to use pandas to write dataframe to .csv file into ADLS, but I got this issue although I put the access key, it didn't work. How can I fix this issue? Thanks!
How do I setup Azure Synapse Link with SQL Managed Instances
Hi. I'm trying to setup Azure Synapse Link to our SQL Managed Instance and I'm having a hard time finding any documentations or walkthrough/howto on this. It seems like all docs and integration available is for Azure SQL Server only. Is this the case? …
Changing the time zone for a Synapse dedicated SQL pool
Hi, inside a Synapse dedicated SQL pool I need to use the GETDATE() T-SQL function for some queries, but it returns a time minor of two hours respect to the Italian time. This is the datetime returned by the GETDATE() using SSMS: The real time is…
How to test a synapse link service from a powershell script?
How to test a synapse link service from a powershell script?
Synapse can't transfer data from parquet file to Azure SQL Database table
Hi! The pipeline in Synapse (workspace) transfers data from parquet files to a database using a copy activity. Everything has been working fine for months, but today it simply times out without a specific error. It seems the problem lies in…
Join cosmos DB when processing streaming data from eventhub in Synapse
Hi Team, Good morning! Would like to check that when we using spark notebook in Synapse to process Eventhub streaming data, we join the streaming data from Eventhub with CosmosDB data. We try both way of connecting CosmosDB transcational store with OLTP,…
How to translate database content with Azure Translator by ADF or Synapse notebook?
There is an Azure Database table. Some of columns need to be translated from one language to another into additional columns. Such as from English to Spanish, or Portuguese to English, etc. I am exploring how I can use ADF or Synapse notebook to…
How to maintain same folder structure as source while sink the processed file
I have a requirement to process the JSON to parquet on daily basis. I have folder A,B,C needs to sink the file to another container with same structure as A,B,C for example if I'm processing a file from folder A it should sink to output container folder…
Writing binary data to ADLS in Synapse notebook
Hi all, Is it possible to write binary data to the ADLS in Synapse notebook using Pyspark? I retrieve data via an API call where the content type is application/excel (basically binary data) and want to save it to the ADLS, a specific location like…
Synapse Python/Spark Notebook code with SQL to query data from linked service to SQL Server
In Synpase, I have linked service to a SQL Server database. I'm looking for a sample notebook code, either in python or spark, that I can run a complex SQL qery within this code to get data from multiple SQL Server tables, put the rsults in a dataframe…
In ADF, I am not able to merge data from multiple streams with different columns with different data types
Hello, I am not able to merge data from multiple streams in ADF DATAFLOWS. Stream 1) flow let - only 1 column. Stream 2) flow let - only 1 column. Stream 3) source columns (54 columns) Stream 4) regular…
How to save spark dataframe (with synaps) in data container (without making folder and SUCCES file)
I want to save a spark dataframe to my data container. It worked with this code: df.write.csv(path_name + "test5.csv") However, this makes a folder called test5.csv with 2 files in it. One which is my dataframe (but with a random generated…
How to improve the apache pool efficiency ?
I have roughly around 3000 jsons and i am creating a dataframe out of it and saving it to delta format and it is taking me hours for that even though my spark pool is of medium size. The spark pool runs at 33% utilization,
reduce waiting time within a synapse pipeline/ improve performance synapse pipeline
A certain synapse pipeline takes a very long time. When I look at the various activities within this pipeline, the activities themselves don't take much time at all. It's mostly the waiting time between them that is consuming a lot of time. We are using…
Cannot edit an existing data factory in synapse
I have a synapse workspace with several pipelines with data factories. Today I have noticed that existing data factories that are running correctly I cannot see the details when I edit them I attach two examples, one that is working and the other that I…
Integration
Can you provide detailed information on the factors that should be considered for hybrid cloud integration between an on-premises private cloud and Azure, with a focus on scalability, performance, and security?
Dedicated SQL Pool Performance issues
We're utilizing Tableau's live connection feature, linked to a dedicated SQL pool. We've taken all necessary steps from the SQL pool side, such as creating required indexes, statistics, assigning appropriate distribution models and partitioning tables,…
How to allow access for Azure Synapse pipeline to connect to logic app which has public access disabled with Vnet, NSG and user defined route configuration enabled
Hi, I need to call logic app through web activity from a synapse pipeline, get the error as "The web app you have attempted to reach has blocked your access". The synapse workspace has managed virtual network configured and the logic app has…
SAP CDC and SAP Tables connectors regarding SAP Note 3255746 ?
Dear Team, Is there any one who when through the SAP Note 3255746 ? What impact shroud we expect regarding the SAP CDC and SAP Tables connectors already implemented for our customers? Thanks for you input. Tarik
Is the 'tzoffset' function no longer compatible within Synapse?
Up until last night, executing the below script worked within our Synapse instance. However, since last night when I attempt execute the below script I receive the error message underneath. Is the 'tzoffset' function no longer compatible within…