I have seen numerous posts about using powershell to run Azure Synapse Pipelines but I am looking for a method to execute a Powershell script within Azure Synapse Pipelines.
The use case is that we currently have a Powershell script that queries our Active Directory and then dumps a .csv into a datalake in Azure once day. We then transform that file and use it in an Azure SQL Database. THis goes from Raw (.csv) -> Processed (.parquet) -> Curated (.parquet). The problem is that since the powershell script overwrites the previous file, the only file history we have is in the Processed as we keep the Curated to a single file.
Any thoughts on how to do this?
or upvote
button whenever the information provided helps you.