write pipelineName into blobstorage

Question

question

arkiboys asked Oct 21, '21 KranthiPakala-MSFT commented Oct 25, '21

write pipelineName into blobstorage

Hello,
What is the best way to write to the blobstorage?
At present, I have an audit mechanism where I use a copy activity to write some log details into blobstorage container in parquet format. So in the sink I have the dataset pointing to the blobstorage container. In the mapping I capture the PipelineName. so the audit only contains the pipelinename.
Question:
As you see I am using a copy activity which requires source. And to get this to work I connect to the sql server and in the sql query I just put select 1 so that there is no error.
Is copy activity the only activity to use for this simple audit capture?

azure-data-factory

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2021-10-21T23:09:16.717Z

KranthiPakala-MSFT answered Oct 21, '21 KranthiPakala-MSFT commented Oct 25, '21

Hi @arkiboys,

Thanks for using Microsoft Q&A forum and posting your query.

Yes, you are correct Copy activity is the best solution for your requirement. You may also use other options like Custom Activity and Azure Function activity but they are a bit costly (as they need compute environments like Azure Batch and Azure Function respectively) when compare to copy activity and also you will have to write your own code to achieve this requirement.

Hope this info helps. Do let us know if you have further query.

Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

· 3

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

arkiboys · Oct 22, 2021 at 05:40 AM

Yes, as I am using the copy activity, it does the job, however I have to set the settings for the source in this case. The source I use select 1 inside the query box. But since I only want to put the pipelinename into the sink (blob container), do I have to have the source? For the source I have to have a connection to the sql server but I do not use anything in this case from sql server, hence "select 1" in the query box.
Is this how it should be done? do I have to have a connection to sql server?

Thank you

0 ·

KranthiPakala-MSFT arkiboys · Oct 22, 2021 at 08:10 PM

Hi @arkiboys,

Thanks for your response. As you need to write the pipeline name to your sink, you will have to use a copy activity to achieve it, though you don't need an actual source in this case, you will have a configure a dummy source as Copy activity requires it else the pipeline will fail. You can either have a dummy source pointed to your SQL or a BLOB. In both the cases you don't need the source data but just the pipeline name using a system variable in an expression. And yes this is the best hacky approach to achieve your requirement in a simpler way. And as I mentioned earlier the source connection can be either SQL or a BLOB.

Hope this clarifies. Do let us know if you have further query.

0 ·

KranthiPakala-MSFT KranthiPakala-MSFT · Oct 25, 2021 at 07:09 PM

Hi @arkiboys,

Just checking in to see if the above suggestion was helpful. If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

0 ·

question