question

suvrajyoti-2505 avatar image
0 Votes"
suvrajyoti-2505 asked MikeUrnun commented

Write data to a csv file using azure function and then upload to sftp location

Hi All,

I have created a Time Triggered Azure Function that runs on a schedule. I am fetching data for API Usage(i am using Azure APIM) using the Azure Rest APIs and want to put it in a csv file. Once the csv file is ready i need to upload it to a SFTP location.

Any pointers on this will be helpful.

Thanks,
SuvraJyoti.

azure-functions
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MikeUrnun avatar image
0 Votes"
MikeUrnun answered MikeUrnun commented

Hi @suvrajyoti-2505 - You can build this very quickly using Logic Apps which provides designer-first services, and it has a concept of plug-and-play Connectors that you can use to go up & running with your scenario:
1. Recurrence trigger (equivalent to TimeTrigger)
2. HTTP Connector for calling your APIM endpoint and fetching data.
3. Perform data operations for the CSV-related data parsing & transformations.
4. SFTP Connector for moving the CSV file to a remote SFTP server.


I suggest Logic Apps here because it's designed to run long-running workflows and can integrate disparate systems (i.e.: APIM, SFTP etc..).

That said, you can certainly use Azure Functions App and build the above in a code-first way as well, in which case, if it were me, I would look to create two Functions in my app where the first one implements steps #1 and #2 above, and the second Function implements steps #3 and #4.


· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks Mike for the response.

Additionally i would like to understand if i need to use the Durable functions here. Since in production env there can be multiple subscriptions and each subscription having access to multiple products and hence apis, so there can be million records per 24 hrs(thats my schedule). I am getting the reports by api operation as that is the need filtered by Subscriptions. I can create a single file with million records or might need to create multiple files by subscription havitng a total of some million records. Do i need to go for Durable function.?

Thanks
SuvraJyoti

0 Votes 0 ·

@suvrajyoti-2505 Yes, Durable Functions would be the way to go, especially if you're pulling 1M records at a time.

1 Vote 1 ·
sadomovalex avatar image
0 Votes"
sadomovalex answered

which programming language you use for AF? If you use C# you may build csv file in memory first and then save it to SFTP e.g. like shown here: SFTP File Upload With C# Application.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.