question

jvdlinden avatar image
0 Votes"
jvdlinden asked MayankBargali-MSFT commented

Other ways to convert CSV to JSON (more performance)

Hello everybody,

I am using an Azure Logic app to convert a CSV file to JSON to transform the data into a SharePoint Online list.

I am using this exact same approach as described here: https://sergeluca.wordpress.com/2018/10/28/microsoft-flow-advanced-tutorial-creating-a-csv-converter-from-scratch/

It works, but it is so incredibly slow.
I am using a CSV file that has 2.481 rows and 12 columns.
The Logic app takes 6 hours to complete the first For each and to populate the JSON array.
I need to apply Concurrency control and put it to 1 for this For each loop, otherwise Logic app mixes up the wrong values making the JSON array invaluable.

I am quite a newbie to this CSV -> JSON space, and I was looking into tips and tricks on how to increase the processing speed. I did find this article https://social.msdn.microsoft.com/Forums/en-US/acae74f9-4f07-419f-8a03-464b5ca69522/how-to-convert-flat-file-to-json-in-logic-apps?forum=azurelogicapps
I really have 0 experience with Azure Functions or whatever.

I am still wondering, what would be the best approach in my scenario?

I hope someone could point me in a direction.
All help is appreciated.

Thanks!

azure-logic-apps
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @jvdlinden

Following up to see if my answer helps. Do let me know if you any queries.

0 Votes 0 ·

Hi @jvdlinden

I hope the answer helps. Please let me know if you any further queries.

0 Votes 0 ·

1 Answer

MayankBargali-MSFT avatar image
1 Vote"
MayankBargali-MSFT answered MayankBargali-MSFT commented

Hi @jvdlinden

If your files are small (check prerequisites in inline code article) you can write your own inline code to convert csv to json using inline code within Azure Logic App.

Some of the reference article how you can convert csv to array/json. You need to modify and test the code as per your business needs.
https://stackoverflow.com/questions/1293147/javascript-code-to-parse-csv-data/1293163#1293163
https://stackoverflow.com/questions/27979002/convert-csv-data-into-json-format-using-javascript

But in case, if you have a big file to be processed then the suggestion would be calling the function app from the logic app. You can create a function in any of the supported languages

Functions have different combinations for input and output bindings. For example, if you want to send data as http request to function but store the data in the storage blob then the input binding will be 'http' whereas the output binding will be 'blobTrigger'. So according to your business needs, you can choose any of the support languages to create different bindings.

If you are using javascript function then you can use the same sample csv to array/json code in the reference articles.
For C# you can use Netwon.json library for conversion: https://github.com/JamesNK/Newtonsoft.Json

Reference articles that talk about Azure Function csv to json conversion. Please modify and test them according to your needs.
https://gist.github.com/nertim/1367ef1cc4a339b105c6d72aafd7bff4
https://github.com/aaronralls/FunctionAppCSVToJSON
https://github.com/paladique/azure-function-csvtojson

Updated:

I have created the nested script and tested it at my end. Please refer to javascript code and modified if needed according to your needs. I have used '_' for differentiating the nested objects.

Sample Input:

 [
   "name,birthday_day,birthday_month,birthday_year,house_type,house_address_street,house_address_city,house_address_state,house_occupants",
   "Lily Haywood,27,3,1995,Igloo,768 Pocket Walk,Honolulu,HI,7",
   "Stan Marsh,19,10,1987,Treehouse,2001 Bonanza Street,South Park,CO,2"
 ]

Sample Output:

 [
   {
     "name": "Lily Haywood",
     "birthday": {
       "day": "27",
       "month": "3",
       "year": "1995"
     },
     "house": {
       "type": "Igloo",
       "address": {
         "street": "768 Pocket Walk",
         "city": "Honolulu",
         "state": "HI"
       },
       "occupants": "7"
     }
   },
   {
     "name": "Stan Marsh",
     "birthday": {
       "day": "19",
       "month": "10",
       "year": "1987"
     },
     "house": {
       "type": "Treehouse",
       "address": {
         "street": "2001 Bonanza Street",
         "city": "South Park",
         "state": "CO"
       },
       "occupants": "2"
     }
   }
 ]

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.



· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @jvdlinden

I have updated my answer with the source code and updated your comments to a single comment. You can click on edit on any of the comments that you have posted and update them.

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.

1 Vote 1 ·

Hi @MayankBargali-MSFT,

After digging a lot more, I found this article: http://brianvanderplaats.com/2015/10/08/generating-json-from-csv-using-powershell/
Since I am generating the CSV file from an on-premise server I realized that I could use this very easy command to convert my CSV file to JSON format. And it is incredibly quick.

Basically I transferred a small piece of processing from Azure to the on-premise server just by implementing this single cmdlet, and very happy with it.

Still, thanks for your efforts in helping me out!

1 Vote 1 ·

Hi @jvdlinden

Thank you for your kind words.

0 Votes 0 ·

Hi @MayankBargali-MSFT,

Thanks for your reply.
I started with the second source
It seems to work but there is one difference I need to fix.
My initial workflow based on Serge is creating nested objects inside the JSON array. And the one above does not.

Current Logic app output
19739-currentlogicappoutput.txt

And now using the link provided above, I get the output without curly brackets.

This is my JavaScript code which I execute from within my Logic App workflow:
19738-newjavascript.txt

For the JavaScipt code to make nested objects as well?


Edited the comments to make it one.

0 Votes 0 ·