I'm trying to investigate options that will allow us to take the response from an API call (ideally in JSON but possibly XML) through the Copy Activity in to a parquet output.. the biggest issue I have is that the JSON is hierarchical so I need it to be able to flatten the JSON
Initially, I've been playing with the JSON directly to see if I can get what I want out of the Copy Activity with intent to pass in a Mapping configuration to meet the file expectations (I've uploaded the Copy activity pipe and sample json, not sure if anything else is required for play)
On initial configuration, the below is the mapping that it gives me... of particular note is the hierarchy for "vehicles" (level 1) and (although not displayed because I can't make the screen small enough) "fleets" (level 2 - i.e. attribute of vehicle)
The output when run is giving me a single row but my data has 2 vehicles with 1 of those vehicles having 2 fleets.. I have set the Collection Reference to "Fleets" as I want this lower layer (and have tried "", "[*]", "") without it making a difference to output (only ever first row), what should I be setting here to say "all rows"? (If I do the collection reference to "Vehicles" I get two rows (with first Fleet object selected in each) but it must be possible to delve to lower hierarchies if its giving the selection option?? Or is this for multiple level 1 hierarchies only? Is it possible to get to level 2?