Hello I am trying to ingest data from an api, I want to try using data flow to transform
and store the data in a sql database.
I successfully copied the json response from the api in a storage account (data lake gen 2).
However I cannot get any preview of the data when mapping it as a source in data flow.
The error is when I want to preview the data as a source in data flow.
at Source 'RawJsonResponse': org.apache.spark.SparkException: Job aborted due to stage failure ...
org.apache.spark.SparkException: Malformed records are detected in schema inference. Parse Mode: FAILFAST.
the entire trace does not seem to have any other useful information.
I have simplified the content to test if it is indeed the json that is malformed, but I can't see anything wrong.
contents of the simplified json
In source settings I have schema drift enabled.
In source options I have Array of documents selected as Document form. ( I have tried the other forms with no success)
What am I doing wrong?