question

RaghuRayala-7358 avatar image
0 Votes"
RaghuRayala-7358 asked MartinJaffer-MSFT commented

Pipeline data flow activity removes the complete column from json when the column value is Null

I am using Data flow activity from azure Data factory pipeline to insert some data in to the cosmos db, and stuck with an issue with having null values for a column in the Json, as the output is currently ignoring properties in the Json which has null values. Can we not store the null value in to json column, as I don't want to replace that with either empty or another value, as this will have more implications to further down the application.

Here is my preview of Data set

135830-image.png

complete data flow

135839-image.png



What do I need to do in order to retain the columns in the JSON which has null values

azure-data-factoryazure-cosmos-db
image.png (10.1 KiB)
image.png (53.9 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @RaghuRayala-7358 and welcome to Microsoft Q&A.

I think I have reproduced your issue.

Given source data like:
{
"A":2,
"B":null,
"C":"foo"
}

after pushing to Cosmos (SQL) I see like
{
"A":2,
"C":"foo"
}

I suspect this lies somewhere in the writing mechanism. If that be the case, then I don't have a ready work-around.
Fortunately, null is what you should get when asking for a property that doesn't exist.

To better understand, can you tell me whether the property is missing from all records, or only the ones where it should be null?

0 Votes 0 ·

0 Answers