Trying to Flatten My Json using CosmosDB Spark connector - Azure Databricks

Question

question

TomarAbhishek-5641 asked Apr 21, '22 PRADEEPCHEEKATLA-MSFT commented May 2, '22

Trying to Flatten My Json using CosmosDB Spark connector - Azure Databricks

Hi,

Using the below cosmos DB query it is possible to achieve the expected output, but how can I do the same with spark SQL in Databricks.

    COSMOSDB QUERY : select c.ReportId,c.ReportName,i.price,p as provider from c join i in in_network  join p in i.provider

Source JSON

 {
     "ReportId":"F0001",
     "ReportName":"ALYX_HLT",
     "in_network":[
      {"provider":[1,2,3,4],"price":10},
      {"provider":[1004],"price":100.2},
      {"provider":[39,52],"price":3}
     ]
 }

Expected Output

 [
  { "ReportId":"F0001","ReportName":"ALYX_HLT","provider":100,"price":10},
  { "ReportId":"F0001","ReportName":"ALYX_HLT","provider":200,"price":10},
  { "ReportId":"F0001","ReportName":"ALYX_HLT","provider":300,"price":1.3},
  { "ReportId":"F0001","ReportName":"ALYX_HLT","provider":400,"price":23.1},
  { "ReportId":"F0001","ReportName":"ALYX_HLT","provider":500,"price":23.1}
 ]

CosmosDB Spark Connector : https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3_2-12/docs/quick-start.md

Question On Databricks: https://community.databricks.com/s/question/0D53f00001sgowcCAA/trying-to-flatten-my-json-using-cosmosdb-spark-connector-azure-databricks

azure-databricks azure-cosmos-db dotnet-ml-big-data

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT · May 02 at 06:23 AM

Hello @TomarAbhishek-5641,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

Please don't forget to click on or upvote button whenever the information provided helps you.

0 ·

Answer 1 · 2022-04-22T07:30:18.42Z

PRADEEPCHEEKATLA-MSFT answered Apr 22, '22 PRADEEPCHEEKATLA-MSFT edited Apr 28, '22

Hello @TomarAbhishek-5641,

Thanks for the question and using MS Q&A platform.

I see your question is already answered on the Databricks community forum.

Please let us know if any further queries.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

question