question

TomarAbhishek-5641 avatar image
0 Votes"
TomarAbhishek-5641 asked PRADEEPCHEEKATLA-MSFT commented

Trying to Flatten My Json using CosmosDB Spark connector - Azure Databricks

Hi,

Using the below cosmos DB query it is possible to achieve the expected output, but how can I do the same with spark SQL in Databricks.



    COSMOSDB QUERY : select c.ReportId,c.ReportName,i.price,p as provider from c join i in in_network  join p in i.provider


Source JSON

 {
     "ReportId":"F0001",
     "ReportName":"ALYX_HLT",
     "in_network":[
      {"provider":[1,2,3,4],"price":10},
      {"provider":[1004],"price":100.2},
      {"provider":[39,52],"price":3}
     ]
 }


Expected Output

 [
  { "ReportId":"F0001","ReportName":"ALYX_HLT","provider":100,"price":10},
  { "ReportId":"F0001","ReportName":"ALYX_HLT","provider":200,"price":10},
  { "ReportId":"F0001","ReportName":"ALYX_HLT","provider":300,"price":1.3},
  { "ReportId":"F0001","ReportName":"ALYX_HLT","provider":400,"price":23.1},
  { "ReportId":"F0001","ReportName":"ALYX_HLT","provider":500,"price":23.1}
 ]



CosmosDB Spark Connector : https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3_2-12/docs/quick-start.md


Question On Databricks: https://community.databricks.com/s/question/0D53f00001sgowcCAA/trying-to-flatten-my-json-using-cosmosdb-spark-connector-azure-databricks

azure-databricksazure-cosmos-dbdotnet-ml-big-data
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @TomarAbhishek-5641,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT edited

Hello @TomarAbhishek-5641,

Thanks for the question and using MS Q&A platform.

I see your question is already answered on the Databricks community forum.

Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.