question

ZhuozhenZhao-2837 avatar image
0 Votes"
ZhuozhenZhao-2837 asked MartinJaffer-MSFT commented

how to retrieve parameterized dataset of datafactory by rest api

In azure data factory, we define some parameterized datasets, for example, we define a common parquet dataset with the parquet file path parameterized.
What I wanna do is to retrieve the dataset with an exact parameter value by rest API.
But with the document(https://docs.microsoft.com/en-us/rest/api/datafactory/datasets/get), it seems not able to retrieve the exact dataset with a specific parquet path.
So is there any other rest API can work it out?

azure-data-factory
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @ZhuozhenZhao-2837 and welcome to Microsoft Q&A.

Are you trying to retrieve the data in the file, or the dataset definition?

The document you pointed to is for fetching the dataset definition. Since it is retrieving the definition, specifying the parameter value doesn't make sense. I think I am misunderstanding your ask.

Could you please clarify?

0 Votes 0 ·

Hi Martin, thanks for your reply.
We are building a metadata service that includes data lineage as well. So schema of the dataset is required. But when the dataset is defined as parameterized, the structure returned by the rest API(https://docs.microsoft.com/en-us/rest/api/datafactory/datasets/get) always return an empty schema. So my question is how to specify the parameter values of the parameterized dataset, so that the schema of the specific dataset could be returned by the rest api.

0 Votes 0 ·

@ZhuozhenZhao-2837

In some cases, the dataset schema was never defined.

If the schema was not saved as part of the dataset, then there is nothing to retrieve, it will come back empty.

First try opening the data factory, and going to the schema and importing the dataset. If parameterized it will ask for a value.

Some dataset types like REST do not even keep the schema. I noticed for the REST dataset type, "schema":[] while in a JSON dataset it was "schema":{} . The JSON had option to import schema while REST did not.

Also, be aware this call does not work with Git. This call queries the published items.

0 Votes 0 ·
Show more comments

0 Answers