Creating Tables with Different Schedules

  • Article

 

Important

This topic and its sub-topics provides the JSON format that is supported by the older versions of Azure PowerShell. If you are using the July 2015 Release version of Azure PowerShell or later, see Creating Tables with Different Schedules for the latest JSON format. You can convert the JSON from old format to new format by using the tool: JSON Upgrade Tool

This topics covers the following two scenarios:

  • Input table with weekly schedule and output table with daily schedule

  • Input with an arbitrary schedule

Input table with weekly schedule and output table with daily schedule

While creating Azure Data Factory pipelines, you can have different schedules (hourly, daily, weekly, and monthly, etc…) on input and output tables. One of the scenarios is to have input tables available weekly and output tables produced daily. The slicing mechanism in Azure Data Factory makes it very easy to depend on a weekly dataset. You can depend on a day in the past week because input slices are weekly and the dependency condition would be met when the past week’s data is ready.

Here is how you can configure your input tables in Azure Data Factory for this scenario:

"inputs": [
    {
        "name": "inputTable",
        "startTime": "Date.AddDays(SliceStart, -7)",
        "length": "1.00:00:00"
    } 
]

OR

"inputs": [
    {
        "name": "inputTable",
        "startTime": "Date.AddWeeks(SliceStart, -1)",
        "endTime": "Date.AddWeeks(SliceEnd, -1)"
    } 
]

Warning

See Data Factory Functions topic for details about functions (AddWeeks etc…) used in this section.

Input with an arbitrary schedule

You can also have your input table produced on an arbitrary schedule and output tables being produced daily. A general solution regardless of the input schedule is to calculate the slice that corresponds to the past week. Here is how you can configure your input tables for this case:

"inputs": [
    {
        "name": "inputTable",
        "startTime": "Date.AddDays(Date.StartOfDay(SliceStart), -7 - Date.DayOfWeek(SliceStart))",
        "length": "7.00:00:00"
    } 
]

Or (for a daily/hourly/minute output schedule)

"inputs": [
    {
       "name": "inputTable",
        "startTime": "Date.AddDays(Date.StartOfDay(SliceStart), -7 - Date.DayOfWeek(SliceStart))",
        "endTime": "Date.AddDays(Date.StartOfDay(SliceEnd), 0 - Date.DayOfWeek(SliceEnd))" 
    } 
]

See Also

Datasets
Structure Element
Location Element
Availability Element
Policy Element
External Tables