Tutorial: Create a staged data analytics solution with Azure and Azure Stack

Applies to: Azure Stack integrated systems and Azure Stack Development Kit

Learn how to use both on-premises and public cloud environments to meet the demands of multi-facility enterprises. Azure Stack offers a rapid, secure, and flexible solution for collecting, processing, storing, and distributing local and remote data. This is important when security, confidentiality, corporate policy, and regulatory requirements may differ between locations and users.

In this pattern, your customers are collecting data that requires analysis at the point of collection so that quick decisions can be made. Often this data collection occurs with no Internet access. When connectivity is established, you may need to do a resource-intensive analysis of the data to gain additional insight. You can still analyze data when a public cloud is too late or unavailable.

In this tutorial, you'll build a sample environment to:

  • Create the raw data storage blob.
  • Create a New Azure Stack Function to move clean data from Azure Stack to Azure.
  • Create a Blob storage triggered function.
  • Create an Azure Stack storage account containing a blob and a queue.
  • Create a queue triggered function.
  • Test the queue triggered function.

Tip

hybrid-pillars.png
Microsoft Azure Stack is an extension of Azure. Azure Stack brings the agility and innovation of cloud computing to your on-premises environment and enables the only hybrid cloud that allows you to build and deploy hybrid apps anywhere.

The whitepaper Design Considerations for Hybrid Applications reviews pillars of software quality (placement, scalability, availability, resiliency, manageability, and security) for designing, deploying, and operating hybrid applications. The design considerations assist in optimizing hybrid application design, minimizing challenges in production environments.

Prerequisites

Some preparation is required to build this solution:

  • An installed and functioning Azure Stack. For more information, see the Azure Stack overview article.

  • An Azure subscription. You can create a free account.

  • Download and install the Microsoft Azure Storage Explorer.

  • You'll need to provide your own data to be processed by the functions. Data must be generated and available to upload to the Azure Stack storage blob container.

Issues and Considerations

Scalability considerations

Azure functions and storage solutions scale to meet data volume and processing demands. For Azure scalability information and targets, see Azure scalability documentation.

Availability considerations

Storage is the primary availability consideration for this pattern. Connection via fast links is required for large data volume processing and distribution.

Manageability considerations

Consider how your development tools and source control will enable you to manage your solution.

Create the raw data storage blob

The storage account and blob container will hold all original data generated by on-premises activities, including machine and employee activity, facility data, production metrics, and other reporting.

  1. Sign in to the Azure Stack portal.

  2. In the Azure Stack portal, expand the menu on the left side to open the menu of services and choose All Services. Scroll down to Storage and choose Storage accounts. In the Storage Accounts window, choose Add.

  3. Use the following information for the account:

    a. Name: Your choice

    b. Deployment model: Resource Manager

    c. Account kind: Storage (general-purpose V1)

    d. Location: West US

    e. Replication: Locally redundant storage (LRS)

    f. Performance: Standard

    g. Secure transfer required: Disabled

    h. Subscription: Choose one

    i. Resource group: Specify a new resource group or select an existing resource group

    j. Configure virtual networks: Disabled

  4. Select Create to create the storage account.

    Create storage account in Azure Stack

  5. Once created, select the name of the storage account.

  6. In the account blade, under the BLOB SERVICE heading, select Containers.

  7. At the top of the blade, select + Container. and select Container.

    Select container in Azure Stack

  8. Name: Your Choice

  9. Public access level: Container (anonymous read access for containers and blobs)

  10. Select OK.

Create an Azure Stack function

Create a new Azure Stack function to move clean data from Azure Stack to Azure.

Create the Azure Stack function app

  1. Sign into the Azure Stack portal.

  2. Select All services.

  3. Select Function Apps in the Web + Mobile group.

  4. Create the function app using the settings specified in the table below:

    Setting Suggested value Description
    App name Globally unique name Name that identifies your new function app. Valid characters are a-z, 0``-9, and -.
    Subscription Your subscription The subscription under which this new function app is created.
    Resource Group
    myResourceGroup Name for the new resource group in which to create your function app.
    OS Windows Serverless hosting is currently only available when running on Windows.
    Hosting plan
    Consumption plan Hosting plan that defines how resources are allocated to your function app. In the default Consumption Plan, resources are added dynamically as required by your functions. In this serverless hosting, you only pay for the time your functions run.
    Location Region nearest you Choose a region near you or near other services your functions access.
    Storage account
    <storage account created above> Name of the new storage account used by your function app. Storage account names must be between 3 and 24 characters in length. The name may only use numbers and lowercase letters. You can also use an existing account.

    Example:

    Define new function app settings

  5. Select Create to provision and deploy the function app.

  6. Select the Notification icon in the upper-right corner of the portal and watch for the Deployment succeeded message.

    Deployment succeeded - new function

  7. Select Go to resource to view new function app.

View new function app

Add a function to the Azure Stack function app

  1. Create a new function by clicking on Functions, then the +New Function button.

    Create a new function

  2. Select Timer Trigger.

    Timer trigger for new function

  3. Select C# as the language and name the function: upload-to-azure. Set the schedule to 0 0 * * * *, which in CRON notation is once an hour.

    New function settings

Create a Blob storage triggered function

  1. Expand the function app and select the + button next to Functions.

  2. In the search field, type blob and then choose the desired language for the Blob trigger template.

    Choose the Blob storage trigger template.

  3. Use the settings as specified in the table below:

    Setting Suggested value Description
    Name Unique in your function app Name of this blob triggered function.
    Path <path from the storage location above> Location in Blob storage being monitored. The file name of the blob is passed in the binding as the name parameter.
    Storage account connection Function App Connection You can use the storage account connection already being used by your function app or create a new one.

    Example:

    Create the Blob storage triggered function.

  4. Select Create to create the function.

Test the function

  1. In the Azure portal, browse to the function. Expand the Logs at the bottom of the page and ensure log streaming isn't paused.

  2. Open Storage Explorer and connect to the storage account created at the beginning of this section.

  3. Expand the storage account, Blob containers, and the blob you created earlier. Select Upload and then Upload files.

    Upload a file to the blob container.

  4. In the Upload files dialog box, select the Files field. Browse to a file on a local computer, such as an image file, select it and select Open and then Upload.

  5. Go back to function logs and verify the blob has been read.

    Example:

    View message in the logs.

Create an Azure Stack storage account

Create an Azure Stack storage account containing a blob and a queue.

Storage Blob Data archiving

This storage account will house two containers. These containers consist of one blob used to hold archive data and a queue used for the processing of data assigned for main office distribution.

Use the steps and settings outlined above to create another storage account and blob container as our archive storage.

Storage Queue Filtered Data holding

  1. Use the steps and settings outlined above to access the new storage account.

  2. In the Storage Account Overview section, select Queue.

  3. Select the + Queue and fill-in Name field and fill-in a name for the new queue.

  4. Select OK.

    Storage queue

    Add name to new storage queue

Create a queue triggered function

  1. Use the steps in the above function creation section to create an additional Azure Stack function with a queue trigger instead of a blob trigger.

  2. Use the settings as specified in the table below:

    Setting Suggested value Description
    Name Unique in your function app Name of this queue triggered function.
    Path <path from the storage location above> Location in storage being monitored. The file name of the queue is passed in the binding as the name parameter.
    Storage account connection Function App Connection You can use the storage account connection already being used by your function app or create a new one.
  3. Select Create to create the function.

Test the queue triggered function

  1. In the Azure Stack portal, browse to the function. Expand the Logs at the bottom of the page and ensure log streaming isn't paused.

  2. Open Storage Explorer and connect to the storage account created at the beginning of this section.

  3. Expand the storage account, Blob containers, and the blob you created earlier. Select Upload and then Upload files.

    Upload a file to the blob container.

  4. In the Upload files dialog box, select the Files field. Browse to a file on a local computer, such as an image file, select it and select Open and then Upload.

  5. Go back to function logs and verify the blob has been read.

    Example:

    View message in the logs.

Securely stored and accessed compliant data

Global enterprise data is securely stored, processed, distributed, and accessed via Azure and Azure Stack Staged Data Analytics and custom endpoint directives. Remote office employee and machinery activities, facility data, and business metrics are continually compiled, stored, tested for compliance, and distributed according to company policy and regional regulation.

Next steps