question

Exie-7304 avatar image
0 Votes"
Exie-7304 asked ShreelakshmiCR-5113 published

No data was found for preview

Using terraform, I've created an IoT hub and a Stream analytics job.

The input verifies correctly, the IoT hub shows data coming in.
111364-screen-shot-2021-07-02-at-93544-pm.png


However when I select Query to test my streaming job, it always says there is no data available in the input.
111385-image.png



When I manually created everything via the UI, it was working. So I am confident the JSON payload is correct and the device is registered correctly.

What am I missing ?

azure-iot-hubazure-stream-analytics
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @Exie-7304 ,

Thanks for all the documentation provided.

I have no experience with Terraform too.


A lot of investigation is done towards the message formatting. But did you check the IoTHub consumer group?

By default, the $default consumer group is used.

Because the built-in eventhub compatible endpoint is used, this endpoint is shared amongst other resources (including tooling like the IoT Explorer).

It's recommended to create a unique consumer group for the Stream Analytics job.

Alternatively, you could test with a separate IoT Hub route pointing to an additional Event hub. PErhaps this will provide more insights.

0 Votes 0 ·

Thanks, I did look at the consumer groups.

I added a new one and changed the input so it was not $default

However it had no effect on the data coming through.

1 Vote 1 ·
Exie-7304 avatar image
0 Votes"
Exie-7304 answered

Here's the terraform script to show what I've configured so far.

 # Configure the Azure provider
 terraform {
   required_providers {
     azurerm = {
       source  = "hashicorp/azurerm"
       version = "~> 2.65"
     }
   }
    
   required_version = ">= 0.14.9"
 }
    
 provider "azurerm" {
   features {}
 }
    
 locals {
   tags = {
     Owner  = "Me"
   }
 }
    
 //--------------------------------------
 //  Resource Group
 //
 resource "azurerm_resource_group" "rg" {
   name     = "MyRG"
   location = "australiaeast"
   tags = local.tags
   lifecycle {
     ignore_changes = [
       tags
     ]
   }
 }
    
 //--------------------------------------
 //  IoT Hub
 //
 resource "azurerm_iothub" "iothub" {
   name                = "My-IoTHub"
   resource_group_name = azurerm_resource_group.rg.name
   location            = azurerm_resource_group.rg.location
    
   sku {
     name     = "B1"
     capacity = "1"
   }
    
   tags = local.tags
   lifecycle {
     ignore_changes = [
       tags
     ]
   }
 }
    
 //--------------------------------------
 //  Database Server
 //
 resource "azurerm_mssql_server" "sqlserver" {
   name                         = "my-sql-server"
   resource_group_name          = azurerm_resource_group.rg.name
   location                     = azurerm_resource_group.rg.location
   version                      = "12.0"
   administrator_login          = var.administrator_login
   administrator_login_password = var.administrator_password
   minimum_tls_version          = "1.2"
    
   tags = local.tags
   lifecycle {
     ignore_changes = [
       tags
     ]
   }
 }
    
 resource "azurerm_mssql_firewall_rule" "example" {
   name             = "FirewallRule1"
   server_id        = azurerm_mssql_server.sqlserver.id
   start_ip_address = "0.0.0.0"
   end_ip_address   = "0.0.0.0"
 }
    
 //--------------------------------------
 //  Database
 //
 resource "azurerm_mssql_database" "database" {
   name           = "my-db"
   server_id      = azurerm_mssql_server.sqlserver.id
   collation      = "SQL_Latin1_General_CP1_CI_AS"
   max_size_gb    = 1
   read_scale     = false
   sku_name       = "Basic"
   zone_redundant = false
    
   tags = local.tags
   lifecycle {
     ignore_changes = [
       tags
     ]
   }
 }
    
 //--------------------------------------
 //  Streaming job - Output (to database)
 //
 resource "azurerm_stream_analytics_output_mssql" "output" {
   name                      = "my-job-output"
   stream_analytics_job_name = azurerm_stream_analytics_job.iot-to-db.name
   resource_group_name       = azurerm_stream_analytics_job.iot-to-db.resource_group_name
    
   server   = azurerm_mssql_server.sqlserver.fully_qualified_domain_name
   user     = azurerm_mssql_server.sqlserver.administrator_login
   password = azurerm_mssql_server.sqlserver.administrator_login_password
   database = azurerm_mssql_database.database.name
   table    = "MyTable"
 }
    
 //--------------------------------------
 //  Streaming job - Input (from MQTT)
 //
 resource "azurerm_stream_analytics_stream_input_iothub" "input" {
   name                         = "my-input"
   stream_analytics_job_name    = azurerm_stream_analytics_job.iot-to-db.name
   resource_group_name          = azurerm_stream_analytics_job.iot-to-db.resource_group_name
   endpoint                     = "messages/events"
   eventhub_consumer_group_name = "$Default"
   iothub_namespace             = azurerm_iothub.iothub.name
   shared_access_policy_key     = azurerm_iothub.iothub.shared_access_policy[0].primary_key
   shared_access_policy_name    = "iothubowner"
    
   serialization {
     type     = "Json"
     encoding = "UTF8"
   }
 }
    
 //--------------------------------------
 //  Streaming job
 //
 resource "azurerm_stream_analytics_job" "my-job" {
   name                                     = "my-job"
   resource_group_name                      = azurerm_resource_group.rg.name
   location                                 = azurerm_resource_group.rg.location
   compatibility_level                      = "1.2"
   data_locale                              = "en-AU"
   events_late_arrival_max_delay_in_seconds = 60
   events_out_of_order_max_delay_in_seconds = 50
   events_out_of_order_policy               = "Adjust"
   output_error_policy                      = "Stop"
   streaming_units                          = 1
    
   tags = local.tags
   lifecycle {
     ignore_changes = [
       tags
     ]
   }
    
   transformation_query = <<QUERY
     SELECT *
     INTO [output]
     FROM [input]
 QUERY
    
 }


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered HimanshuSinha-MSFT commented

Hello @Exie-7304,
Thanks for the ask and using the Microsoft Q&A platform .
I am not terrform literate but then I have a hypothesis that may the message coming in are having a different serialization format . At this time its is set to


serialization {
type = "Json"
encoding = "UTF8"
}


This what i tried on my side I have a working ASA with EH with serialization format as JSON , I update the serialization of somethig else . The message stopped showing up on the ASA query .

You can test serialization from the portal UI .

111436-image.png


Please do let me know how it goes .
Thanks
Himanshu
Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members



image.png (88.9 KiB)
· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you for the suggestion Himanshu,

I have checked that and can confirm it is set to JSON and UTF-8

I also checked the payload been sent and confirmed it is valid JSON and should be encoded correctly.

0 Votes 0 ·

After some more digging, I discovered that:
1. Creating a new Streaming Analytics Job via the UI still presented the same problem.
2. Creating a new IoT hub via the UI and creating a new "input" works.

So I am lead to believe the problem could be in the IoT hub somewhere/somehow.

I cant put my finger on what is different between the one I created manually in the portal vs what I have configured in terraform. Clicking throught he UI they appear identical.

Many of the examples I can find also configure a azurerm_storage_account but I didn't consciously do that when configuring an IoT Hub via the portal, and cant find any reference to storage in the one I created manually.


0 Votes 0 ·

On further investigation on key difference between the good version (manual click-ops) vs the faulty (terraform) version appears to be a block for storageEndpoints seen here:
111348-screen-shot-2021-07-03-at-103619-am.png



I've just updated the Aure provider to 2.66 but still not sure if the issue is in the Azure API or in the provider. I feel like I'm getting in pretty deep at this point and it feels like a bug in here somewhere.

0 Votes 0 ·

Just to extend on that - storageEndpoints is been configured via the portal UI (click-ops) seen on the left, but is NOT configured by terraform as seen on the right.

Looking at the provider source code, it seems to have some provision for the storageEndpoint here:
https://github.com/terraform-providers/terraform-provider-azurerm/blob/090fa6506f7693306b53b613c08ff6de86fc64e2/azurerm/internal/services/iothub/iothub_resource.go#L559

But I do not understand Golang well enough to test/debug this.

0 Votes 0 ·

Thanks @Exie-7304 for sharing all teh details here .
if you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.
Subscription ID:
Subject : Attn Himanshu
Please let me know once you have done the same.

Thanks
Himanshu

2 Votes 2 ·
BilalMalik-5429 avatar image
0 Votes"
BilalMalik-5429 answered ShreelakshmiCR-5113 published

@Exie-7304 did you find the solution, if yes please share here, I am also facing the same issue via terraform

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@Exie-7304, The same problem exists for me. please let me know if you get a solution. Is there any way to check whether there is a connection properly set up to the event hub, if the connection is successful then is the message reaching? if yes .,is the message reaching in the right form.?

0 Votes 0 ·