'InputDeserializerError.InvalidData' | Invalid Avro Format, drop invalid record

Surya Agarwal 21

I am sending AVRO messages using Java Kafka API to event hub, and using stream analytics to route the data to BLOB Storage.

Although after successfully producing the avro message to eventhub, under ASA logs, I can see Diagnostic error.

"ErrorCode": "DiagnosticMessage",  
        "Message": "First Occurred: 6/2/2021 6:13:03 AM | Resource Name: ehubin2 | Message: Source 'ehubin2' had 1 occurrences of kind 'InputDeserializerError.InvalidData' between processing times '2021-06-02T06:13:03.4028487Z' and '2021-06-02T06:13:03.4184861Z'. Invalid Avro Format, drop invalid record.",  
        "Type": "DiagnosticMessage",

Because of this error, no output to BLOB storage. I am successfully able to read using kafka-avro-console-consumer

The reason could be the schema error or Deserilizer issue, But I dont know how to define the schema registry, as I need a schema-registry URL to create subject schema version (to auto update in case of schema change) which I dont find in Eventhub schema registry.

On high level,

'InputDeserializerError.InvalidData' | Invalid Avro Format, drop invalid record in ASA
Schema Registry issue to process avro message in ASA

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-06-02T22:15:32.537+00:00

Hello @Surya Agarwal ,
Thanks for the ask and using the Microsoft Q&A platform .
As you may be aware that the Azure schema registry in EH is still in Preview and I am not sure if you have gone through this link . Does this link help you at all ?

Please do let me know how it goes .
Thanks
Himanshu
Surya Agarwal 21 Reputation points

2021-06-03T16:29:12.227+00:00

Hi Himanshu,

I tried with the following link and am able to send AVRO messages to event hub using Azure avro serializer (com.microsoft.azure.schemaregistry.kafka.avro.KafkaAvroSerializer) with Eventhub schema registry.
But while consuming the messages using ASA, I still get the same error.

Is there any way to provide eventhub schema registry (with schema group) or external schema registry to deserialize the AVRO messages?
(Seeking for a quick and reliable help as I am being charged for services)

Thanks,
Surya
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-06-09T18:28:26.027+00:00

Hello @Surya Agarwal ,
Thanks for the patience , unfortunately we are not getting the kind of response from the team here . if you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.
Subscription ID:
Subject : Attn Himanshu
Please let me know once you have done the same.

Thanks
Himanshu
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-06-16T03:31:13.34+00:00

Hello @Surya Agarwal ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if have already reached out to the Microsoft Support , please do share the SR# , othereise request you to please do the needful as requested in the last response .
Thanks
Himanshu
Surya Agarwal 21 Reputation points

2021-06-18T06:29:12.353+00:00

Hi @HimanshuSinha-msft , I have dropped this approach and moved forward by deserialising the messages before sending to Azure Event Hub. I am now deserialising the Avro message to JSON converting all complex and logical data types, and then after sending to Event Hub.

I am really disappointed that Azure don't have support for Schema Registry if it supports Avro messages in Azure Stream Analytics. Even if it has via some third party tools, there is no documentation to refer.
Thanks a lot for following up!! I will surely raise a support ticket or send an email in case the requirement comes up exactly as in asked question and still I don't see any update from Azure.

Thanks,
Surya Agarwal