How to read inputs from beginning?

Philipp Marinovic-Brünjes 0

Hi there, currently I'm testing Stream Analytics for a streaming application and facing the issue that my job only reads data that is incoming after it has been started. Data that has been present before won't be processed. I tested this behaviour with Kafka and BLOB storage as input. Is there a way to change this behaviour (similar to Kafka's offset setting)?

1 answer

Sander van de Velde 28,386 Reputation points MVP

2024-01-15T18:02:00.8566667+00:00

Hello @Philipp Marinovic-Brünjes , welcome to this moderated Azure community forum. If I start an existing ASA job, I get this dialog in the portal:

As you can see, I can start fresh by ingesting newly incoming messages or I can start right where I left when the message was stopped. This is seen with the IoT Hub as input but I expect this with other resources too. The Kafka input is in preview, I do not see any other behavior mentioned in the documented limitations.

---If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.
Please sign in to rate this answer.
Philipp Marinovic-Brünjes 0 Reputation points

2024-01-16T08:44:28.7233333+00:00

Thanks for the quick reply @Sander van de Velde . However this isn't really a solution for my case. Choosing now or when last stopped would still miss data that has been in the source before the initial run. Without testing this custom might work but also could lead to duplicates in the output.

I'm consindering to deploy the whole architecture backwards starting at the end. This makes first tests a bit of a hustle though.

Also on a related note: when is Kafka input going live? I couldn't find any date in the docs or Azure's roadmap.

Sander van de Velde 28,386 Reputation points MVP

2024-01-16T20:56:01.18+00:00

Hello @Philipp Marinovic-Bruenjes,

when is Kafka input going live

I'm just a volunteer, don't shoot the messenger ;-)

This is up to the Microsoft product team. Microsoft normally ships it 'when it's ready'. I'm not sure why this 'when last stopped' option should not work for you?

If you suspect this leads to missing or duplicate messages, please submit feedback on this portal.

Philipp Marinovic-Brünjes 0 Reputation points

2024-01-17T13:32:10.66+00:00

Hi @Sander van de Velde attached you can find a diagram trying to explain my problem. Option "when last stopped" would resume just where the asa job stopped (1). Point 2 represents option "now" (basically any given time after stopping the job). However I'm not able to reach point 3. I'd assume that when initially starting the asa job it should pick data up from this point but it starts consuming data at point 2 which leads to missing data. Now when choosing a custom point in time I might be able to get back to point 3 but it would result in consuming all data again up to now and therefore produce duplicates by consuming data between last start and stop. Hope this makes my point clear. BR, Philipp

Sander van de Velde 28,386 Reputation points MVP

2024-01-17T14:59:55.13+00:00

Hello @Philipp Marinovic-Brünjes, with "when last stopped" the ASA service the last moment data was ingested. So if the service is ingesting data and it's stopped at 13:00H, ingestion holds. You can do some maintenance. Once you start the service with 'when last stopped', it will try to ingest all messages made available after 13:00H. So, no data is missed. Azure Stream Analytics works with 'at least once' so you need to be able to cope with duplicates in the flow later. In the past, I tested this with the Azure IoT Hub which has a rention time up to 7 days. There we have seen we were able to see a continous flow.
Sign in to comment