question

MartinPoeckl-2046 avatar image
MartinPoeckl-2046 asked ·

Increase size of ASA Parquet file output

Is there a way to (further) increase the size of the parquet files written by ASA output? The current settings is "minimum rows: 10000" and "maximum time: 2 hours". However the files are written by ASA every 5 seconds and the size per file is ~10MB. This generates a huge number of small parquet files which is very bad for further processing / analysis and currently requires an additional step of file consolidation.

azure-stream-analytics
1 comment
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @MartinPoeckl-2046 ,

Thanks for using Microsoft Q&A platform and posting your query. I have reached out to internal team to check if there is a way to increase these limits. Will get back to you as soon as I have an update from the team.

Thank you for your patience.

1 Vote 1 · ·

1 Answer

KranthiPakala-MSFT avatar image
KranthiPakala-MSFT answered ·

Hi MartinPoeckl-2046,

I have a confirmation from product team that there is no way to increase that 10,000 rows limit or maximum time limit for now. On ASA side, we don't want to hold too much data in the batch as it could cause the node to crash and fail the job. hence there is a 10,000 rows limit for now. Since it's a hard-limit, there is no way to say allow a customer/subscription to go beyond that limit.

In case if you have high scale of data to handle, I would recommend you to please add a feature request suggestion in Azure Stream Analytics feedback forum: https://feedback.azure.com/forums/270577-stream-analytics

All the feedback shared in this forum are actively monitored and reviewed by ASA engineering team. Also please do share the feature request suggestion link here, so that other users with similar idea can up-vote and/or comment on your suggestion.

Hope this info helps.

Thank you



Please do consider to click on "Accept Answer" and "Upvote" on the post that helps you, as it can be beneficial to other community members.


Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.