We run Azure Synapse Serverless on top of Time Series Insight data that are stored as parquet.
It works well, except that the Time Series Insight service appends to parquet files for up to 10 minutes at a time.
During those windows we get an error analyzing the data:
Msg 15813, Level 16, State 1, Line 1
Error handling external file: 'Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.'. File/External table name: 'dbo.foo'.
With openrowset:
Msg 15813, Level 16, State 1, Line 1
Error handling external file: 'Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.'. File/External table name: 'https://foo.dfs.core.windows.net/env-foo/V=1/PT=Time/Y=2021/M=04/foo.parquet'.
Is there a way to ignore those files gracefully in Synapse?
Thanks.