My question is to do with .parquet files.
At present, I have the following process in place.
ADF transfers data from on-prem sql server views into .parquet files which get automatically generated into a blob storage container such as
For every load, an automated .parquet file is generated with fields:
Field1, Field2, Field3 which hold information about the load, etc.
I have created a view which reads the data of the three fields above.
All good so far.
Now, I added an extra field i.e. Field4 into the source view...
I can see the new sink .parquet files do have the extra field in there but the original view to read the .parquet file still only reads the first three fields.
So I re-created the view which reads the .parquet files in the Box/AuditLog directory in blobstorage.
If I delete the old .parquet files (Which do not have Field4), and yet keep the new ones with the new field (Field4), then the select from view shows all four fields.
Question. every time I add a new field, I do not want to have to remove the old .parquet files. Is there a way to read all the .parquet files (Old ones with three fields and the new ones with the new field) at the same time?