Hi,
I'm testing creating Views and External Tables over partitioned data in Azure Storage which is using the Delta Lake format. When I create a View and use the partitioned column in the delta lake to filter then I see reduced data processed (which is good). However when I create an External Table over the same partitioned Delta lake data, the partitioned column appears NULL in the results and filtering does not work.
For example:
CREATE EXTERNAL TABLE LDW.DeltaFactSalesOrder
(
CustomerKey int ,
ProductKey int ,
DueDateKey int ,
ShipDateKey int ,
OrderDate date,
SalesOrderNumber nvarchar(20)
)
WITH
(
LOCATION = 'spark/delta/factinternetsales/',
DATA_SOURCE = ExternalDataSourceDataWarehouse,
FILE_FORMAT = SynapseDeltaFormat
)
OrderDate is the partition column in the Delta Lake (created using the PARTITION BY spark sql syntax) but it's just NULL and doesn't seem to work when filtering.