HI,
I have previously reported the issue below
I followed the advice and split the queries in order to decrease the number of files in each query.
But I have hit the same issue once again. As I am aware, I shouldn't be hitting +100k files.
Moreover if I try to rerun the query sometimes it succed in the second or third attempt.
Can you somehow see what causes the error?
The files targeted is in parquet, do you think their could be a gain if they were in delta-format instead?
Regards,
Lasse