question

LAgerlund avatar image
0 Votes"
LAgerlund asked SamaraSoucy-MSFT commented

Synapse "There is insufficient system memory in resource pool 'VDWFrontendPool' to run this query"

HI,

I have previously reported the issue below

https://docs.microsoft.com/en-us/answers/questions/388588/synapse-34there-is-insufficient-system-memory-in-r.html#answer-388549

I followed the advice and split the queries in order to decrease the number of files in each query.

But I have hit the same issue once again. As I am aware, I shouldn't be hitting +100k files.

Moreover if I try to rerun the query sometimes it succed in the second or third attempt.

Can you somehow see what causes the error?

The files targeted is in parquet, do you think their could be a gain if they were in delta-format instead?


Regards,

Lasse

azure-synapse-analytics
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi LAgerlund!

I'll be sure to reach out to Pradeep to see if there is any additional info he found that could indicate why this is a continuing issue.

A couple of things that I'm curious about:
- When you divided up the files into separate queries, are they somewhat distributed or is there one that is big enough that it could be causing the same sort of issue?
- Are you running the queries in series or in parallel?

As far as parquet vs delta, it might if you are going to "delete" enough data from the delta lake while keeping the files themselves in place, then maybe. In most cases, it may make things worse as you have to read all the parquet files plus the delta lake transaction logs.

0 Votes 0 ·

Hi Lasse,

I just wanted to check with you- in particular, I'm wondering about the parallel execution of the new queries vs running them in sequence. If you are trying to run them in parallel than that's the likely cause for the error continuing to popup. If not I can investigate further.

0 Votes 0 ·

Hi @LAgerlund,

I wanted to check in with you to see if you have had some time to look at the follow up questions, or if you have been able to resolve your issue yourself.

0 Votes 0 ·
Show more comments

1 Answer

LAgerlund avatar image
0 Votes"
LAgerlund answered SamaraSoucy-MSFT commented

Hi @SamaraSoucy-MSFT

Without changing anything, the jobs have been running without any problems for the last week.

But anyway to answer your question. After splitting up the queries they have been executed in sequence.

II'll let you know if the issue arise again.

Regards,

Lasse

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you for the update. If you are updating the broken up queries in sequence it should have the desired effect of not overwhelming the system. I don't know how much other traffic there is on your pool, but if it reoccurs the next thing I can think of is what other expensive queries might be running at the same time.

0 Votes 0 ·