question

FlashGordon-7254 avatar image
0 Votes"
FlashGordon-7254 asked FrankPalmieri-1863 commented

ms sql 2019 community polybase log file filling up disk drive

Hi, I have ms sql community 2019. I noticed an update for ms sql recently. Today my hard drive filled up, normally I have about 1/2 of it or 250 gig free. In Treesize below you see that in folder ~MSSQL -> Log -> Polybase it says that folder is 345 gig, so something has gone nuts. This is a nothing, client based, database I use to remember sql, ssis, and vs 2019 techniques, it is less than a megabyte in size of data. I never used polybase. How do you clear this Polybase log? Would it be OK to just delete the Polybase directory? Or is my only option to uninstall MS Sql? This problem is being fairly well behaved. There are a few gigs of free disk space left and nothing is failing because of no disk space. I think this is the latest ms sql server update is below: KB4583458 - Description of the security update for SQL Server 2019 GDR: January 12, 2021 80862-polybasetreesize.jpg [1]: /answers/storage/attachments/80862-polybasetreesize.jpg 80855-polybasealone.png80872-dumpdirectory.jpg


sql-server-general
polybasetreesize.jpg (305.7 KiB)
polybasealone.png (53.2 KiB)
dumpdirectory.jpg (59.0 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ErlandSommarskog avatar image
0 Votes"
ErlandSommarskog answered FrankPalmieri-1863 commented

This is a known issue that has been fixed. Get the most recent https://support.microsoft.com/en-us/topic/kb5000642-cumulative-update-9-for-sql-server-2019-97ad5c3e-e002-4b6d-b566-698bf70ca44a
here.

Although, if you don't plan to use the Polybase feature, I would consider to turn off the Polybase services (in SQL Server Configuration Manager), or even uninstalling the component.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I tried to install CU 11 to correct this issue on Developer Edition and the install failed because the Polybase database wasn't running and couldn't be stopped by the install. I needed to follow the instructions in this link to enable the TCP/IP connection to SQL Server 2019 with the Connection Manager so that I could get Polybase to start up properly so that I could apply the CU:

https://nielsberglund.com/2019/11/20/fix-polybase-in-sql-server-2019-developers-edition/

0 Votes 0 ·
Cathyji-msft avatar image
0 Votes"
Cathyji-msft answered Cathyji-msft edited

Hi @FlashGordon-7254,

I never used polybase. How do you clear this Polybase log? Would it be OK to just delete the Polybase directory? Or is my only option to uninstall MS Sql?

You do not need to uninstall SQL server instance, suggest you uninstall Polybase feature only , since you do not need to use this feature.

Control Panel\Programs\Programs and Features . Right click MS SQL 2019 setup> Uninstall > only choose the Polybase feature on select feature page.

81001-screenshot-2021-03-24-111421.jpg

I think this is the latest ms sql server update is below: KB4583458

CU 9 (KB5000642) is the latest Cumulative Update for SQL 2019, KB4583458 is the latest GDR for SQL 2019. Suggest you applying CU9 for SQL 2019. Refer to MS SQL server versions list.

ms sql 2019 community polybase log file filling up disk drive

Since these dumps are being created specifically in the polybase subdirectory, you may want to focus specifically on polybase dumps. You need to analyze the dump file to fix the issue about multitudinous polybase dumps being created and this fill up disk space.

But since you do not use Polybase, uninstalling Polybase feature is the best choice.


If the response is helpful, please click "Accept Answer" and upvote it, thank you.



5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

FlashGordon-7254 avatar image
0 Votes"
FlashGordon-7254 answered Cathyji-msft edited

Erland/Cathy,

    Thanks for the prompt reply.   When I turned on the machine this morning this c: disk space looked normal in File Explorer (see screen print below), even though I didn't do anything.  The Polybase and Dump directory still showed 374 gig in File Explorer.  Is it possible that the Polybase disk usage reporting is just spurious?   I actually work integrating different database environments, so Polybase might be interesting, although getting access to db's like Teradata is difficult outside of a big company environment.  But if I have to, I'll uninstall it.


81147-afterdiskspace.jpg



afterdiskspace.jpg (61.5 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Looks normal? With 303 MB left, the disk is about full!

Anyway, slap on CU9 and clean up the dump files. Or well, clean up the dump files before you run the CU, so that you have room to breathe...

What I have found with Polybase it is very unfriendly to use at home, or more precisely outside of a domain. I must be logged in as sa. My normal Windows account does not cut it, although I am sysadmin. This is at home where I only have a workgroup.

0 Votes 0 ·

Erland, Thanks, agree the disk is full. I thought it was 303 gig. I think I have a handle on this now. I'm waiting one day to make sure. I always do a power down reboot overnight. When I report tomorrow I think I will be able to give others some tools that will manage this without deleting Polybase or taking other drastic action. At least until the Microsoft ms sql update comes along.

0 Votes 0 ·

Yeah, I also read 303 GB at first, but then I asked myself, why is that circle blue all the way around? So I looked a little closer at the numbers, and could not get the math together. Until I noticed the unit.

0 Votes 0 ·
FlashGordon-7254 avatar image
0 Votes"
FlashGordon-7254 answered FlashGordon-7254 edited

Ok, this is now understood. Polybase did not just fill my disk drive in one day. What it does do is add a unique1/2 gig Polybase time stamped log file every time you start Management Studio against the database?. Polybase does not reuse this log file, it creates a new one for every Management Studio session. I saved the March polybase log files so everyone can see them below. The breadcrumb on my machine to get to the polybase directory is this:

C:\Program Files\Microsoft SQL Server\MSSQL15.MSSQLSERVER\MSSQL\Log\Polybase\dump

I had ~500 of these files hence the 250 gig on my hard drive. It's easy to fix once you know what's happening. You can delete these files with File Explorer on an admin id. If maintaining it like that bothers you, you could write a script. Kudos to TreeSize for presenting a graphical image of disk usage that allowed me to find this problem. Thanks, Erland and Cathy for the help.

81597-finalpolybasedir.jpg81598-finaldiskspace.jpg



finalpolybasedir.jpg (381.2 KiB)
finaldiskspace.jpg (61.2 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

FlashGordon-7254 avatar image
0 Votes"
FlashGordon-7254 answered FlashGordon-7254 edited

Ok, I'm a liar. When I deleted all prior Polybase log files I could see how many log files got created in a day. I signed on to Management Studio maybe 5 hours ago, I did nothing in Mgmt Studio, and it has created 7 of these 1/2 gig log files just today (see below). So you may have to manage these log files pretty actively, depending on how much free space you have. The ms sql database engine starts if you boot your machine and creates these polybase log files. This problem is only for the release of ms sql community that I am on, 15.0.2080.9. Erland says CU9 fixes this big polybase log file problem.


[1]: /answers/storage/attachments/81645-onedaypolybaselog.png


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I think the easiest way out is install CU9. At least I don't seem to get those log files any more.

0 Votes 0 ·
FlashGordon-7254 avatar image
0 Votes"
FlashGordon-7254 answered ErlandSommarskog commented

I followed Erland's advice and upgraded to CU9. Erland points to CU9 above and the upgrade was fast, easy, and had no technical glitches. MS Sql has stopped adding those 1/2 gig polybase log files. My MS Sql release is 15.0.4102.2 now. Now to try some Polybase.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Good luck with Polybase! Tip: if you are trying this home, outside of a domain, you may have to connect as sa to use Polybase.

0 Votes 0 ·