question

microracl avatar image
0 Votes"
microracl asked BenMiller-DBAduck commented

SQL Server 2017 transaction log does not shrink logfile after rebuild index

Hello,

I have 10GB database and i have configured transaction log shipping. My rebuild indexes job executed indexes fillfactor changed from 100 to 70 and transaction logfile has growth 60GB.

I have tried so many things to shrink such as

Destroy log shipping configuration bring simple recovery mode and shrink not release also database has above 6000 virtual log file,
I have get full backup and transaction log backup.
I have get full backup and restore for new database.
I remember when I configuring log shipping I have set 72 hour "delete files older than". I don't think because of this log can not shrink, but maybe it can be reason after 72 hour later these virtual logs LSN can be expired and I can shrink.

Also DBCC LOGINFO shows except of 4 files every virtual log file has create LSN.
196166-ekran-resmi-2022-04-25-115423.png







sql-server-general
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

BenMiller-DBAduck avatar image
0 Votes"
BenMiller-DBAduck answered

Transaction log VLFs do not expire, they get cleared after a Log backup. So the setting in retention of 72 hours will not make a difference of whether the VLFs clear to 0 instead of 2 on status.

You may need to issue a CHECKPOINT in your database and then see if that clears the status to 0, then you can shrink the log file.

Log backups are the only way to clear the status. The other way is to clear the log, is to change the Recovery Model to SIMPLE, but then to get FULL back you would need to change it to FULL and do a FULL Backup so that you establish the FULL recovery model. (This is not the way I would recommend for other reasons, so use this as a last resort, be patient for the log to clear after the log backups)

But if there is an open transaction at the end of the log, then it will not clear that VLF, so you basically have to wait for the transactions to wrap around to the front of the log and after the next log backup if there are no open transactions left in the end of the log then it will clear.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

TomPhillips-1744 avatar image
0 Votes"
TomPhillips-1744 answered

The physical SIZE of the log file never shrinks unless you manually shrink it.

If your database is set to "Full recovery", the log will grow until a log backup is complete. It will then mark the log data backed up INTERNALLY as available for reuse. It does not change the size of the physical file.

Unless you need "point in time" recovery, you can usually set the database to "Simple recovery". This will mark log space for reuse as soon as the transaction is complete.

See:

https://docs.microsoft.com/en-us/sql/relational-databases/backup-restore/recovery-models-sql-server?view=sql-server-ver15

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

microracl avatar image
0 Votes"
microracl answered microracl commented

Thanks, dears i understand but my problem completely different. As i told i already set simple take backup, set full take level 0 take tx log. I did all what you described. But somehow SQLServer doesnt not clear vlf thatswhy not shrink. Below is output take full backup, tx log backup and shrink.

BACKUP DATABASE [XYZ] TO DISK = N'L:\BACKUP\XYZ\XYZ.bak' WITH NOFORMAT, NOINIT, NAME = N'XYZ-Full Database Backup', SKIP, NOREWIND, NOUNLOAD, STATS = 10
GO

checkpoint

BACKUP LOG [XYZ] TO DISK = N'L:\BACKUP\XYZ\XYZ.trn' WITH NOFORMAT, NOINIT, NAME = N'XYZ-Full Database Backup', SKIP, NOREWIND, NOUNLOAD, STATS = 10
GO

USE
XYZ
GO
DBCC SHRINKFILE (N'XYZ_LOG' , 1)
GO

USE
master
GO
ALTER DATABASE XYZ MODIFY FILE ( NAME = N'XYZ_LOG', SIZE = 4096MB, FILEGROWTH = 256MB )
GO


 60 percent processed. 
 70 percent processed. 
 80 percent processed. 
 90 percent processed. 
 100 percent processed. 
 Processed 7953125 pages for database 'XYZ', file 'XYZ_LOG' on file 1. 
 Processed 115866 pages for database 'XYZ', file 'XYZ_LOG2' on file 1. 
 BACKUP LOG successfully processed 8068991 pages in 168.457 seconds (374.214 MB/sec). 
 Cannot shrink log file 2 (XYZ_LOG) because of minimum log space required. 
     
 (1 row affected) 
 DBCC execution completed. If DBCC printed error messages, contact your system administrator. 
 Msg 5039, Level 16, State 1, Line 18 
 MODIFY FILE failed. Specified size is less than or equal to current size. 
     
 Completion time: 2022-04-25T18:29:11.3225585+03:00
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

First of all, you will not be able to shrink the log file as long as everything is 2 in status, it just won't happen.
If all of the VLFs are status 2 then there is something else going on like CDC or Replication on that database and the replication is keeping it from changing from status 2 to 0.

 select name, is_published
 FROM sys.databases
 WHERE name = 'XYZ'

Having 2 log files is not necessary either, SQL will not write to multiple log files simultaneously, it will be linear from one to the next when it fills up. Typically a second log file is only used when a disk runs out of space, but you will not be able to shrink a log file until status = 2 on the VLFs anyway, regardless of which log file you try it on.

I will say I have seen this happen on SQL 2017 (not sure CU), but I had to restart SQL Server to get them to clear. I am not sure what the issue was, but that is what I had to do. The SQL Server I was dealing with was in an Availability Group so I am not sure whether you are in the same situation.

Things to check:

  1. Check if Replication / CDC is configured on this database

  2. Check if Availability Groups is enabled and the Secondary is not receiving logs (check the dashboard)

  3. Check to see how big the log backups are that you take to see if there is log records flowing through even though it does not change status = 0





0 Votes 0 ·
microracl avatar image microracl BenMiller-DBAduck ·

Sorry i didnt see your messages not any replication, cdc, always on, log shipping. This is standalone instance any no any disaster solution.
query returns is_published=0

0 Votes 0 ·
AmeliaGu-msft avatar image
0 Votes"
AmeliaGu-msft answered microracl edited

Hi microracl,

Welcome to Microsoft Q&A.
It seems the status of VLF is active (status=2), which means they cannot be truncated.
Please use the query to check it there are any open-running or uncommitted transactions:

 SELECT [s_tst].[session_id],
 [database_name] = DB_NAME (s_tdt.database_id),
 [s_tdt].[database_transaction_begin_time], 
 [sql_text] = [s_est].[text] 
 FROM sys.dm_tran_database_transactions [s_tdt]
 INNER JOIN sys.dm_tran_session_transactions [s_tst] ON [s_tst].[transaction_id] = [s_tdt].[transaction_id]
 INNER JOIN sys.dm_exec_connections [s_ec] ON [s_ec].[session_id] = [s_tst].[session_id]
 CROSS APPLY sys.dm_exec_sql_text ([s_ec].[most_recent_sql_handle]) AS [s_est];

Msg 5039, Level 16, State 1, Line 18
MODIFY FILE failed. Specified size is less than or equal to current size.

And DBCC SHRINKFILE doesn't shrink a file past the needed stored data size. For example, if 7 MB of a 10-MB data file is used, a DBCC SHRINKFILE statement with a target_size of 6 shrinks the file to only 7 MB, not 6 MB. Please check the doc for more details.

Best Regards,
Amelia


If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.


· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi,

Query has return,

(0 rows affected)

Completion time: 2022-04-26T09:41:28.4734231+03:00


Unfortunately i am thinking this problem not known bug or something different.

0 Votes 0 ·

Hi microracl,

Thanks for your reply.
What is the version of SQL Server?
Could you please check if the VLFs are still in active?

Best Regards,
Amelia

0 Votes 0 ·

Hi,

What have i done virtual log files doesnt comes inactive state. All 7000 virtual log files shows state 2

Microsoft SQL Server 2017 (RTM-CU29) (KB5010786) - 14.0.3436.1 (X64)
Mar 18 2022 13:21:03
Copyright (C) 2017 Microsoft Corporation
Standard Edition (64-bit) on Windows Server 2016 Standard 10.0 <X64> (Build 14393: ) (Hypervisor)

0 Votes 0 ·
TomPhillips-1744 avatar image
0 Votes"
TomPhillips-1744 answered microracl commented

The error you are getting is:

  Cannot shrink log file 2 (XYZ_LOG) because of minimum log space required. 

This is because you are running the command:

 DBCC SHRINKFILE (N'XYZ_LOG' , 1)


The log file cannot be shrunk to 1mb. You must put in a bigger number than 1.

· 8
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Yes exactly because of virtual logfiles active not turning to inactive
I tried bigger size same error

 LogicalName    FileType    FilegroupName    PhysicalFileLocation    FileSizeMB    UsedSpaceMB    FreeSpaceMB
 XYZ_LOG    LOG    NULL    F:\LOG\XYZ_LOG.ldf    71404.00    71349.87    54.13
 XYZ_LOG2    LOG    NULL    F:\LOG\XYZ_LOG2.ldf    3208.00    3205.56    2.44
 XYZ    ROWS    PRIMARY    E:\DATA\XYZ.mdf    10846.31    10653.06    193.25


0 Votes 0 ·

What does this return

 SELECT name, log_reuse_wait_desc FROM sys.databases;


0 Votes 0 ·
microracl avatar image microracl TomPhillips-1744 ·

sometimes waits for checkpoint sometimes waits for oldest_page but when i check and see nothing i tried shrink but not completed. whatever i do virtual log files status not turning to 0. Its always 2

0 Votes 0 ·

Why do you have TWO log files?

0 Votes 0 ·

Can you script your database (create database) and show us your db files (data and log files). Thanks.

0 Votes 0 ·
Show more comments

2 logfiles wasnt always exists. i have created for shrink tests.

0 Votes 0 ·
microracl avatar image
0 Votes"
microracl answered microracl edited

USE [master]
GO

 /****** Object:  Database [XYZ]    Script Date: 26.04.2022 17:40:31 ******/ 
 CREATE DATABASE [XYZ] 
  CONTAINMENT = NONE 
  ON  PRIMARY  
 ( NAME = N'XYZ', FILENAME = N'E:\DATA\XYZ.mdf' , SIZE = 11106624KB , MAXSIZE = UNLIMITED, FILEGROWTH = 65536KB ) 
  LOG ON  
 ( NAME = N'XYZ_LOG', FILENAME = N'F:\LOG\XYZ_LOG.ldf' , SIZE = 73641984KB , MAXSIZE = 204800000KB , FILEGROWTH = 262144KB ),  
 ( NAME = N'XYZ_LOG2', FILENAME = N'F:\LOG\XYZ_LOG2.ldf' , SIZE = 3416064KB , MAXSIZE = 204800000KB , FILEGROWTH = 65536KB ) 
 GO 
     
 IF (1 = FULLTEXTSERVICEPROPERTY('IsFullTextInstalled')) 
 begin 
 EXEC [XYZ].[dbo].[sp_fulltext_database] @action = 'enable' 
 end 
 GO 
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

JingyangLi avatar image
0 Votes"
JingyangLi answered microracl commented

Thanks for posting your db script. You need to review your log file sizes (size, filegrowth, maxsize) and the goal of your shrinking file.

Remove the second log file not used (SQL server will not use it as you are expecting).

ALTER DATABASE [XYZ] REMOVE FILE XYZ_LOG2

Let's see whether you can move forward with your actions.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I added which script i executed and output is below. Unfortunately can not drop logfile. Something wrong or maybe its not known bug ?

0 Votes 0 ·
microracl avatar image
0 Votes"
microracl answered microracl commented
 BACKUP DATABASE [XYZ] TO  DISK = N'L:\BACKUP\XYZ\XYZ.bak' WITH NOFORMAT, NOINIT,  NAME = N'XYZ-Full Database Backup', SKIP, NOREWIND, NOUNLOAD,  STATS = 10
 GO
 --
 checkpoint
 --
 BACKUP LOG [XYZ] TO  DISK = N'L:\BACKUP\XYZ\XYZ.trn' WITH NOFORMAT, NOINIT,  NAME = N'XYZ-Full Database Backup', SKIP, NOREWIND, NOUNLOAD,  STATS = 10
 GO
 --
 checkpoint
    
 USE 
 XYZ 
 GO
 DBCC SHRINKFILE (N'XYZ_LOG' , 4096) 
 GO
 --
 USE 
 XYZ 
 GO
 DBCC SHRINKFILE (N'XYZ_LOG2' , 1) 
 GO
    
 USE 
 XYZ 
 GO 
 ALTER DATABASE XYZ MODIFY FILE ( NAME = N'XYZ_LOG', SIZE = 4096MB, FILEGROWTH = 256MB ) 
 GO
    
 ALTER DATABASE XYZ  REMOVE FILE XYZ_LOG2; 
 GO
    
 10 percent processed.
 Processed 1364888 pages for database 'XYZ', file 'XYZ' on file 1.
 20 percent processed.
 30 percent processed.
 40 percent processed.
 50 percent processed.
 60 percent processed.
 70 percent processed.
 80 percent processed.
 90 percent processed.
 100 percent processed.
 Processed 9820817 pages for database 'XYZ', file 'XYZ_LOG' on file 1.
 Processed 574365 pages for database 'XYZ', file 'XYZ_LOG2' on file 1.
 BACKUP DATABASE successfully processed 11760070 pages in 231.959 seconds (396.085 MB/sec).
 10 percent processed.
 20 percent processed.
 30 percent processed.
 40 percent processed.
 50 percent processed.
 60 percent processed.
 70 percent processed.
 80 percent processed.
 90 percent processed.
 100 percent processed.
 Processed 32772 pages for database 'XYZ', file 'XYZ_LOG' on file 1.
 Processed 8191 pages for database 'XYZ', file 'XYZ_LOG2' on file 1.
 BACKUP LOG successfully processed 40963 pages in 1.257 seconds (254.587 MB/sec).
 Cannot shrink log file 2 (XYZ_LOG) because of minimum log space required.
    
 (1 row affected)
 DBCC execution completed. If DBCC printed error messages, contact your system administrator.
 Cannot shrink log file 3 (XYZ_LOG2) because of minimum log space required.
    
 (1 row affected)
 DBCC execution completed. If DBCC printed error messages, contact your system administrator.
 Msg 5039, Level 16, State 1, Line 27
 MODIFY FILE failed. Specified size is less than or equal to current size.
 Msg 5042, Level 16, State 2, Line 30
 The file 'XYZ_LOG2' cannot be removed because it is not empty.
    
 Completion time: 2022-04-26T22:50:31.9897663+03:00




· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Is it possible for you to stop SQL and start SQL services and see whether that helps? This seems to be a very specific issue and one that I have seen before, but if it is not possible to stop and start SQL Server services then we will have to keep plugging at it.

0 Votes 0 ·
microracl avatar image microracl BenMiller-DBAduck ·

No i can stop but what should i do after stop service ?

0 Votes 0 ·

Just stop and start the SQL Services and then check the VLFs with DBCC LOGINFO and see if they cleared.

0 Votes 0 ·
Show more comments
microracl avatar image
0 Votes"
microracl answered microracl commented

Not changing anything i have tried before but now again i tried again same vlf show status still 2 thatswhy i cannot shrink, drop.
What you have done when you faced before ?

196705-ekran-resmi-2022-04-26-233003.png



· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Basically I stopped and started SQL and then ran a backup of the LOG and they magically cleared. Again my scenario could have been different than this one, but that is what happened to me and it was SQL 2017.

0 Votes 0 ·
microracl avatar image microracl BenMiller-DBAduck ·

Same, mine is Microsoft SQL Server 2017 (RTM-CU29) (KB5010786) - 14.0.3436.1 (X64) Mar 18 2022 13:21:03 Copyright (C) 2017 Microsoft Corporation Standard Edition (64-bit) on Windows Server 2016 Standard 10.0 <X64> (Build 14393: ) (Hypervisor)

What can i do more? I am thinking about take logical backup and import to new database. In oracle i can use datapump for logical backup and its consistency but for sql server generate scripts makes me worry about consistency. Do you have an advice to me ?

0 Votes 0 ·
ErlandSommarskog avatar image
0 Votes"
ErlandSommarskog answered microracl commented

Here is another thing to try. It's quite wild, and if Ben calls out "DON'T DO IT", Ben is right. Thus, don't do this until, Ben has given go ahead.

First you create a database snapshot of this database:

CREATE DATABASE snap ON (NAME = 'snap', FILENAME = 'E:\Data\Snap.mdf') AS SNAPSHOT OF YourDB

Next, possibly you should perform some update in the source database. Adding or updating a single row should do. (And I am not sure that it is required.)

Next you revert to the snapshot:

RESTORE DATABASE YourDB FROM DATABASE_SNAPSHOT = 'snap'

At least it used to be the case when you revert from a database snapshot, the transaction is thrown away and is replaced with one that is mere 0.5 MB size. Thus, you should immediately grow it to a reasonable size.

Whether this will actually work in your case, I don't know.

· 20
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Erland,
Actually i didnt understand your advice how i can gonna make it? Can you share with any doc or forum url for learn whats it?

0 Votes 0 ·

The advice is to run the commands above. But not until Ben approves.

A database snapshot is a readonly version of a database at a given point in time. The logical size is the same as the source database, but it is a so-called sparse file. So originally, it hardly takes up any physical space at all. But as pages in the source database are updated, the original pages are written to the sparse file.

But that is not very relevant in this context. What is relevant is that a database snapshot can also be used as way for a quick restore. And with the special "feature" that the transaction log is thrown away entirely.

0 Votes 0 ·
microracl avatar image microracl ErlandSommarskog ·

So ok then i will be waiting approval from Ben

0 Votes 0 ·
Show more comments