question

aemhD-7221 avatar image
0 Votes"
aemhD-7221 asked SeeyaXi-msft edited

Replication issue

I have a sudden replication issue.
I am working on SQL 2008R2 with Transactional replication with updated transactions, suddenly the data are not synced from the publisher to any subscriber.

I am receiving the following, while checking the replication monitor view, in the Log Reader Agent:
"The process is running and is waiting for a response from the server"
"the process could not execute sp_replcmds "

I have checked both:
- Connection is Correct
- Database security is OK

I have now more that 1 million record resulting from Sp_replcounters

Also, adding the "-Output "D:\VerboseLog\Log.txt" -Outputverboselevel 3" to the job properties doesn't show any additional info.

Swift reply will be appreciated.

sql-server-general
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Soccan avatar image
0 Votes"
Soccan answered
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

TomPhillips-1744 avatar image
0 Votes"
TomPhillips-1744 answered TomPhillips-1744 edited

The simplest answer is to reinit the subscription using a snapshot.

The answer to your question is likely it is timing out while running. This can be caused by many things. Including too many transactions to process at once. When you have lots of transactions, the default settings are not good enough to handle that and you need to tweak the agent settings. I would not do that unless it happens again, or you can't reinit using snapshot.

If you cannot reinit, try changing CommitBatchSize to 5. This will reduce the number of commands requested to be applied to the subscriber. The agent must be restarted to take effect. I would also change from "continuous" mode to a schedule of every 1 min. That will create a log every time it runs, instead of never closing the log file. https://docs.microsoft.com/en-us/sql/relational-databases/replication/agents/replication-distribution-agent?view=sql-server-ver15

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SeeyaXi-msft avatar image
0 Votes"
SeeyaXi-msft answered

Hi @aemhD-7221,

This error occurs most often when you have a large transaction(s).
To begin with, check if there are any open transactions with DBCC OPENTRAN() . Maybe you can get the message which shows Oldest LSN.
Then run the following T-SQL statements to explore the transaction log:

 select  [Transaction ID], count([Transaction ID]) as 'Count log entries'
 from::fn_dblog('YourOldest LSN',null)
 where [Description] = 'REPLICATE' 
 group by [Transaction ID] 
 GO

Note: The fn_dblog() function accepts two parameters, Start LSN and End LSN.

If there is a large number of logs records per given transaction, please increase the QueryTimeOut for the LogReader Agent Profile.
For short-term monitoring, you can also set the Agent ReadBatchSize = 1 and VerboseHistory = 2 to get more feedback from the Agent.
For more information, see the last part of this article which could give your more details.
https://repltalk.com/2019/01/31/the-process-could-not-execute-sp_replcmds/

Hope your problem could be solved!


Best regards,
Seeya


If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

aemhD-7221 avatar image
0 Votes"
aemhD-7221 answered TomPhillips-1744 commented

Thanks everyone,
the error seems to be passed after adjusting the QueryTimeOut, I was changing it from a wrong place.

but I may need you help also with the current status, the log reader seems to be working with such info:

"Delivering replicated transactions, xact count: 1 command count 47001"
&
"The log reader agent is scanning the transaction log for commands to be replicated. approximately 950000 log records have been scanned in pass #2, 0 of which were marked for replication "

but actually, no thing is being replicated!
I left it over 8 hours and no results, or replicated items.
can any one guide me through.. I really can't reinitialize again.


· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

It is still working on the 47000 commands. It is going to take a while.

Make the change I mentioned above. You may even need to change the parameter to 1 until it gets cleared up.

0 Votes 0 ·

Thanks, yes after changing the parameter to 1, it finally deliver the transactions to each subscriber after more than a day.
Many thanks.

Is there any way to rapid the process for each subscriber ? i have over million undistributed commands for each subscriber now.

0 Votes 0 ·

When this problem happens, it is usually faster to reinit the subscribers than wait for the queue to flush all the commands from the queue. Unless your database is very big, or the connection to the subscribers is very slow.

0 Votes 0 ·
SeeyaXi-msft avatar image
0 Votes"
SeeyaXi-msft answered

Hi @aemhD-7221,

"The log reader agent is scanning the transaction log for commands to be replicated. approximately 950000 log records have been scanned in pass #2, 0 of which were marked for replication "

Once the transaction gets committed by the application, the Log Reader Agent needs to scan those huge data changes performed on the Publisher database transaction logs. Thus, you can think of it as a warning.

"Delivering replicated transactions, xact count: 1 command count 47001"

Because the amount of data is too large and the transaction is huge, we may see such information.
This will happen till the Log Reader Agent scans the complete set of changes and identifies that transaction was fully read from the Transactional Logs.

For more information, please see this article. Several ways are post to resolve these problems.
https://codingsight.com/sql-server-transactional-replication-issues/


Best regards,
Seeya

If the answer is the right solution, please click "Accept Answer" and kindly upvote it.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

aemhD-7221 avatar image
0 Votes"
aemhD-7221 answered SeeyaXi-msft converted comment to answer

Thanks, yes it seems working and needed more time, it finally deliver the transactions to each subscriber after more than a day.
Many thanks.

Is there any way to rapid the process for each subscriber ? I have over million undistributed commands for each subscriber now.

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

When this problem happens, it is usually faster to reinit the subscribers than wait for the queue to flush all the commands from the queue. Unless your database is very big, or the connection to the subscribers is very slow.

0 Votes 0 ·

If this happens often, you should change CommitBatchSize to a more reasonable value like 100. This will generally avoid the problem.

0 Votes 0 ·

ok, I have reverted the CommitBatchSize to the default value to 100, hoping to affect the process.
I am really try to avoid reinitialize process, it would be a such headache for such database and subscribers count.

Thanks again for your both kind help

0 Votes 0 ·
SeeyaXi-msft avatar image
0 Votes"
SeeyaXi-msft answered SeeyaXi-msft edited

Hi @aemhD-7221,

There are some other methods for your reference.
Way 1: CREATE New SQL Stored Procedure
It will help execute the Stored Procedure article on the Subscriber instead of replicating all individual data changes that were happening.
Way 2: Configuring MaxCmdsInTran, ReadBatchSize, and ReadBatchThreshold Log Reader Agent Parameters
Way 3: Configuring the Best Values for SubscriptionStreams Parameter
The default value is 1 suggesting only one stream or connection from the distribution to the subscriber database. Values can be any between 1 to 64. If more Subscription streams are added, it might end up on CXPACKET congestion (in other words, parallelism). Hence, you should take care while configuring this option in Production.

For more details and explanation, please see the link i post before.

Best regards,
Seeya

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks.. I have already adjusted the mentioned parameters, but the transaction seems to be highly increasing and nothing is replicated although!

I unfortunately , very sad to say, will have to reintit

So, in general, what to advice for better performance, and avoid such issue that I even can't find what caused it!

0 Votes 0 ·

You apparently have a lot of individual transactions to apply to the subscribers. Setting CommitBatchSize to a value which works is the simplest answer. I would try something like 5 and let it finish and run a few days. Then ramp it up to 10-25 and see how it goes.

Also, increate QueryTimeOut to a huge number like 50000.

0 Votes 0 ·

Also, I highly suggest you do NOT run the distributor in "Continuous", but schedule it to run every few mins. In Continuous mode, it never closes the log and you cannot see the messages easily.

0 Votes 0 ·

Hi @aemhD-7221,

Let's sort it out. > This error occurs most often when you have a large transaction(s).
You can normalize your big transaction so that it is not a single execution, such as creating a stored procedure. It will help execute the Stored Procedure article on the Subscriber instead of replicating all individual data changes that were happening.
Or set the maximum value of some parameters to control the transaction volume.
How to create a stored procedure and set the maximum value is very detailed in the link I provided before. You can read that article, I believe this is helpful to you.
Have a nice day.


Best regards,
Seeya


If the answer is the right solution, please click "Accept Answer" and kindly upvote it.

0 Votes 0 ·