Scheduled Task running Powershell Script that calls Robocopy not finishing copy operation

Question

question

Richard-0258 asked · Feb 04 at 09:22 AM

Scheduled Task running Powershell Script that calls Robocopy not finishing copy operation

I have created a PowerShell script that is designed to connect to remote application servers, create a zip archive containing the server's application backup and then Robocopy the zip file back to my file server. There are 53 servers listed in a CSV and the script is working perfectly when run manually. The application backups are correctly zipped on the application servers, the zip files are correctly RoboCopied back to my file server, perfect!

Logic of script is as follows:

Import Server List From CSV
Using a ForEach Loop to :
Invoke-Command against each server in list to Zip up the days application backup on the application server
Create source and destination variables
Initiate Robocopy operation along the lines of -

RoboCopy $copysource $copydestination /MAXAGE:1 /MOV /LOG+:$copylog

The issue arises when the script is called by a Scheduled Task. The RoboCopy operation starts and the zip files begin to copy over the network, however, looking at the RoboCopy log, it just stops at anywhere between 1% or sometimes up to 40%. The scheduled task shows has having completed, but it is abandoning the RoboCopy operation mid copy without generating any error codes.

I have tried all the usual options on the scheduled task, such as running with highest privileges etc, but it clearly isn't a permission thing as it does get some of the way through the copy. It almost feels as though the job is timing out, but I've cleared the "Stop the task if it runs longer than:" setting and it is stopping after only a few minutes, certainly not a few days.

windows-server-powershell

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2021-02-23T11:27:50.457Z

KeepRichard-5302 answered · Feb 23 at 11:27 AM

Does anyone have an answer to this, it's doing my nut in!

·

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 2 · 2021-02-23T13:27:32.143Z

MotoX80 answered · Feb 23 at 01:27 PM

Try using "Invoke-Command -SessionOption " and pass a New-PSSessionOption and setting -OperationTimeout to 0.

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/new-pssessionoption?view=powershell-5.1

· 2 ·

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KeepRichard-5302 · Feb 23 at 02:03 PM

Cheers MotoX80, the issue is that the RoboCopy jobs are all running locally on the file server not as remote PS-Sessions. Each RoboCopy runs in a separate PowerShell job.

Any other ideas?

0 ·

MotoX80 KeepRichard-5302 · Feb 23 at 03:07 PM

Capture stdout/stderr, lastexitcode, and log a transcript.

 Start-Transcript  c:\temp\robo.log
 $src = 'Z:\foobar'  
 $dest = 'Y:\foobar'
    
 get-job | Remove-Job
 $j =Start-Job  -scriptblock {
        
     "Hello, I am Robocopy"
     & robocopy $using:src   $using:dest   /mir 
     "Robocopy RC=$lastexitcode"
        
 } 
 "Waiting"
 wait-job -id $j.Id
    
 "******** Job Output ********************************************************************** "
 Receive-Job -id    $j.id
 "******** End Job Output ********************************************************************** "
 Stop-Transcript

Then look at the log.

 cls
 Get-Content  c:\temp\robo.log

0 ·

Answer 3 · 2021-03-04T10:00:18.947Z

Richard-0258 answered · Mar 04 at 10:00 AM

MotoX80,

The transcript doesn't really give any further information, only that the background tasks are being created. The tasks are running as they should and data is being copied to the file server from all the remote sites, but the RoboCopy logs are simply stopping when x% is complete with no error codes or messages logged anywhere. In other words, Powershell is doing what it should, it is creating multiple Robocopy jobs, the script has all the appropriate permissions as the copy operation is working for a period of time, no time outs or disconnect errors in the RoboCopy logs, it simply just stops copying even though the copy hasn't reached 100%.

Tried replacing the RoboCopy with Start-Bitstransfer only to find out you can't use Start-BitsTransfer with a scheduled task as it needs to be run in the foreground and only seems able to do 3 or 4 jobs concurrently.

The files I'm copying are around 700MB in size and some of the sites we are copying from have flaky connections, so Copy-Item and XCopy aren't options either.

Also tried changing the script so the RoboCopy operations are kicked off on each remote server, however, again this doesn't seem to be achievable, due to the double hop authentication required to make it work.

I'm now contemplating trying to do this with DFS Replication, but there are over 80 servers in 80 sites that I need to collect up the local backups from and this will be a major pain in the backside having to created so many DFS Replication groups.

Before I give up on this, any further ideas?

· 2 ·

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

MotoX80 · Mar 04 at 02:14 PM

If they work when run manually, then a timeout of some kind would be the logical suspect. But I can't find any default job timeout.

You're running the robocopy's on the destination file server, correct?

Are you launching 53 jobs all at one time? Have you tried limiting that where you only run say 10 at a time, and launching the "next one" when one finishes?

What return code ($lastexitcode) do you get when robocopy suddenly stops? If it's non-zero, you could test that and retry the robocopy operation. Limit it to 5 retries before giving up.

Or don't use jobs, just do a Start-Process. Maintain a collection of PSCustom objects that contain the PID and "job name". Again I would recommend some kind of throttle where you limit the number of concurrent copy operations so that you don't overload memory and network throughput. Given your flaky connection comment.

0 ·

MotoX80 MotoX80 · Mar 04 at 04:10 PM

Here's sample that I wrote for another user that you might be able to use.

https://docs.microsoft.com/en-us/answers/questions/233453/using-robocopy-multithreading-with-powershell-to-c.html

0 ·

question