Using Robocopy Multithreading with Powershell to copy files simultaneously

Question

I am using below command to copy files from locations same time. But this is copying one file at a time.
I am using Robocopy Multithread as well , but still not working, pls correct me.
Also , I am trying to track progress of Copying using write-progress , please help me on that as well.

 foreach($db in $dbs){

                $filename = gci -Path $BkpsPath\$db  |  Where-object {$_.Name -match $datefilter} | select -ExpandProperty name

                #starting job for every item in source list
                Write-Progress -Activity "Copying $db" -Status 'Copying $db file' -PercentComplete 0
                 start-job -ScriptBlock {
                 ROBOCOPY $($BkpsPath + "\" + $db) $($Destination + "\" + $db) $filename /MT:32 /NP /TS /FP | Out-File $outfile -Append
                 }}

Answer 1

I think that the main issue is the number of databases that you need to backup (both now and in the future), and the impact that they might have on system performance. If you have 50 DB's, you probably don't want to launch them all at the same time.

Try this example. Monitor system performance and vary $RunLimit to find what works best on your system.

Get-Job | Remove-Job  -ErrorAction Stop              # clear out old job information, stop if there's a problem  
  
$dbs = @('db1','db2','db3','db4','db5','db6','db7','db8','db9','db10','db11','db12','db13','db14','db15')      # some bogus names to play with   
  
$RunLimit = 10                       # Number of concurrent robocopy's to execute   
  
foreach ($db in $dbs) {  
    "{0} - Scheduling robocopy for {1}" -f (get-date -Format u), $db   
    $Atmax = $true                  #  look at how many we have running   
    while ($Atmax) {  
        $running = (Get-Job | Where-Object { $_.State -eq 'Running' })            # How many jobs are actively running?   
        if ($running.Count -le $RunLimit) {             # are we at max?   
            $Atmax = $false                             # no, it is ok to launch another job      
        } else {  
            start-sleep 3                               # yes, sleep a bit and look again    
        }  
    }    
      
    $job = Start-Job  -ScriptBlock {  
            # put real robocopy here  
            "********************************************************************************************************"    
            "****         {0}" -f $using:db   
            "********************************************************************************************************"  
            Robocopy.exe c:\temp\$($using:db)\ e:\temp\$($using:db)\  *.ini  /l /np   
        }   
}  
  
"{0} - All jobs have been launched. " -f (get-date -Format u)   
  
$Atmax = $true                  #   not really max, I just reused the flag to wait for all jobs to complete      
while ($Atmax) {  
    $running = (Get-Job | Where-Object { $_.State -eq 'Running' })            # How many jobs are actively running?   
    if ($running.Count -eq 0 ) {                    # are all jobs finished?    
        $Atmax = $false                             # yes, we are done      
    } else {  
        start-sleep 3                               # no, sleep a bit and look again    
    }  
}    
  
"{0} - All jobs have completed. " -f (get-date -Format u)   
  
Get-Job| Receive-Job                          # retrieve job output. Could be redirected to Out-File       
 

  
  

Answer 2

But this is copying one file at a time.

Well, yes, you are specifying $filename in the command, so for each instance of robocopy.exe you are only copying one file.

If you want to copy multiple files then you can't use $db in the source and destination folder names. The file name will need to have a wildcard (*.db) and you need to add the /E switch to search subdirectories.

ROBOCOPY $($BkpsPath + "\") $($Destination + "\" ) *.db /MT:32 /NP /TS /E

Use the /L switch to see what files would get copied. It all depends on what files/folders you have. You may need to use the /XF and/or /XD switches to exclude unwanted DB's.

I am using Robocopy Multithread as well , but still not working, pls correct me.

What exactly isn't working?

I am trying to track progress of Copying using write-progress

What criteria do you intend to query to know if you are 10, 50, 75 percent completed? I would suggest that you focus on getting robocopy working the way that you need/want it to and not bother with a progress display right now. ]

One obvious problem that you have is that each instance of robocopy that you launch is writing it's output to the same $outfile. If you just called robocopy, that would be fine, because each instance would run one after the other. By using Start-Job you are running them in parallel, essentially at the same time. You should at least add the $db name to the log file name.

If you have 30 databases, then you could have 30 robocopies running and your overall throughput might suffer because you're just trying to run too many jobs.