- To get robocopy fast you have to disable all console output, something like five extra flags.
by chihuahua
1 subcomments
- To properly appreciate a post like this one, it should ideally be paired with a Raymond Chen post that argues in Hercule Poirot style irrefutable logic how a combination of backwards-compatibility with CP/M and some 1990s programming done by raccoons means that this messed up state of affairs is logically the only way it could possibly be.
- OP mentions using "Cat 7" cables - please don't buy these. Cat 7 isn't something that exists in TIA/EIA standards, only in ISO/IEC and it requires GG45 or TERA connectors. Cat 7 with RJ45 connectors isn't standardized, so you have no idea what you're actually getting. Stick with pure copper Cat 6A.
- Invoke-WebRequest is also very slow if you forget to disable the progress bar with $ProgressPreference = 'SilentlyContinue'
PowerShell has some "interesting" design choices...
- Be nice if you listed any particular settings on the commands....
For robocopy, for example, if you're copying small files/bunch of directories use the /MT:$number flag. It's so much massively faster it's not like the same application.
Also is this a newer version of Windows that supports smb3, Explorer is likely using that to copy more in parallel.
by charcircuit
2 subcomments
- >no scripting options at all on Windows that come close to the fearsome power of File Explorer’s copy and paste out of the box
You can use Powershell.
$shell = New-Object -ComObject Shell.Application
$shell.Namespace("C:\Source").ParseName("myfile").InvokeVerb("copy")
$shell.Namespace("C:\Destination").Self.InvokeVerb("paste")
- I am not surprised. My Windows 11 systems with modern and beefy hardware frequently runs very slow for reasons unknown. I did use https://github.com/Raphire/Win11Debloat recently and that seemed to have helped. Windows by default comes with a lot of crap that most of us do not use but it consumes resources anyway.
I have been considering a move back to Linux. It is only Microsoft Teams on Windows that I have to use daily that is holding me back.
- @op did you try via command prompt (cmd.exe) eg copy or xcopy ?
by DustinEchoes
0 subcomment
- Never assume anything done in Powershell is fast.
- Robocopy has options for unbuffered IO (/J) and parallel operations (/MT:N) which could make it go much faster.
Performing parallel copies is probably the big win with less than 10 Gb/s of network bandwidth. This will allow SMB multichannel to use multiple connections, hiding some of the slowness you can get with a single TCP connection.
When doing more than 1-2 GB/s of IO the page cache can start to slow IO down. That’s when unbuffered (direct) IO starts to show a lot of benefit.
- It's fortunately been years since I have used Windows, but it looks like the old staples are still ahead of the curve:
https://fastcopy.jp/
https://www.codesector.com/teracopy
(I have certainly forgotten at least one...)
- Just tried copying a 20GB file to my Windows desktop from a mounted Samba share through gigabit ethernet (nvme on both sides). Explorer, Copy-Item and robocopy all saturated the connection with no issues.
There's definitely something off about OP's setup, though I have no idea what it could be. I'd start by checking the latency between the machines. Might also be the network adapter or its drivers.
- The page is 404 now. It looks like something went wrong when the author was trying to push a small edit to the page. The content is viewable at https://github.com/hiAndrewQuinn/til/blob/main/copy-item-is-...
- One thing I don't understand with Windows Server is that it seems no matter how fast the nvme drives I use, or I pair/pool, I can't get a normal file copy to go faster than around 1.5GB/s (that's local, no network). The underlying disks show multi GB/s performance under crystal disk mark. But I suspect something in the OS must get in the way.
by zaptheimpaler
3 subcomments
- ugh, I don't know why copying files and basic I/O is so fucked on Windows. Recently I was trying to copy some large movie files between 2 folders on an NVME SSD formatted to ExFAT in a good USB-C enclosure connected over 20Gbps USB-C port and explorer would literally just freeze & crash doing that. I had to copy one file at a time to make it not crash, and then it would have this weird I/O pattern where the transfer would do almost nothing for 1-2 minutes, then the speed eventually picked up.
This isn't even going into WSL. I specifically stopped using WSL and moved to a separate linux devbox because of all the weirdness and slowness with filesystem access across the WSL boundary. Something like listing a lot of files would be very slow IIRC. Slightly tangentially, the whole situation around sharing files across OSes is pretty frustrating. The only one that works without 3rd party paid drivers on all 3 major OSes is ExFAT and that is limited in many other ways compared to ext4 or NTFS.
- Looking at the source code or copy-item (assuming the author is using a recent version of PowerShell) at https://github.com/PowerShell/PowerShell/blob/master/src/Mic... which calls https://github.com/PowerShell/PowerShell/blob/master/src/Sys..., there seems to be quite a bit of (non-OS) logic that takes place before copying across the network. Copying many small files probably triggers some overhead there.
Then, when the copying happens, this seems to be the code that actually copies the file, at least when copying from remote to local, using the default file system provider: https://github.com/PowerShell/PowerShell/blob/master/src/Sys...
Unless I've taken a wrong turn following the many abstraction layers, this file copy seems to involve connecting to a remote server and exchanging the file contents over a base64 stream (?) using nothing but a standard OutputStream to write the contents.
This means that whatever performance improvements Microsoft may have stuffed into their native network filesystem copy operations doesn't seem to get triggered. The impact will probably differ depending on if you're copying Windows-to-Windows or SAMBA-to-Windows or Windows-to-SAMBA.
I'm no PowerShell guru, but if you can write a (C#?) cmdlet to invoke https://learn.microsoft.com/en-us/windows/win32/api/shellapi... with a properly prepared https://learn.microsoft.com/en-us/windows/win32/api/shellapi... rather than use the native Copy-Item, I expect you'd get the exact same performance you'd get on Windows Explorer.
However, the other measurements do show some rather weird slowdowns for basic filesystem operations over SFTP or WSL2. I think there's more at play there, as I've never seen sftp not reach at least a gigabit given enough time for the window sizes to grow. I think the NAS itself may not be very powerful/powerful enough to support many operations per second, limiting the output for other copy tools.
As an alternative, Windows contains an NFS client that can be tuned to be quite fast, which should have minimal performance overhead on Linux if kernel-NFS is available.
by perryizgr8
0 subcomment
- At this point Microsoft needs to start over with Windows, preferably build on Linux. Valve did it and built a better gaming OS. Microsoft can do it for the rest of us.
- > SFTP is an encrypted protocol, so maybe those CPU cycles add up to a lot of extra work over time or slowdown. That… shouldn’t feel convincing to anyone who gives it more than 15 seconds of thought, but we all live with our eyes wide shut at times.
FWIW, I previously spent some time trying to get the maximum possible throughput when copying files between a Windows host and a Linux VM, and the encryption used by most protocols did actually become a bottleneck eventually. I expect this isn't a big factor on 1gbps ethernet, but I've never measured it.
by itsthecourier
0 subcomment
- want to see rsync WSL 1 in that comparison
filesystem should be faster in WSL2 but not if the file resides in the windows path I think