Slow file copy over LAN with Windows 7 cont.

Another possible solution...

Windows Vista/7 has a network setting named "task offload" which is on by default. What this does is take same some processing away from your CPU and hand it off to the processor on your network card i.e. computing checksums and such like. This should give you performance benefits.

Turning task offload off may speed up the copying of large files over your LAN with Windows 7. You need to open a command shell with admin rights. To list which interfaces have task offload switched on:

C:> netsh int ip show offload

To switch task offloading off (globally):

C:> netsh int ip set global taskoffload=disabled

This should return "Ok". Now disable and then enable your network adapter and it should be done. Try a file copy.

Old Skool

Check out retrospec where a group of talented developers have (for some years) been recreating old school classic video games from the era that gave us the Spectrum 48k and the Commodore 64.

These are not ports as such, but have been built from the ground up and updated for newer hardware (better gfx, better sound).

Their version of Head Over Heels is amazing and faithful to the original game (which I played on the Speccy and owned on both C64 and Amiga 500). It's even better than playing the Amiga version in WinUAE!

New Tool - MST

Yet another command line application to help me run backup scripts.

It occurred to me that 90% of my backup scripts just run robocopy commands, many of which do not rely on a previous command - so could I launch a batch file and have all the commands in it run simultaneously in multiple threads rather than each command having to complete before the next one can run? Seems logical and realistic based on today's multi-core processors and gigabytes of RAM.

Then I thought that actually some of my commands do rely on a predecessor completing first, so what I want is a multi-stage solution where one stage cannot begin until the last stage has completed but within each stage all the commands should be executed simultaneously.


I could find no easy way to do this with traditional batch files and so began Mutli Stage Threading (or MST) for short.


It's a fairly raw little application with no command line help what-so-ever (at least not at the time of writing) and no command line parameters. To use it is pretty simple; put the executable in a folder then in the same folder create a file named Stage1.cmd in to which you add your batch commands. Create a Stage2.cmd file and pop in some more commands. Create Stage3.cmd and so on. Run MST and all the commands in Stage1.cmd will all be created in separate threads. Stage2.cmd will not begin until every command in Stage1.cmd has completed.


I chose to use the .cmd extension so that the files remained proper batch files that could also be run outside of MST.


It does have some limitations with scope for improvement:

• Stage files must be in the same folder as MST.exe, so if you want to run multiple jobs you need to keep multiple copies of MST.exe in a number of folders. Solution: add a command line parameter that specifies the folder that contains the stage files you want to launch. DONE
• Stage files must have single, standalone commands on each line. MST does not support proper batch files that have labels, error levels and forking. Solution: have MST examine the first line of each stage file. If it is "REM NOTHREADING" then execute the whole file in a single thread and wait for it to complete. DONE
• The first word on each line of the stage files must be a command application i.e. robocopy. It can not be "call" or "start". Solution: probably same as previous solution - execute whole file.
• Stage files must be named Stage1.cmd, Stage2.cmd etc.. Solution: have MST look for a file named stages.mst. If this file exists then assume that it contains a list of scripts to execute so start at the top and work down. DONE
• If you have Stage1.cmd to StageN.cmd and you delete StageN-6.cmd then the loop is broken because MST stops as soon as the next stage file does not exist. Solution: have a command line parameter in which you can specify the minimum number of files to check for so that if Stage48.cmd does not exist MST will still check for the existence of Stage49.cmd so long as you specify on the command line to look for 49 or more stages. DONE
• There is currently no way to specify the maximum amount of threads to use. If a stage file contains 30 commands then MST will launch 30 threads. If this is too many then you need to split the commands over multiple stage files. Solution: have a command line parameter to specify the maximum number of threads to use per stage.

Slow file copy over LAN with Windows 7

If you search Google regarding this then you find that lots of people have this problem. I need to copy a 20GB Microsoft SQL Server backup file across the network each night from Server 2008 to a new Windows 7 Professional x64 box with 4GB RAM - this was taking in the region of 7 hours!

If I copy the same file to my Vista Ultimate x86 laptop it takes 40 minutes, that's more like it, after all we do have a gigabit LAN.

After reading several forums I found 5 main (possible) resolutions to this:

1. Disable TCP auto tuning by running this command with administrator privileges: netsh int tcp set global autotuninglevel=disabled (you can set is back again with =normal).
2. Turn off the Remote Differential Compression feature. Go to Control Panel, Programs and Features, Turn Windows features on or off, switch off RDC.
3. In the same place, also switch off Windows Meeting Space.
4. In networking, switch off IPv6 support.
5. Stop and disable the Superfetch service and reboot.

I took steps 1, 2 and 5 but I think that 5 made the most impact. Now the same 20GB file copies over the LAN to the Windows 7 machine in 18 minutes! Looks like I may need to apply these settings to Vista.

Network Connections

I am forever disabling and enabling my Wireless connection depending on whether wireless is available and if I have a cable plugged in. If both are available I will always plug in the cable but to ensure that I use the cable I disable my Wireless connection.

I wondered therefore if I could script this (yes, I could just disconnect or even use the switch on the side of my laptop but I like to script stuff!) - turns out you can:

netsh interface set interface "Wireless Network Connection" disable

and as you might expect:

netsh interface set interface "Wireless Network Connection" enable.

Just use the correct name for the network connection you want to enable/disable.

New Tool - HashFile

I am working on implementing a remote backup solution - basically something I can script that will securely transfer some files over the Internet to a remote server. As yet I have not made my mind up on the transfer method but it will most likely be either FTPS, FTP (but sending encrypted data) or using a VPN to secure the transmission.

Either way I have some large files to backup - 20GB database backup files, that sort of thing and so I am going to use Delta Compression (of which I am a fan and will no doubt blog about some other time). I will use my favourite DC tool, xdelta.

As I said, I want to script the backup - I like DOS style batch files (I have yet to get in to Power Shell, but it's on my list of things to do - it's a long list). I am primarily a software developer and so creating new command line applications to help me create a scriptable solution is not a problem, but I like to create lots of small tools that I can reuse again and again for different tasks rather than code the whole job in a single executable that will have no other use.

As well as transmitting a "binary difference" or "patch" file, I also want to send over the signature of the most up-to-date copy of the file I am backing up, so that when the patch is applied at the other end to the older file I can compare signatures to make sure my file is intact and matches. The easiest way to do this is to use a Hash algorithm (also known as a Message Digest).

So I have built myself a very small, very simple command line application that will calculate the hash of any file and write it to an INI file (you might consider INI files a little old fashioned, a little Windows 3.1 - but they remain an excellent way to store application settings and I would always choose an INI file over the registry, and Delphi has excellent classes for working with INI files). That way I can also transmit the INI file (which due to it's nature could be reused and so contain many hashes for many files).

Today then, HashFile was conceived, incubated and born.

Usage:

HashFile <switches> [ini_file] [file_to_hash]

<switches>

-h{algorithm}: can be one of the following: MD2, MD4, MD5, SHA1, RIPEMD160, default is MD5.

-s{ini section}: set the name of the ini section header, default is HASH_FILE.

-k{ini key}: set the name of the key in the ini file, default is the file name of the file being hashed.

Assuming I am happy with the default values, I could just run:

HashFile D:\Temp\Test.ini D:\Temp\Untitled.jpg

The contents of Test.ini look like this:

[HASH_FILE]

Untitled.jpg=3b7101d7d93f80dc7ac0135830d5ccd3

You can use 0, 1, 2 or all 3 switches. Here is an example of using them all:

HashFile -hSHA1 -sMY_SECTION -kMY_VALUE D:\Temp\Test.ini D:\Temp\Untitled.jpg

Because Test.ini already exists from the last example the new content is appended (nothing is overwritten because the values are different) and it now looks like this:

[HASH_FILE]

Untitled.jpg=3b7101d7d93f80dc7ac0135830d5ccd3

[MY_SECTION]

MY_VALUE=64f280a904f7dc729ee8ea9ba932dbfe136d2c10

That's it. It will do for now, but I am already planning on expanding it:

• Allowing the user to pass multiple files to be hashed instead of just one at a time (which is why I made sure that the file to be hashed was the very last parameter - you will just add more).
• Adding another switch that will direct the output to be a XML file rather than an INI file.

Virtualisation

(or virtualization if you are from across the pond)

I love virtualisation! I have been using it in one form or another since circa 2005 and it just has so many uses:

• Evaluating software that you don't want to install on your main PC.
• Testing your own installs on a virgin OS.
• Testing against multiple operating systems.
• Running alternative operating systems i.e. run Linux on your Windows desktop and vice versa.
• For remote support so you don't have to install all manner of weird and wonderful VPN Clients on your main PC.
• Being able to hack together a temporary server without needing a physical box.
• Converting your main PC from physical-to-virtual (known as P2V) so that you can rebuild your PC but still be able to use a virtual copy of your old PC until the new build is fully up, configured and running.

The list just goes on... The best bit? Because competition is so fierce in this arena, most of the decent hyper-visors (the software that lets you run virtual machines) are free!

I hope to blog more on virtualisation (especially the new "Boot from VHD" option in Windows 7 - which also works on Vista with a little hacking about) because it is so cool and I have plenty of tips and tools to share. In the meantime, check out some of these:

VirtualBox - my favourite desktop hyper-visor. It is open source and ahead of the game. One of the coolest features is that it will not only boot images from it's own native virtual hard disk format (vdi) but it will also boot from VMware images (vmdk) and Microsoft's format (vhd).

Microsoft Virtual PC and Microsoft Virtual Server - both good but the latest version of VPC which only runs on Windows 7 has only just caught up with some of the features offered by VirtualBox for ages such as a seamless mode (XP Mode) and USB support.

Microsoft Hyper-V - the successor to Virtual Server and should be great, but not yet tried it as I don't have a server with a compatible CPU. Hyper-V can sit on top of Server 2008 (a type 2 hyper-visor) or you can install Hyper-V Server as a standalone bare metal install (a type 1 hyper-visor) i.e. it is not running on top of Windows.

VMware - great hyper-visors backed with years of experience, but not my cup of tea and I don't know why.

Citrix XenServer - looks very good, but don't have a couple of spare servers with compatible CPUs to play with.

There are other hyper-visors out there, go and find the right one for you.