New Tool - MST

Yet another command line application to help me run backup scripts.

It occurred to me that 90% of my backup scripts just run robocopy commands, many of which do not rely on a previous command - so could I launch a batch file and have all the commands in it run simultaneously in multiple threads rather than each command having to complete before the next one can run? Seems logical and realistic based on today's multi-core processors and gigabytes of RAM.

Then I thought that actually some of my commands do rely on a predecessor completing first, so what I want is a multi-stage solution where one stage cannot begin until the last stage has completed but within each stage all the commands should be executed simultaneously.


I could find no easy way to do this with traditional batch files and so began Mutli Stage Threading (or MST) for short.


It's a fairly raw little application with no command line help what-so-ever (at least not at the time of writing) and no command line parameters. To use it is pretty simple; put the executable in a folder then in the same folder create a file named Stage1.cmd in to which you add your batch commands. Create a Stage2.cmd file and pop in some more commands. Create Stage3.cmd and so on. Run MST and all the commands in Stage1.cmd will all be created in separate threads. Stage2.cmd will not begin until every command in Stage1.cmd has completed.


I chose to use the .cmd extension so that the files remained proper batch files that could also be run outside of MST.


It does have some limitations with scope for improvement:

• Stage files must be in the same folder as MST.exe, so if you want to run multiple jobs you need to keep multiple copies of MST.exe in a number of folders. Solution: add a command line parameter that specifies the folder that contains the stage files you want to launch. DONE
• Stage files must have single, standalone commands on each line. MST does not support proper batch files that have labels, error levels and forking. Solution: have MST examine the first line of each stage file. If it is "REM NOTHREADING" then execute the whole file in a single thread and wait for it to complete. DONE
• The first word on each line of the stage files must be a command application i.e. robocopy. It can not be "call" or "start". Solution: probably same as previous solution - execute whole file.
• Stage files must be named Stage1.cmd, Stage2.cmd etc.. Solution: have MST look for a file named stages.mst. If this file exists then assume that it contains a list of scripts to execute so start at the top and work down. DONE
• If you have Stage1.cmd to StageN.cmd and you delete StageN-6.cmd then the loop is broken because MST stops as soon as the next stage file does not exist. Solution: have a command line parameter in which you can specify the minimum number of files to check for so that if Stage48.cmd does not exist MST will still check for the existence of Stage49.cmd so long as you specify on the command line to look for 49 or more stages. DONE
• There is currently no way to specify the maximum amount of threads to use. If a stage file contains 30 commands then MST will launch 30 threads. If this is too many then you need to split the commands over multiple stage files. Solution: have a command line parameter to specify the maximum number of threads to use per stage.

Slow file copy over LAN with Windows 7

If you search Google regarding this then you find that lots of people have this problem. I need to copy a 20GB Microsoft SQL Server backup file across the network each night from Server 2008 to a new Windows 7 Professional x64 box with 4GB RAM - this was taking in the region of 7 hours!

If I copy the same file to my Vista Ultimate x86 laptop it takes 40 minutes, that's more like it, after all we do have a gigabit LAN.

After reading several forums I found 5 main (possible) resolutions to this:

1. Disable TCP auto tuning by running this command with administrator privileges: netsh int tcp set global autotuninglevel=disabled (you can set is back again with =normal).
2. Turn off the Remote Differential Compression feature. Go to Control Panel, Programs and Features, Turn Windows features on or off, switch off RDC.
3. In the same place, also switch off Windows Meeting Space.
4. In networking, switch off IPv6 support.
5. Stop and disable the Superfetch service and reboot.

I took steps 1, 2 and 5 but I think that 5 made the most impact. Now the same 20GB file copies over the LAN to the Windows 7 machine in 18 minutes! Looks like I may need to apply these settings to Vista.

Network Connections

I am forever disabling and enabling my Wireless connection depending on whether wireless is available and if I have a cable plugged in. If both are available I will always plug in the cable but to ensure that I use the cable I disable my Wireless connection.

I wondered therefore if I could script this (yes, I could just disconnect or even use the switch on the side of my laptop but I like to script stuff!) - turns out you can:

netsh interface set interface "Wireless Network Connection" disable

and as you might expect:

netsh interface set interface "Wireless Network Connection" enable.

Just use the correct name for the network connection you want to enable/disable.

New Tool - HashFile

I am working on implementing a remote backup solution - basically something I can script that will securely transfer some files over the Internet to a remote server. As yet I have not made my mind up on the transfer method but it will most likely be either FTPS, FTP (but sending encrypted data) or using a VPN to secure the transmission.

Either way I have some large files to backup - 20GB database backup files, that sort of thing and so I am going to use Delta Compression (of which I am a fan and will no doubt blog about some other time). I will use my favourite DC tool, xdelta.

As I said, I want to script the backup - I like DOS style batch files (I have yet to get in to Power Shell, but it's on my list of things to do - it's a long list). I am primarily a software developer and so creating new command line applications to help me create a scriptable solution is not a problem, but I like to create lots of small tools that I can reuse again and again for different tasks rather than code the whole job in a single executable that will have no other use.

As well as transmitting a "binary difference" or "patch" file, I also want to send over the signature of the most up-to-date copy of the file I am backing up, so that when the patch is applied at the other end to the older file I can compare signatures to make sure my file is intact and matches. The easiest way to do this is to use a Hash algorithm (also known as a Message Digest).

So I have built myself a very small, very simple command line application that will calculate the hash of any file and write it to an INI file (you might consider INI files a little old fashioned, a little Windows 3.1 - but they remain an excellent way to store application settings and I would always choose an INI file over the registry, and Delphi has excellent classes for working with INI files). That way I can also transmit the INI file (which due to it's nature could be reused and so contain many hashes for many files).

Today then, HashFile was conceived, incubated and born.

Usage:

HashFile <switches> [ini_file] [file_to_hash]

<switches>

-h{algorithm}: can be one of the following: MD2, MD4, MD5, SHA1, RIPEMD160, default is MD5.

-s{ini section}: set the name of the ini section header, default is HASH_FILE.

-k{ini key}: set the name of the key in the ini file, default is the file name of the file being hashed.

Assuming I am happy with the default values, I could just run:

HashFile D:\Temp\Test.ini D:\Temp\Untitled.jpg

The contents of Test.ini look like this:

[HASH_FILE]

Untitled.jpg=3b7101d7d93f80dc7ac0135830d5ccd3

You can use 0, 1, 2 or all 3 switches. Here is an example of using them all:

HashFile -hSHA1 -sMY_SECTION -kMY_VALUE D:\Temp\Test.ini D:\Temp\Untitled.jpg

Because Test.ini already exists from the last example the new content is appended (nothing is overwritten because the values are different) and it now looks like this:

[HASH_FILE]

Untitled.jpg=3b7101d7d93f80dc7ac0135830d5ccd3

[MY_SECTION]

MY_VALUE=64f280a904f7dc729ee8ea9ba932dbfe136d2c10

That's it. It will do for now, but I am already planning on expanding it:

• Allowing the user to pass multiple files to be hashed instead of just one at a time (which is why I made sure that the file to be hashed was the very last parameter - you will just add more).
• Adding another switch that will direct the output to be a XML file rather than an INI file.

Virtualisation

(or virtualization if you are from across the pond)

I love virtualisation! I have been using it in one form or another since circa 2005 and it just has so many uses:

• Evaluating software that you don't want to install on your main PC.
• Testing your own installs on a virgin OS.
• Testing against multiple operating systems.
• Running alternative operating systems i.e. run Linux on your Windows desktop and vice versa.
• For remote support so you don't have to install all manner of weird and wonderful VPN Clients on your main PC.
• Being able to hack together a temporary server without needing a physical box.
• Converting your main PC from physical-to-virtual (known as P2V) so that you can rebuild your PC but still be able to use a virtual copy of your old PC until the new build is fully up, configured and running.

The list just goes on... The best bit? Because competition is so fierce in this arena, most of the decent hyper-visors (the software that lets you run virtual machines) are free!

I hope to blog more on virtualisation (especially the new "Boot from VHD" option in Windows 7 - which also works on Vista with a little hacking about) because it is so cool and I have plenty of tips and tools to share. In the meantime, check out some of these:

VirtualBox - my favourite desktop hyper-visor. It is open source and ahead of the game. One of the coolest features is that it will not only boot images from it's own native virtual hard disk format (vdi) but it will also boot from VMware images (vmdk) and Microsoft's format (vhd).

Microsoft Virtual PC and Microsoft Virtual Server - both good but the latest version of VPC which only runs on Windows 7 has only just caught up with some of the features offered by VirtualBox for ages such as a seamless mode (XP Mode) and USB support.

Microsoft Hyper-V - the successor to Virtual Server and should be great, but not yet tried it as I don't have a server with a compatible CPU. Hyper-V can sit on top of Server 2008 (a type 2 hyper-visor) or you can install Hyper-V Server as a standalone bare metal install (a type 1 hyper-visor) i.e. it is not running on top of Windows.

VMware - great hyper-visors backed with years of experience, but not my cup of tea and I don't know why.

Citrix XenServer - looks very good, but don't have a couple of spare servers with compatible CPUs to play with.

There are other hyper-visors out there, go and find the right one for you.

Date and Time Injection for Command Line Parameters

I like to backup files using tools such as robocopy, richcopy and 7-Zip, and sometimes I want to include the day or date and time in the backup path or file name. For example, I might want to copy some files to an external hard disk and have them go in to a Monday folder on a Monday, a Tuesday folder on a Tuesday etc.

After looking around I failed to find a satisfactory solution so being a software developer I decided to create my own solution - the result was DateParseCmd.exe.

This is a very simple command line tool written in Delphi 7. The parameters for this command are actually just another command with it's parameters. However, in the child command's parameters, if you want to specify the current date and/or time you can do so by enclosing a Delphi style DateTime Format String in {[ and ]}.

Suppose I want to use a single robocopy script to backup to a DayOfTheWeek folder, I can now do something like this:

DateParseCmd robocopy "C:\My Projects" "F:\Backups\{[dddd]}\My Projects" /MIR /ZB /R:3 /W:10

DateParseCmd first parses it's command line parameters and {[dddd]} will resolve to "Monday" on a Monday. It will then launch the robocopy command with it's newly evaluated parameters i.e.

robocopy "C:\My Projects" "F:\Backups\Monday\My Projects" /MIR /ZB /R:3 /W:10

If I want to archive a folder using the standalone 7-Zip command line tool and have the current date in the archive name so that I never overwrite an archive:

DateParseCmd 7za.exe a -r "D:\My Backups\Source Code {[yyyy-mm-dd]}.7z" "C:\Source Code\*.*"

This will create an archive named "Source Code 2010-04-14.7z". Simple stuff, but very useful.

Shadows in the dark

(or the elusive vshadow.exe)

vshadow is a very cool command line tool from Microsoft for working with the Volume ShadowCopy Service (VSS). If you don't know what VSS is then I'm not about to go in to the details now but it's what Windows uses for System Restore Points, retrieving Previous Versions of documents and allows you to make copies of files that are in use.

vshadow is much better and easier to use than vssadmin, especially on non-server operating systems on which vssadmin won't let you create a snapshot (or shadow volume). The trouble is that it's not the easiest tool to get hold of - it is actually a demo tool that is supplied with various Microsoft SDKs as an example of how to use the VSS API. As such, the tool is limited and there is plenty that it does not do, however if you are looking for a solution to backing up files that are in use then combined with dosdev.exe (another elusive Microsoft tool found in some SDKs) and robocopy it provides a very robust and scriptable solution.

The annoying part is that the VSS API appears to be different on all versions of Windows and so you need a copy of vshadow that was compiled for the version of Windows you want to use it on (i.e. XP, Vista, Win7, Server 2003 or Server 2008). No one seems willing to host the compiled copies of vshadow as there is some debate as to whether this tool is redistributable under the terms of the SDK - personally I think this tool deserves a place on sysinternals.

Ok, so how do you get hold of vshadow.exe?

XP and Server 2003

Download and install the Volume Shadow Copy Service SDK 7.2. Once installed, browse here: C:\Program Files\Microsoft\VSSSDK72\TestApps\vshadow\bin and you will find the release folders for both XP and Server (2003).

Vista, Server 2008 and Win7

Download and install the relevant Microsoft Windows Software Development Kit:

Vista: Windows SDK v6.0

Server 2008: Windows SDK v6.1

Windows 7: Windows SDK v7.0

You can either install these by using the Web Installer or you can download the ISOs. If you just want the compiled versions of vshadow and don't care about the source code or anything else in the SDK then when installing just choose to only install the Win32 Development Tools.

All 3 SDKs install to C:\Program Files\Microsoft SDKs\Windows plus a folder for each SDK (v6.0, v6.1 and v7.0 respectively). Within each version folder head in to Bin\vsstools and you will find the prize.

In order to script a backup of in use files using vshadow, dosdev and robocopy, take a look at this blog entry which contains a section with a script example.

If this all seems a little too complicated and you just want to backup some in use files and you don't need all the cool options that robocopy has then take a look at Hobocopy. This is a tool inspired by robocopy but uses the VSS API to create a shadow volume from which the files are actually copied. Because it uses the VSS API you must make sure you use the correct build of Hobocopy for the OS you are running it on.