Edit [2021-02-17]: we have reversed the changes back to the original, as it seems to work better as it was. The suggested change was from using -auvhPW instead of -rutvahz (see below shell script commands). Also, we highly recommend the AOMEI for full system backups and hard drive cloning. We now use a combination of rsync, Windows File History and AOMEI.
Edit [2020-08-14]: we have made some changes to this guide based on user feedback.
Over our years providing software development services, we have used many different file backup techniques, including: manually copying files, running Windows backups, 3rd-party software solutions, Rich/Robo Copy, shell scripts (which we’ll talk about), and we even tried one of those external hard-drive software bundles, like WD Passport.
We reluctantly installed Windows 10 on our main computers, figuring that Windows 10 has now had ample time to get most of the kinks worked out, and to our surprise “Backup and Restore” from Windows 7 is being deprecated, and the “Backup and Restore (Windows 7)” in Windows 10 seems to fail no matter what was tried. The only option is to use their new backup option, File History, which we use to an extent.
However, we ran into issues with File History when trying to select specific folders under the AppData directory. It didn’t seem to target them properly and the only way to add them was to add the entire AppData directory, instead of specific directories from within.
So now we create our own backup scripts using a file-transfer command-line tool called rsync.
rsync is an open source utility that provides fast incremental file transfer. rsync is freely available under the GNU General Public License and is currently being maintained by Wayne Davison.
It’s a bit of a process to get this setup on Windows, but it’s worth it. If you have a linux system, then you already have rsync installed by default.
- Download and install Notepad++; used to instill proper line-endings for the shell script we will create.
- Download, install, and configure Cygwin; this program simulates a shell in Windows, and also provides rsync as an option during installation.
- Go through some rsync command-line examples.
1. Notepad++
Notepad++ is a great tool for quick notes and file editing/previewing. It allows for multiple documents to be open at once, and it remembers unsaved documents for you. You can download Notepad++ from here: https://notepad-plus-plus.org/
We will use this program to create a shell file (.sh extension) that we can run from Cygwin. Notepad++ helps us solve an issue where regular text files created in Windows have different line ending characters (\r\n) than those that are used in a Unix/Linux environment (\n).
We’ll use Notepad++ to set the line endings of our shell script to be that of Unix decent. Open Notepad++ and create a new file, then click on Edit > EOL Conversion > Unix (LF), then save your file with a .sh extension (eg. mybackup.sh).
In the very first line of that file, write the following:
#!/bin/bash
All remaining commands will go below this line.
2. Download, install and configure Cygwin
Cygwin provides a local shell (command-line) environment with configuration for many of the same capabilities as you’d have on a Linux OS or Mac machine, and this program will allow us to run the rsync command.
- Go to https://www.cygwin.com/
- Download your supported executable (.exe), 64 or 32-bit, file from the links found under the section Installing Cygwin.
- Open the file to begin the installation process…
- During installation, you will be asked which modules to install; you can use the search box at the top to help locate the modules we need. Find and select both ssh and rsync.
- Once installed, open the Cygwin terminal.
- If you make a mistake or need to install additional modules, you may run the Cygwin installer again to select additional modules.
3. Rsync commands
To view all accessible drives from Cygwin, type the following in your Cygwin terminal:
cd /cygwin/
From here, we can access our files on our main drives, and plan our backup strategy. Let’s take a look at a rsync command that copies everything from C-drive to D-drive:
rsync -rutvahz --stats /cygdrive/c/ /cygdrive/d/
The above command uses some very specific options/flags, which are explained below, starting with -auvhPW:
- a = archive, includes -r (recursive, into directories) and -t (times, preserve)
- u = update, skips files if they are newer
- v = verbose, to improve quality of transfer
- h = human-readable output
- z = compress during transfer (removed)
- P = equivalent to –partial –progress (providing file transfer progress %)
- W = copy files whole (equivalent to –whole-file)
- stats = provides statistics about the transfer upon completion
- Other flags can be found, and are described, on the rsync website.
This first example will copy all files from your user (MyUserName) /Documents/ folder on your C-drive, to a folder called /Documents/ on your D-drive, which can be an external hard drive, USB stick, etc.:
rsync -rutvahz --stats /cygdrive/c/Users/MyUserName/Documents/ /cygdrive/d/Documents/
- exclude ‘folder/file’ = allows you to dictate which files or folders to skip during the sync. You may chain –exclude flags to ignore multiple directories:
rsync -rutvahz --stats --exclude 'Private' --exclude 'SemiPrivate' /cygdrive/c/Users/MyUserName/Documents/ /cygdrive/d/Documents/
You may run the above commands right from the Cygwin terminal, or – especially if you have multiple rsync commands – you can save those commands to mybackup.sh and then run the following command:
./cygwin/c/path/to/file/mybackup.sh
This has been a really very useful guide to getting this up and running on some old Win7 machines, just backing up to ext drives. Im still trying to figure out how to make sure I dont keep creating the folder inside the folder by leaving off trailing slashes, once yuove made the initial data backup, so hope my next tests will work out.
Other things to note for other total noobs like me:
– To install only the packages you want from Cygwin, do not select the top’All’/install. It WILL install everything. Cygwin is massive, around 18gb I think, in total. Go to the specific packages and select install from those.
– Select a mirror that is close to your location, and preferably use ftp for speed.
– To access the C:/cygwin folder from your terminal, go up a level to C, as your initial terminal window is in your local user level.
– Any spaces in names of folders or user names can be surrounded by apostrophes to escape the spaces.
– Note that names of folders for ‘Documents’, ‘Pictures’ etc do not contain ‘My Pictures’, for example, they are just called ‘Documents’, Pictures’ when accessed from C.
Hope this helps someone else getting started with this.
rsync creates the end folder name in the target folder *unless* you include the / at the end of both paths.
IMPORTANT WARNING: please use “–whole-file” instead of allowing the partial file to remain if the command is stopped or fails for some reason. I have found that binary partial copies don’t always copy properly on restart. I don’t know why and partial updates have worked in the past, but if this is a backup, don’t take chances. I move 100s of TB on large Linux databases quite often with parallel runs of rsync. It is a great tool, but the possible corruption issue has bit me in the tail more than once.
The following creates the end paths which you don’t want. (note -a includes -r and -t)
rsync -auvh –stats –progress –whole-file /src/path/folder1 /target/path/folder1
results /target/path/folder1/folder1
Adding the forward slashes, I know it is not well documented, copies the contents of /src/path/folder1 into /target/path/folder1
So the following does what you want.
rsync -auvh –stats –progress –whole-file /src/path/folder1/ /target/path/folder1/
results /target/path/folder1
“Im still trying to figure out how to make sure I dont keep creating the folder inside the folder by leaving off trailing slashe”
I had this problem too! You can use dot syntax (something like ‘path/to/./folder’ – some experimentation probably required to work out exactly what you want) to avoid creating unnecessary folders
Instead of Notepad** you can add dos2unix to cygwin if the original install did not get it. ‘which dos2unix =’ will tell you).
Now do: cd ~/Downloads; ./setup-…..exe [~ is shorthand for the home directory in Unix/Linux.]
In the ‘Select Packages’ window pick ‘Not Installed’, in the empty box named View type dos2unix.
Now do a direct install or do in two steps; first download (default location: ~/Downloads), then install.
dos2unix does an in-place conversion. For more read the (automatically installed) manual
man dos2unix.