It’s been an interesting week, with some great discussions and research on how to transfer larges files to a lot of computers as efficient as possible. Time to write down some of that research. The scenario is: You have a bunch of machine, say student computers, you have limited infrastructure, and you want to deploy large files (apps, virtual machines, lab files etc.). What are your options for doing that? Well so far, and here is were I would love to get your feedback/input, I have found the following, divided into the following two categories:
- Techniques for shrinking the size of the data being copied
- Techniques for efficient data transfer over the network
Note: Obviously, the above techniques can very well can be combined.
Techniques for shrinking the size of the data being copied
- Put the files in a VDHX file and enable data deduplication on the file
- Zip the files using 7-Zip, WinRAR
- Zip the files to a WIM/ESD file using ImageX or DISM
- Zip the files using ZPAQ which uses data deduplication
Put the files in a VDHX file and enable data deduplication on the file
Quite easy, but requires Microsoft data deduplication support from the operating system, which limits you to Windows Server 2012 and above (or Windows 8.1 with some serious hacks). Here is a post on how to do that:
Beyond Zip – How to store 183 GB of VMs in a 16 GB file using PowerShell
https://deploymentresearch.com/406/Beyond-Zip-How-to-store-183-GB-of-VMs-in-a-16-GB-file-using-PowerShell
Zip the files using 7-Zip, WinRAR
Well, as it sounds, simply archive the files using ImageX or DISM.
- Pros: Easy to use, can be transferred over the network or via USB media (large USB 3.0 sticks are getting cheaper)
- Cons: Slow, and not very efficient on virtual machines
Zip the files to a WIM/ESD file using ImageX or DISM
About the same as the preceding 7-Zip/WinRAR, archive the files using ImageX or DISM. Archiving 107 GB of VMs (various Windows and System Center VMs) with highest compression (Ultra, solid archive) resulted in a 50 GB file, and it took about 10 hours to create the archive.
- Pros: Easy to use, can be transferred over the network or via USB media
- Cons: Slow, and not very efficient on virtual machines
Zip the files using ZPAQ which uses data deduplication
ZPAQ is a ZIP style program that uses data deduplication. Archiving 107 GB of VMs (various Windows and System Center VMs) resulted in a 16 GB file, and it took about 25 minutes to create the archive.
- Pros: Easy to use, very efficient on virtual machines, can be transferred over the network or via USB media
- Cons: Still being compressed, so it does takes some time, even though less time compared with 7-Zip / WinRAR or WIM/ESD files.
ZPAQ archived 107 GB of VMs (Win7, W81, WS2012R2 etc.) into a 16 GB file. Total time about 25 minutes.
Beyond ZIP Part II – Data DeDuplication Archives for all Windows versions via ZPAQ
https://deploymentresearch.com/493/Beyond-ZIP-Part-II-Data-DeDuplication-Archives-for-all-Windows-versions-via-ZPAQ
Techniques for efficient data transfer over the network
- Use multicast to transfer the file (works best with few large files)
- Use Torrents to transfer the files
- Use BranchCache to transfer the files
Use multicast to transfer the file (works best with few large files)
This was something I tried back in 2010, and it works quite well.
Multicasting – not only for operating system deployment
http://www.deployvista.com/Blog/JohanArwidmark/tabid/78/EntryID/151/Default.aspx
Use Torrents to transfer the files
Torrents are often connected with the dark side of Internet, but can be very useful for legit reasons as well. Credits goes to Mikael Nyström who mentioned they were using this for one of there customers in Sweden.
Here are some links:
Deploy large files over LAN with torrent
http://www.rarst.net/software/torrent-deploy-files
Transferring Files over your LAN via BitTorrent
http://www.instructables.com/id/Transfering-Files-over-your-LAN-via-BitTorrent/
Deploying Large Files in Local Network
http://kirsanov.net/post/2012/08/27/Deploying-Large-Files-in-Local-Network.aspx
Use BranchCache to transfer the files
BranchCache could also be used for limit the data transfer, and it actually supports Data DeDuplication (little known feature).
Have not tested this yet, but Andreas Hammarskjöld has put together a nice blog post explaining how it works:
BranchCache & De-Duplication
https://2pintsoftware.com/branchcache-dedup/