Extreme Deployment – Techniques for deploying large files over LAN

It’s been an interesting week, with some great discussions and research on how to transfer larges files to a lot of computers as efficient as possible. Time to write down some of that research. The scenario is: You have a bunch of machine, say student computers, you have limited infrastructure, and you want to deploy large files (apps, virtual machines, lab files etc.). What are your options for doing that?  Well so far, and here is were I would love to get your feedback/input, I have found the following, divided into the following two categories:

  • Techniques for shrinking the size of the data being copied
  • Techniques for efficient data transfer over the network

Note: Obviously, the above techniques can very well can be combined.

world map technology style against fiber optic background

Techniques for shrinking the size of the data being copied

  • Put the files in a VDHX file and enable data deduplication on the file
  • Zip the files using 7-Zip, WinRAR
  • Zip the files to a WIM/ESD file using ImageX or DISM
  • Zip the files using ZPAQ which uses data deduplication

Put the files in a VDHX file and enable data deduplication on the file

Quite easy, but requires Microsoft data deduplication support from the operating system, which limits you to Windows Server 2012 and above (or Windows 8.1 with some serious hacks). Here is a post on how to do that:

Beyond Zip – How to store 183 GB of VMs in a 16 GB file using PowerShell
https://deploymentresearch.com/406/Beyond-Zip-How-to-store-183-GB-of-VMs-in-a-16-GB-file-using-PowerShell

Zip the files using 7-Zip, WinRAR

Well, as it sounds, simply archive the files using ImageX or DISM.

  • Pros: Easy to use, can be transferred over the network or via USB media (large USB 3.0 sticks are getting cheaper)
  • Cons: Slow, and not very efficient on virtual machines

Zip the files to a WIM/ESD file using ImageX or DISM

About the same as the preceding 7-Zip/WinRAR, archive the files using ImageX or DISM. Archiving 107 GB of VMs (various Windows and System Center VMs) with highest compression (Ultra, solid archive) resulted in a 50 GB file, and it took about 10 hours to create the archive.

  • Pros: Easy to use, can be transferred over the network or via USB media
  • Cons: Slow, and not very efficient on virtual machines

Zip the files using ZPAQ which uses data deduplication

ZPAQ is a ZIP style program that uses data deduplication. Archiving 107 GB of VMs (various Windows and System Center VMs) resulted in a 16 GB file, and it took about 25 minutes to create the archive.

  • Pros: Easy to use, very efficient on virtual machines, can be transferred over the network or via USB media
  • Cons: Still being compressed, so it does takes some time, even though less time compared with 7-Zip / WinRAR or WIM/ESD files.

image
ZPAQ archived 107 GB of VMs (Win7, W81, WS2012R2 etc.) into a 16 GB file. Total time about 25 minutes.

Beyond ZIP Part II – Data DeDuplication Archives for all Windows versions via ZPAQ
https://deploymentresearch.com/493/Beyond-ZIP-Part-II-Data-DeDuplication-Archives-for-all-Windows-versions-via-ZPAQ

Techniques for efficient data transfer over the network

  • Use multicast to transfer the file (works best with few large files)
  • Use Torrents to transfer the files
  • Use BranchCache to transfer the files

Use multicast to transfer the file (works best with few large files)

This was something I tried back in 2010, and it works quite well.

Multicasting – not only for operating system deployment
http://www.deployvista.com/Blog/JohanArwidmark/tabid/78/EntryID/151/Default.aspx

Use Torrents to transfer the files

Torrents are often connected with the dark side of Internet, but can be very useful for legit reasons as well. Credits goes to Mikael Nyström who mentioned they were using this for one of there customers in Sweden.

Here are some links:

Deploy large files over LAN with torrent
http://www.rarst.net/software/torrent-deploy-files

Transferring Files over your LAN via BitTorrent
http://www.instructables.com/id/Transfering-Files-over-your-LAN-via-BitTorrent/

Deploying Large Files in Local Network
http://kirsanov.net/post/2012/08/27/Deploying-Large-Files-in-Local-Network.aspx

Use BranchCache to transfer the files

BranchCache could also be used for limit the data transfer, and it actually supports Data DeDuplication (little known feature).

Have not tested this yet, but Andreas Hammarskjöld has put together a nice blog post explaining how it works:

BranchCache & De-Duplication
https://2pintsoftware.com/branchcache-dedup/

About the author

Johan Arwidmark

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments

>