Windows Server 8 - Deduplication awesomeness
Mar
18
Written by:
3/18/2012 1:36 PM
This was the weekend when my main demo environment was reinstalled with Windows Server 8... I love the new Deduplication feature, a feature that removes duplicate chunks in files on the file system, even inside VHD files etc... Meaning if you have many files that are full or partly identical, the data will only claim the real hard drive space for the chunks in the files that are redundant.
Disclaimer:
Please note that Deduplication works best for content folder stores, virtualization depots or backup stores etc. It's not really intended for live, constantly changing data, like Hyper-V hosts.
Warning:
Never, ever run the scheduled dedup chunking jobs when you have vms running (I crashed a few vm's beyond salvation during a scheduled dedup operation). For a demo hyper-v host, disable all the dedup schedules in task scheduler, and only run it manually every once in a while when all machines are turned off.
Estimate the Deduplication savings
If you want to examine how much space you can save on a volume, without actually enabling deduplication, you can run the ddpeval.exe tool. You can also copy the ddpeval.exe file from a Windows Server 8 installation to a Windows Server 2008 R2 machine and run it. Useful to find out if your machine would benefit from a Windows Server 8 upgrade in terms of deduplication. You can run ddpeval.exe against local drives or remote shares.
Here is the output from running ddpeval.exe on one of my Windows Server 2008 R2 deployment servers.
Enabling Deduplication
Deduplication is a File Services role that you add via server manager, and after doing that you can enable data deduplication on your data drives (not the os volume).
In my demo environment I had a few Hyper-V hosts with about a terabyte of virtual machines and ISO files. After installing Windows Server 8 beta and restoring my backup of files, my disks looked like this... E.g. before adding deduplication
The I added the Deplication file services role via server manager, and forced an immediate data deduplication schedule (via the task scheduler). After about three hours or so my drives looked like this:
I still have the same data on the drive, I just have 372 GB free space instead of only 73 GB. Life is good... :)
For more info about Deduplication, check the following
Data Deduplication Planning and Deployment
http://technet.microsoft.com/en-us/library/hh831700.aspx
/ Johan
12 comment(s) so far...
Re: Windows Server 8 - Deduplication awesomeness
I haven't seen a solution for the VMware thing, but I expect it to be fixed shortly.
/ Johan
By Arwidmark on
3/28/2012 9:00 AM
|
Re: Windows Server 8 - Deduplication awesomeness
Deduplication has nothing to do with the compact operation of the vm, and never ever run dedup when you have running vm's... I crashed five vm's during a scheduled dedup operation... for demo machines, disable all the dedup schedules in task scheduler, and only run it manually every once in a while when all machines are closed...
/ Johan
By Arwidmark on
6/8/2012 10:00 AM
|
Re: Windows Server 8 - Deduplication awesomeness
Todd, this is good to know, thanks for sharing the issue with the NIC. That's definitely a show stopper for me. I understand that this is Beta code we're tallking about but I would expect to have at least the basic drivers work. In my case, I was not planning on running Server 8 natively but rather in a VM. Just wondering if Johan knows a work around for the issue with VMware tools.
Once Server 8 is released, I"ll move to that OS on my laptop and switch to Hyper-V over VMware Workstation.
By rroman on
3/26/2012 4:02 AM
|
Re: Windows Server 8 - Deduplication awesomeness
Thanks Johan.
By rroman on
3/30/2012 4:40 AM
|
Re: Windows Server 8 - Deduplication awesomeness
Johan, a while ago you posted that VMware tools breaks Server 8 on boot up which I also learned on my own! Did you find a way around this or are you running Server 8 as your host OS with Hyper-V?
I run my labs from a W7 HP EliteBook 8650w with Dual Quad Core and 16GB RAM. Nice setup but I can't fully enjoy Server 8 without the tools so I haven't updated my labs yet. I'm looking to build a new CM12 and Server 8 lab to get ready for MMS. Thank you sir and I'll see you next month and June at TechEd Orlando!
By rroman on
3/23/2012 10:08 AM
|
Re: Windows Server 8 - Deduplication awesomeness
Awesomeness indeed! Now if only they would release this for Windows 8 as well as the server edition.
By bartrumb on
3/18/2012 8:35 PM
|
Re: Windows Server 8 - Deduplication awesomeness
Hi Johan,
Have you considered placing your VMs on a virtual volume/storage space? I'm thinking maybe not since it would require you to keep track of when physical disk space is running out. I guess what I'm really wondering (for other than IT Pro purposes ;-) is whether or not deduplication works on a storage space volume, and I bet you know the answer.
Thanks,
Carl
By extcon on
8/6/2012 4:50 AM
|
Re: Windows Server 8 - Deduplication awesomeness
Johan, speaking of rebuilding your demo environment, what is the ideal environment for developing Windows 8 client test images and deploying them using MDT 2012? My current setup is running Windows Server 2008 R2 with build and lite-touch MDT 2012 in VMware virtual machines. Are you now running Windows Server 8 with build and lite-touch MDT 2012 in Hyper-V?
By toddmeyers on
3/23/2012 2:05 AM
|
Re: Windows Server 8 - Deduplication awesomeness
rroman, I'm in the same boat. My lab is a fully-loaded HP 8760w which I'm hoping to rebuild prior to MMS. I would LOVE to see Johan write a post about building the ultimate demo environment with all of the latest tools in mind. I'm sure he has tried every configuration by now, whereas I haven't had the time yet and can't really afford much downtime to experiment.
By toddmeyers on
3/23/2012 3:20 PM
|
Re: Windows Server 8 - Deduplication awesomeness
Well, I have tested the Windows Server 8 waters now, and my Intel Centrino 6300 NIC is not working. Looks like the driver is fine, but no wireless networks can be found. Windows 8 Consumer Preview works just fine. Obviously not affecting everyone, but a bunch of Google links find I am not alone in that regard. The troubleshooter says "unable to bind the TCP/IP stack" or something like that, so this is not going to work for me until that is resolved. The wired NIC works, but since the entire point is to create a mobile lab, I do rely on the wireless working. Disappointed.
By toddmeyers on
3/23/2012 11:27 PM
|
Re: Windows Server 8 - Deduplication awesomeness
rroman,
To fix the VMWare tools issue, choose a custom install and unselect the video driver ( from memory, if this does not work try other drivers ). I did get this going on vmware workstation 7. Or get your hands on the beta build e.x.p build-646643, works fine with this version.
jd
By jd on
4/15/2012 4:33 PM
|
Re: Windows Server 8 - Deduplication awesomeness
I have a Windows 2012 RC hyper-v host running a Windows 2012 RC hyper-v guest with deduplication installed along with DFS-R. We replicated 250 GB of data via DFS-R, and the guests's vhdx file for the data drive grew to 275gb as expected.
We then applied deduplication (set days to 0 and manually forced job) and it shrunk to about 125 GB.
We then shut down the guest and ran a Compact operation on the vhdx. It finished pretty much immediately and did not shrink the vhdx at all. Any ideas why?
Thanks!
By pesos on
6/7/2012 6:56 PM
|