This week started pretty harsh, a ton of customers reaching out to our team having WSUS issues. Everything from the "traditional" CPU and Memory spikes, to severe network traffic over port 8530 to the WSUS/SUP server. Basically Clients downloading massive amount of info, some customers reporting up to 700 MB per endpoint.
Note #1: One ongoing issue right now seem to be that the Windows version next updates contains a ton of metadata, causing a massive headache for WSUS admins. See below for scripts to help cleanup the mess, and to perform needed maintenance tasks. Also, if you are missing some info here, let me know. I'm @jarwidmark on Twitter.
WARNING: Whatever solution you pick for the maintenance of your WSUS/SUP server, ensure that you do not sync your WSUS/SUP during the maintenance process!
Until Microsoft replaces WSUS with something better, you have to do some housekeeping for WSUS to behave. Your mileage is going to vary, but you simply have to keep the WSUS database in shape, as well as declining unused updates. Here are a few resources that can help when WSUS goes bad.
The network traffic from WSUS can also be heavily loaded due an out-dated Microsoft Compatibility Appraiser version on the machines. See this KB:
Unexpected high network bandwidth consumption when clients scan for updates from local WSUS server
I have also published a PowerShell script you can run, either via remote PowerShell, or via the "Run Script" feature in ConfigMgr:
Checking the Microsoft Compatibility Appraiser version to prevent unwanted network traffic
Step 1 – Buy you some time
When all 8 CPU's on your site site server are constantly at 95-100 percent, there is little room for any admin work, nor cleanup. So make sure to throttle CPU on the WsusPool application pool, to give you some working room.
Here is a good write-up of the preceding steps.
ConfigMgr Software Update Point: Out-of-Control App Pool
Step 2 – More application pool settings, and the WSUS web.config file
Next step is to configure everything else in the application pool, together with the web.config file. I was lazy so I "borrowed" some settings from Sherry's post below, and added them to a PowerShell script: http://github.com/DeploymentResearch/DRFiles/blob/master/Scripts/Invoke-WSUSConfiguration.ps1
The script came from a series of ConfigMgr Configuration Items posted by Sherry Kissinger
WSUS Administration, WSUSPool, web.config, settings enforcement via Configuration Items
Step 3 – Decline weird stuff
Use any or all of the listed solutions to get rid of junk in your WSUS database:
Tip: Before starting to run decline scripts, PowerShell / SQL etc., make sure your SUSDB is not heavily fragmented. Use the Maintenance Solution from Ola Hallengren to optimize the SUSDB indexes: http://ola.hallengren.com/
Optional Speed Tip: If you don't mind going totally unsupported, you can create additional indexes in the WSUS database that speeds up the cleanup dramatically. More info here: http://kickthatcomputer.wordpress.com/2017/08/15/wsus-delete-obsolete-updates, a great post by Scott Williams (@ip1). Again, not supported by Microsoft so don't blame me if something happens 🙂 Fun fact: In my environment that change made the deletions go 30 times faster!!!
Here is a copy of the "code": http://github.com/DeploymentResearch/DRFiles/blob/master/Scripts/Create-WSUS-Index.sql
Decline weird stuff #1 – Fully Automate Software Update Maintenance in Configuration Manager
As the title implies, a script that automates software updates, including cleanup, optimization and more. Written by Bryan Dam (@bdam555).
Update April 17, 2018: Bryan recently updated the script to support standalone WSUS too, below you find a sample syntax for that:
.\Invoke-DGASoftwareUpdateMaintenance.ps1 -UpdateListOutputFile .\UpdateListOutputFile.csv -StandaloneWSUS WSUS01 -RunCleanUpWizard -DeclineSuperseded -DeclineByTitle @('*Itanium*','*ia64*','*Beta*') -DeclineByPlugins -Force
Decline weird stuff #2 – SQL Cleanup scripts
Some shiny SQL scripts from paul salwey @psalwey
Especially checkout the WSUSSQLMaintenance_4_DeclineUpdates_XML_Lengthover5000.sql one. I had not seen that before.
Tip on usage:
- Obsolete script
- Superseded script
- XML script
- Reindex again
- Reboot server
Tip #1: If you have a lot of obsolete updates (Script 2.), consider using an alternate version below that runs in batches, and also shows total number of updates. The script is from Scott Williams (see Resource #6 further down this post). I just added a comment on where to change the batch numbers.
Tip #2: If you just want to quickly see how many obsolete updates you have, use this script:
Tip #3: Benjamin Reynolds (@SqlBenjamin), with Microsoft, has put together a combination of creating indexes for speed with a more optimized version of cleaning up obsolete updates, and Steve Thompson (@Steve_TSQL), has it all explained and published here: http://stevethompsonmvp.wordpress.com/2018/05/01/enhancing-wsus-database-cleanup-performance-sql-script/
Decline weird stuff #3 – Decline Updates Script by Jeff Carreon
In the same post as the SQL script to view updates with large metadata (In the "Additional Resources" section" further down this post), you find a great decline update scripts by Jeff Carreon (@jeffctangsoo10). It's in the same post as the SQL script, but kind of hidden if you don't look carefully. Here is a direct link:
By default the script run in "What-If" mode ($TrialRun set to $True). Here is a syntax to run it in declining mode, without sending an email report:
.\Run-DeclineUpdate-CleanupV3.ps1 -Servers CM01 -TrialRun:$false -EmailReport:$false
Decline weird stuff #4 – WSUS Automated Maintenance (Formerly Adamj Clean-WSUS
I have not personally tested this one, but the community seems to like it quite a bit. Cleanup and DB script from Adam Marshall (@Adamj_1)
Here follows some additional resources that I found useful:
Resource #1 – Script to view updates with large metadata
Here is another contribution from Sherry's team. This SQL script was put together by Jeff Carreon, after working with Microsoft support on a WSUS performance issue. Very shiny.
The script is used to identify and measure the metadata that the clients are downloading, it tells you what articles (fancy word for update metadata) the are deployable and the size of each article.
Resource #2 – The complete guide to Microsoft WSUS and Configuration Manager SUP maintenance
Info from Microsoft. The title is a bit misleading, since it's not actually a complete guide. But there is still lots of good info.
Resource #3 – Clients cannot report Scan Results back to WSUS
During the day, Matthew Krause (@MatthewT_Krause) also provided info on an issue he was having: Quite many clients, 75 percent out of 6500,were not reporting back the scan results to WSUS. Basically the server got overloaded with IIS 500 errors as the clients kept trying to report scan results, fail, and then try again. In the WindowsUpdate.log on the client they found that clients would get the error message stating invalid parameter but the sub message was Message:parameters.InstalledNonLeafUpdateIDs (see below).
WindowsUpdate.log on a client failing to report back scan results.
So if you are running into the non-leaf error message, one solution that proved to be working was changing the maxInstalledPrerequisites value in the WSUS Web.config file, and then do an IIS Reset. Doing this change made 90% of clients report scan results back within one day for this environment.
Change WSUS Web.config from:
<add key="maxInstalledPrerequisites" value="400"/>
<add key="maxInstalledPrerequisites" value="800"/>
Resource #4 – Optimizing WSUS with Configuration Manager, via Adaptiva
Good WSUS overview article with a few technical tricks in it. Written by Matt Tinney (@mnt2556) from Windows Management Experts.
Resource #5 – Unleash WSUS performance, via Pawel Jarosz
Here is another reading I found useful.
Simon says – unleash WSUS performance
Resource #6 – WSUS Delete Obsolete Updates, via Scott Williams
Yet another useful resources. Written by Scott Williams (@ip1).
WSUS Delete Obsolete Updates
That all for now,
Happy Deployment / Johan