Treating WIM Files as Build Artifacts In a CI/CD Pipeline

One Does Not Simply ….

As the famous LotR internet meme says: “One Does Not Simply ” – Update Windows.

Anyone who has had to spend any amount of time using Windows will echo this statement. Windows Systems Administrators will have a more profane or colourful version, but Windows Updates remain an ongoing bug bear of Windows Users and Engineers alike. Unfortunately they are also unavoidable, in this age of cyber security threats and exploits; maintaining the integrity of your Windows estate is critical.

One way to reduce the number of updates needed is to maintain up to date WIM images (Windows Image files), containing all of the apps, configs, drivers and most Windows Updates that your machines will need. An up to date WIM image applied to a machine can minimise or even eliminate the time spent installing Windows Updates.

Keeping these WIM files up to date requires installing Windows Updates. The more WIM images you need to maintain the more Windows Updates will need to be installed.

Here at G-Research we made a decision to automate the creation of our WIM files and build a CI/CD pipeline around it. What else is a WIM image, if not a build artifact?

Our CI/CD pipeline consists of three rings for the tall elven kings major elements:

• Jenkins
• Ansible
• VMware

To trigger the pipeline we rely upon the Eye of Sauron Splunk.

The Lord of Mordor Splunk Sees All

Our infrastructure teams rely heavily upon Splunk to provide us knowledge, events and alerts around our environment.

One of our numerous exports of data comes from the SQL Database back end of our internal WSUS servers.

Every morning a Splunk Alert runs; if any updates show as being approved within the last 24 hours, Splunk will carry out three tasks:

• It sends an email to all appropriate Windows teams informing them that new updates have been approved, along with a table of what updates specifically
• It creates a new ticket within our ticketing system indicating it is time to release updated WIM images.
• It sends 7 webhooks, to 7 waiting Jenkins Pipelines.

Build Me a WIM File Worthy of Mordor

Seven Jenkins Pipelines in their halls of stone begin executing. We seperated the pipelines so that if any of the WIM fail to build it has no impact on the others.

The input provided by the pipelines is tightly controlled, so as to reduce the possibility of a malicious webhook being triggered to do something unsavoury.

Each pipeline is configured to trigger the same Ansible Playbook but provide it with different inputs resulting in the WIM images we need.

From an Ansible perspective each WIM type is assigned a specific Role within the playbook allowing us to fine tune each Role to suit its eventual purpose.

Nine Build Steps

Whilst the WIM files do have different Operating Systems (Windows 10, 2016 and 2019) and wildly different base applications a common process and approach is used in their forging*.

• Ansible creates an appropriately sized VM within VMware
• Ansible sets the VM to boot into Windows PE
• Windows PE installs the base operating system
• Ansible Configures the base operating system
• Ansible Installs all requested applications via Chocolatey
• Ansible Executes Windows Updates
• Ansible reboots the machine back into Windows PE
• Windows PE captures the contents of Drive C into a WIM file stored on a deployment file share
• Ansible Emails the Windows Engineering Team to inform them that a new WIM file is ready.
• Unit Tests are performed by Jenkins to confirm everything has completed successfully.

Nine build steps for mortal men doomed to die creating our WIM images.

_{*I do not use this comparison lightly, working with Windows can be all too similar to forging a piece of metal with a hammer – without the satisfaction of hitting something.}

It is worth mentioning that we rely on a similar process to the above for the creation and maintenance of our Virtual Machine Templates, however in their case the end product is a VM Template – rather than a build artifact.

In Summary

This article proves to Windows Administrators everywhere that the DevOps movement is as much for you as your Linux Administering Comrades and that with Open source tools you can transform Windows and allow it to take its place amongst your Codified Infrastructure.

Written by Carl Heinst, an Engineer at G-Research.