Happy New Year 2021
In this post, we will discuss updating VMs using azure update management.
Companies are increasingly subject to computer attacks.
It is therefore essential to keep track of and maintain the updates on your fleet.
Presentation:
Azure update management is a component of azure automation that allows you to manage your updates like SCCM.
To monitor the status of updates, update management requires Log Analytics.
You will find on the following link an ARM template allowing you to deploy the whole solution:
https://docs.microsoft.com/fr-fr/azure/automation/update-management/enable-from-template
Deploying agents on machines:
You need to deploy the agents on your machines before you attach them to the solution.
Azure provides an initiative (Enable Azure Monitor fo VMs) to deploy the different agents according to machine types:
Warning: before running this policy and attaching your machines to your Log Analytics, check first if they are not already attached to another one.
To do this, you can use the following Azure graph query (replace ## Central Log Analytics Workspace ID ## with the id of your Log Analytics, or delete this line):
Resources
| where type == "microsoft.compute/virtualmachines/extensions"
| where properties.publisher in ("Microsoft.EnterpriseCloud.Monitoring", "Microsoft.Azure.Monitor")
| where properties.settings.workspaceId != "## Central Log Analytics Workspace ID ##"
| extend vmid = split(id,'/')
| project properties.type, RG = vmid[4], VMName = vmid[8], location
You can check that your agent has been deployed using the following Log Analytics query:
Heartbeat | where Computer == "My virtual machine".
If your agent does not appear, you can check if it is attached to the correct Log Analytics using the following azure graph query (replace ## ENTER YOUR VM NAME HERE ## with your VM name):
Resources
| where type == "microsoft.compute/virtualmachines/extensions"
| where properties.publisher in ("Microsoft.EnterpriseCloud.Monitoring", "Microsoft.Azure.Monitor")
| extend vmid = split(id,'/'), workspaceId = properties.settings.workspaceId
| project properties.type, RG = vmid[4], VMName = vmid[8], location, workspaceId
| where VMName == "## ENTER YOUR VM NAME HERE ##"
Example:
Link your agents to update management:
To attach your machines, go to the 'Update Management' section of your Automation.
Once your machines are attached, they should appear in the update management console after 15 minutes.
If this is not the case, check that your machine is listed in the Log Update of your Log Analytics.
Update | where Computer == "ubuntu1804"
If your machine does not come back up even after restarting your agent, run the diagnostic script:
- for windows: https://www.powershellgallery.com/packages/Troubleshoot-WindowsUpdateAgentRegistration/1.1
- For linux: https://gallery.technet.microsoft.com/scriptcenter/Troubleshooting-utility-3bcbefe6
Please note that some systems such as redhat8 are not compatible (RedHat 8 should be made compatible mid-February this year).
The list of compatible OS can be found here:
https://docs.microsoft.com/fr-fr/azure/automation/update-management/overview
If your machines have been shut down, they will no longer be visible in the console.
However, you can check that a machine has already been attached in the hybrid worker groups menu.
Planning updates
Scheduling updates is done via the 'Schedule update deployment' icon.
There are two menus for selecting your machines 'Groups to Update' and 'Machines to update'.
The 'Groups to Update' menu allows you to select your Azure machines via filters (subscription, RG, location and tags).
The 'Machines to update' menu allows you to select your machines individually or via a query.
Warning: selecting machines via this last menu requires that your machines are started.
An interesting option is update management, which offers the possibility of running scripts before and after the update.
However, it is only possible to run powershell scripts.
Microsoft provides scripts to manage the start-up and shutdown of your machines.
These scripts are available in the runbooks galery:
- UpdateManagement-TurnOnVms
- UpdateManagement-TurnOffVms
If you want to run several scripts in pre or post task, you will not be able to do so via the menu.
You should either :
- create a runbook that calls the different runbooks via the Start-AzAutomationRunbook command or a webhoob call
- create a single script that performs the various actions desired
The client I work for wanted to perform the following functions in prescript
- create a snapshot for machines that have not had updates for a long time
- Start the machines if they are stopped
- Do not run updates if all machines in the same AvailabilitySet are scheduled at the same time
Below you will find the scripts made for this purpose (based on 'UpdateManagement-TurnOnVms' and 'UpdateManagement-TurnOffVms'):
UpdateManagement-AzVMSnapShot.ps1:
#source: UpdateManagement-TurnOnVms
# https://docs.microsoft.com/fr-fr/azure/virtual-machines/windows/snapshot-copy-managed-disk
<#
.SYNOPSIS
Start VMs as part of an Update Management deployment and create a snapshot
.DESCRIPTION
This script is intended to be run as a part of Update Management Pre/Post scripts.
It requires a RunAs account.
This script will ensure all Azure VMs in the Update Deployment are running so they recieve updates.
This script will store the names of machines that were started in an Automation variable so only those machines
are turned back off when the deployment is finished (UpdateManagement-AzTurnOffVMs.ps1)
.PARAMETER SoftwareUpdateConfigurationRunContext
This is a system variable which is automatically passed in by Update Management during a deployment.
#>
[CmdletBinding()]
Param
(
#This is a system variable which is automatically passed in by Update Management during a deployment.
[String]$SoftwareUpdateConfigurationRunContext,
#this parameter define the percentage of vm in an AvSet who can't be launched in the same update.
[double]$avset_purcentage_check = 66.7,
#this parameter define if you want to snapshot all vm before the update
[ValidateSet('True','False')]
[String]$createsnapshot = 'True'
)
#region ---modules---
#endregion ---modules---
#region ---Functions---
#endregion ---Functions---
#region ---variables---
#Requires -version 5
#requires -Modules ThreadJob
$ErrorActionPreference = 'Stop'
#endregion ---variables---
#region ---body---
Try {
#region BoilerplateAuthentication
#This requires a RunAs account
$ServicePrincipalConnection = Get-AutomationConnection -Name 'AzureRunAsConnection'
Add-AzAccount `
-ServicePrincipal `
-TenantId $ServicePrincipalConnection.TenantId `
-ApplicationId $ServicePrincipalConnection.ApplicationId `
-CertificateThumbprint $ServicePrincipalConnection.CertificateThumbprint
#endregion BoilerplateAuthentication
#If you wish to use the run context, it must be converted from JSON
$context = ConvertFrom-Json $SoftwareUpdateConfigurationRunContext
$vmIds = $context.SoftwareUpdateConfigurationSettings.AzureVirtualMachines
$runId = "PrescriptContext" + $context.SoftwareUpdateConfigurationRunId
if (!$vmIds)
{
#Workaround: Had to change JSON formatting
$Settings = ConvertFrom-Json $context.SoftwareUpdateConfigurationSettings
$vmIds = $Settings.AzureVirtualMachines
if (!$vmIds)
{
throw "No Azure VMs found"
}
}
#region create avset variable
$avsetlist = @()
foreach ($sub in (Get-AzSubscription).SubscriptionID)
{
Select-AzSubscription $sub
$avsetlist += Get-AzAvailabilitySet | where {$_.VirtualMachinesReferences.id.count -gt 1}
}
foreach ($avset in $avsetlist)
{
#check for each AvSet if (the number of vm plan for this update / the number of the VM in the AvSet) is greater than ($avset_purcentage_check / 100)
if ( (($avset.VirtualMachinesReferences.id | foreach {if ($_ -in $vmIds) {$_}}).count / $avset.VirtualMachinesReferences.id.count) -gt ($avset_purcentage_check/100))
{
throw "AvSet $($avset.Name) in ResourceGroup $($avset.ResourceGroupName) contain more than $avset_purcentage_check% of its VM"
}
}
#endregion create avset variable
#https://github.com/azureautomation/runbooks/blob/master/Utility/ARM/Find-WhoAmI
# In order to prevent asking for an Automation Account name and the resource group of that AA,
# search through all the automation accounts in the subscription
# to find the one with a job which matches our job ID
Select-AzSubscription -SubscriptionId $ServicePrincipalConnection.SubscriptionID
$AutomationResource = Get-AzResource -ResourceType Microsoft.Automation/AutomationAccounts
foreach ($Automation in $AutomationResource)
{
$Job = Get-AzAutomationJob -ResourceGroupName $Automation.ResourceGroupName -AutomationAccountName $Automation.Name -Id $PSPrivateMetadata.JobId.Guid -ErrorAction SilentlyContinue
if (!([string]::IsNullOrEmpty($Job)))
{
$ResourceGroup = $Job.ResourceGroupName
$AutomationAccount = $Job.AutomationAccountName
break;
}
}
#This is used to store the state of VMs
New-AzAutomationVariable -ResourceGroupName $ResourceGroup -AutomationAccountName $AutomationAccount -Name $runId -Value "" -Encrypted $false
$updatedMachines = @()
$startableStates = "stopped" , "stopping", "deallocated", "deallocating"
$jobIDs= New-Object System.Collections.Generic.List[System.Object]
#create scriptblock var (this script block create a snapshot and start the VM)
$ScriptBlock = {
param
(
$resource,
$vmname,
$vmlistid,
$vmId,
$AutomationJobId
)
$vm = Get-AzVM -ResourceGroupName $resource -Name $vmname
$snapshot = New-AzSnapshotConfig `
-SourceUri $vm.StorageProfile.OsDisk.ManagedDisk.Id `
-Location $vm.location `
-CreateOption copy
New-AzSnapshot `
-Snapshot $snapshot `
-SnapshotName "BeforeUpdate_${vmname}_${AutomationJobId}" `
-ResourceGroupName $resource
if ($vmId -in $vmlistid)
{
Start-AzVM -ResourceGroupName $resource -Name $vmname
}
}
#Parse the list of VMs and start those which are stopped
#Azure VMs are expressed by:
# subscription/$subscriptionID/resourcegroups/$resourceGroup/providers/microsoft.compute/virtualmachines/$name
$vmIds | ForEach-Object {
$vmId = $_
$split = $vmId -split "/";
$subscriptionId = $split[2];
$rg = $split[4];
$name = $split[8];
Write-Output ("Subscription Id: " + $subscriptionId)
Select-AzSubscription -Subscription $subscriptionId | Out-Null
#Query the state of the VM to see if it's already running or if it's already started
$state = ($(Get-AzVM -ResourceGroupName $rg -Name $name -Status).Statuses[1].DisplayStatus -split " ")[1]
if($state -in $startableStates) {
Write-Output "Starting '$($name)' ..."
#Store the VM we started so we remember to shut it down later
$updatedMachines += $vmId
if ($createsnapshot -eq 'False')
{
$newJob = Start-ThreadJob -ScriptBlock { param($resource, $vmname) Start-AzVM -ResourceGroupName $resource -Name $vmname} -ArgumentList $rg,$name
$jobIDs.Add($newJob.Id)
}
}
else
{
Write-Output ($name + ": no action taken. State: " + $state)
}
# create a snapshot and start the vm if $createsnapshot -eq $true
if ($createsnapshot -eq 'True')
{
$newJob = Start-ThreadJob -ScriptBlock $ScriptBlock -ArgumentList $rg,$name,$updatedMachines,$vmId,$($context.SoftwareUpdateConfigurationRunId)
$jobIDs.Add($newJob.Id)
}
}
$updatedMachinesCommaSeperated = $updatedMachines -join ","
#Wait until all machines have finished starting before proceeding to the Update Deployment
$jobsList = $jobIDs.ToArray()
if ($jobsList)
{
Write-Output "Waiting for machines to finish starting..."
Wait-Job -Id $jobsList
}
foreach($id in $jobsList)
{
$job = Get-Job -Id $id
if ($job.Error)
{
Write-Output $job.Error
}
}
Write-output $updatedMachinesCommaSeperated
#Store output in the automation variable
Set-AzAutomationVariable -Name $runId -Value $updatedMachinesCommaSeperated -ResourceGroupName $ResourceGroup -AutomationAccountName $AutomationAccount -Encrypted $false
}
Catch {
$ErrorMessage = $_.Exception.Message
$ErrorLine = $_.InvocationInfo.ScriptLineNumber
Write-error "UpdateManagement-AzVMSnapShot : Error on line $ErrorLine. The error message was: $ErrorMessage"
}
#endregion ---body---
UpdateManagement-AzTurnOffVms.ps1:
#source: UpdateManagement-TurnOffVms
<#PSScriptInfo
.VERSION 1.1
.GUID 9606f2a1-49f8-4a67-91d6-23fc6ebf5b3b
.AUTHOR zachal
.COMPANYNAME Microsoft
.COPYRIGHT
.TAGS UpdateManagement, Automation
.LICENSEURI
.PROJECTURI
.ICONURI
.EXTERNALMODULEDEPENDENCIES ThreadJob
.REQUIREDSCRIPTS
.EXTERNALSCRIPTDEPENDENCIES
.RELEASENOTES
Removed parameters AutomationAccount, ResourceGroup
.PRIVATEDATA
#>
<#
.DESCRIPTION
This script is intended to be run as a part of Update Management Pre/Post scripts.
It requires a RunAs account.
This script will ensure all Azure VMs in the Update Deployment are running so they recieve updates.
This script works with the Turn Off VMs script. It will store the names of machines that were started in an Automation variable so only those machines are turned back off when the deployment is finished.
#>
<#
.SYNOPSIS
Stop VMs that were started as part of an Update Management deployment
.DESCRIPTION
This script is intended to be run as a part of Update Management Pre/Post scripts.
It requires a RunAs account.
This script will turn off all Azure VMs that were started as part of TurnOnVMs.ps1.
It retrieves the list of VMs that were started from an Automation Account variable.
.PARAMETER SoftwareUpdateConfigurationRunContext
This is a system variable which is automatically passed in by Update Management during a deployment.
#>
param(
[string]$SoftwareUpdateConfigurationRunContext
)
#region ---variables---
#Requires -version 5
#requires -Modules ThreadJob
$ErrorActionPreference = 'Stop'
#endregion ---variables---
#region ---body---
Try {
#region BoilerplateAuthentication
#This requires a RunAs account
$ServicePrincipalConnection = Get-AutomationConnection -Name 'AzureRunAsConnection'
Add-AzAccount `
-ServicePrincipal `
-TenantId $ServicePrincipalConnection.TenantId `
-ApplicationId $ServicePrincipalConnection.ApplicationId `
-CertificateThumbprint $ServicePrincipalConnection.CertificateThumbprint
Select-AzSubscription -SubscriptionId $ServicePrincipalConnection.SubscriptionID | Out-Null
#endregion BoilerplateAuthentication
#If you wish to use the run context, it must be converted from JSON
$context = ConvertFrom-Json $SoftwareUpdateConfigurationRunContext
$runId = "PrescriptContext" + $context.SoftwareUpdateConfigurationRunId
#https:/github.com/azureautomation/runbooks/blob/master/Utility/ARM/Find-WhoAmI
# In order to prevent asking for an Automation Account name and the resource group of that AA,
# search through all the automation accounts in the subscription
# to find the one with a job which matches our job ID
$AutomationResource = Get-AzResource -ResourceType Microsoft.Automation/AutomationAccounts
foreach ($Automation in $AutomationResource)
{
$Job = Get-AzAutomationJob -ResourceGroupName $Automation.ResourceGroupName -AutomationAccountName $Automation.Name -Id $PSPrivateMetadata.JobId.Guid -ErrorAction SilentlyContinue
if (!([string]::IsNullOrEmpty($Job)))
{
$ResourceGroup = $Job.ResourceGroupName
$AutomationAccount = $Job.AutomationAccountName
break;
}
}
#Retrieve the automation variable, which we named using the runID from our run context.
#See: https://docs.microsoft.com/en-us/azure/automation/automation-variables#activities
$variable = Get-AutomationVariable -Name $runId
if (!$variable)
{
Write-Output "No machines to turn off"
return
}
$vmIds = $variable -split ","
$stoppableStates = "starting", "running"
$jobIDs= New-Object System.Collections.Generic.List[System.Object]
#This script can run across subscriptions, so we need unique identifiers for each VMs
#Azure VMs are expressed by:
# subscription/$subscriptionID/resourcegroups/$resourceGroup/providers/microsoft.compute/virtualmachines/$name
$vmIds | ForEach-Object {
$vmId = $_
$split = $vmId -split "/";
$subscriptionId = $split[2];
$rg = $split[4];
$name = $split[8];
Write-Output ("Subscription Id: " + $subscriptionId)
Select-AzSubscription -Subscription $subscriptionId | Out-Null
$vm = Get-AzVM -ResourceGroupName $rg -Name $name -Status
$state = ($vm.Statuses[1].DisplayStatus -split " ")[1]
if($state -in $stoppableStates) {
Write-Output "Stopping '$($name)' ..."
$newJob = Start-ThreadJob -ScriptBlock { param($resource, $vmname) Stop-AzVM -ResourceGroupName $resource -Name $vmname -Force} -ArgumentList $rg,$name
$jobIDs.Add($newJob.Id)
}else {
Write-Output ($name + ": already stopped. State: " + $state)
}
}
#Wait for all machines to finish stopping so we can include the results as part of the Update Deployment
$jobsList = $jobIDs.ToArray()
if ($jobsList)
{
Write-Output "Waiting for machines to finish stopping..."
Wait-Job -Id $jobsList
}
foreach($id in $jobsList)
{
$job = Get-Job -Id $id
if ($job.Error)
{
Write-Output $job.Error
}
}
}
Catch
{
$ErrorMessage = $_.Exception.Message
$ErrorLine = $_.InvocationInfo.ScriptLineNumber
Write-error "New-VMSnapShot : Error on line $ErrorLine. The error message was: $ErrorMessage"
}
finally
{
#Clean up our variables:
Remove-AzAutomationVariable -AutomationAccountName $AutomationAccount -ResourceGroupName $ResourceGroup -name $runID
}
#endregion ---body---
Conclusion:
Update management is quite easy to handle.
However, it has some limitations:
- it does not support all OS types (e.g. redhat8 at the moment)
- it supports the attachment of only one Log Analytics per machine
- pre and post script execution only supports powershell 5.1 (higher versions are not currently supported by azure automation).
About the author
Article written by Vincent Fléchier
15 years of expertise in IT, including 3 years in the Azure Cloud, Vincent is a certified Azure Cloud Architect.
His specialities: powershell, ansible, azure...
