Patch Fast
-
@scottalanmiller said in Patch Fast:
In today's world we can snapshot and roll back patches so easily that the threats from bad patches are normally trivial. And it is not like the vendors have not already tested the patches. These are not beta releases, these are already tested in environments much larger and more demanding than our own.
And yet still, mistakes can happen. Two of our vendors here have had to recall patches because they caused more problems than they fixed (can't fuss at Microsoft... this time)... and they were released for days before the recalls happened.
But as you say, this is the reason we should have snapshots and backups to recover from said mistakes and bad patches. There is no real reason for businesses of any size to not be able to backup (at bare minimum) and / or snapshot their systems before running patches.
-
@dafyre said in Patch Fast:
@scottalanmiller said in Patch Fast:
In today's world we can snapshot and roll back patches so easily that the threats from bad patches are normally trivial. And it is not like the vendors have not already tested the patches. These are not beta releases, these are already tested in environments much larger and more demanding than our own.
And yet still, mistakes can happen. Two of our vendors here have had to recall patches because they caused more problems than they fixed (can't fuss at Microsoft... this time)... and they were released for days before the recalls happened.
But as you say, this is the reason we should have snapshots and backups to recover from said mistakes and bad patches. There is no real reason for businesses of any size to not be able to backup (at bare minimum) and / or snapshot their systems before running patches.
Yes exactly, the days of painful patching are behind us. Patching always has risk, but planned risk with great mitigation. But the risks of not patching are continuing to grow at quite a pace.
-
@dafyre said in Patch Fast:
There is no real reason for businesses of any size to not be able to backup (at bare minimum) and / or snapshot their systems before running patches.
Who here snapshots their systems before patching their Microsoft servers? Scott says it's so easy to snapshot and roll back, so perhaps I'm missing a trick here? I can see that it's easy if you're manually installing patches, but who does that?
The other problem is that you may not realise that a patch has broken something for a couple of days, and by then it's likely to be too late to satisfactorily restore from backup.
-
@Carnival-Boy said in Patch Fast:
@dafyre said in Patch Fast:
There is no real reason for businesses of any size to not be able to backup (at bare minimum) and / or snapshot their systems before running patches.
Who here snapshots their systems before patching their Microsoft servers? Scott says it's so easy to snapshot and roll back, so perhaps I'm missing a trick here? I can see that it's easy if you're manually installing patches, but who does that?
The other problem is that you may not realise that a patch has broken something for a couple of days, and by then it's likely to be too late to satisfactorily restore from backup.
We schedule our snapshots here (VMware) to run an hour before our patch time... and we do the patches manually.
-
Tell me more. How often do you patch? Does the same person do it? When do you do it, Sundays? How do you to check that server applications aren't getting broken?
I need to get more organised and am looking for best practice.
-
@Carnival-Boy said in Patch Fast:
Tell me more. How often do you patch? Does the same person do it? When do you do it, Sundays? How do you to check that server applications aren't getting broken?
I need to get more organised and am looking for best practice.
I don't know about "best practices" but what we do here...
Every SysAdmin has a list of systems they are responsible for. So the systems we are responsible for are also the ones we patch. We have a daily maintenance Window from 6am to 7am for patches and software upgrades and such.
-
That's ok at a larger organisation, but trickier at a smaller one where there's only one or two IT staff, or they use an MSP. Having a maintenance window during the week is nice though.
-
@Carnival-Boy said in Patch Fast:
Tell me more. How often do you patch? Does the same person do it? When do you do it, Sundays? How do you to check that server applications aren't getting broken?
I need to get more organised and am looking for best practice.
We patch every six hours with a randomizer to keep patching from pounding our WAN. So each server has a few hours of randomization, but update four times a day. We don't snap before patching, because we use primarily Linux and the risks are effectively zero because patches are better tested, patch footprint is smaller, the patching events are smaller (four times a day, not one time a week) and patch rollbacks are trivial.
-
@Carnival-Boy said in Patch Fast:
That's ok at a larger organisation, but trickier at a smaller one where there's only one or two IT staff, or they use an MSP. Having a maintenance window during the week is nice though.
If you use an MSP it would be simple. Just tell your MSP what patch process you want
-
@Carnival-Boy Patches are applied with yum-cron or dnf-automatic. Snapshots are taken before any system changes, and after testing is completed, but not before or after patching.
-
-
Can't edit the last link due to wifi issues. But here is the real link...
http://www.sccmog.com/sccm-powercli-auto-snapshot-before-patching-task-sequence-script/
-
Never used this but take a look...
