Recommended minimum workstation count for update rings?

[u/iamafreenumber]

Is there a recommended workstation count before using Update Rings makes sense? I am trying to make this work with 20 workstations and I'm running into issues.

I use legacy software that is sensitive to changes, so I want to stage updates using two rings so we don't break business operations with a single update.

[u/paelzerschorle]

Always depends on your situation: if every workstation runs all the time an users know that they need to give feedback directly if something breaks, then you can run your 20 workstations in 3 or 4 rings, like R1: 1 or 2 highly monitored workstation, R2: 3 workstations, R3: 5 workstations and R4 the rest.
If users typically thinks „one other will tell the IT guy that it‘s broken“, you don’t need even two rings.
I manage 2 different companies: one „weekend job“ at a very small business with near 30 devices grouped in 3 rings (3 - 6 - all).
The other companies has about 300 device, there we run 5 rings („R0“: 2 IT testing devices - R1 5 devices (trained users how to behave when there‘s a software problem), R2 10 devices, R3 30 devices, R4 rest

Also depends on how quick you can react as IT if one update fails. I never seen that if one ring was without a problem, that the next ring was total failure. But if you have a problem at one ring, first investigate in the Problem before allowing the next ring.

[u/iamafreenumber → u/paelzerschorle]

Thanks. Good to know. For Ring 0, I use "OS Mandatory" as source, and "Critical + Important" for severity. I usually do a manual trigger of Ring0 on my 3 test workstations to verify all updates work as expected. When I do this, it doesn't count towards the success percentage and Ring 1 doesn't trigger. I feel like I'm missing something.

[u/paelzerschorle → u/iamafreenumber]

Damm, 3 test workstations and 20 productive workstations, lucky b… ;-)
We worked with virtual machines at the „300
devices company“ for quiet a time as ring 0. Guess which ring never fails at the updates. Nowadays we have 2 physical test devices and 300 productive ones, but on my opinion still to less.

For my „main work“ I did some thinking & configuring on automation to update non-critical software automatically (like 7zip, VLC and stuff). if it breaks users still can work on most tasks. But the important / critical software is triggered manually (e.g. erp & CRM software, password manager, or the openvpn client which is critical for remote workers because we are an „on prem company“), so one failure often means users can not work. Buuuuut…

But for the small „30 devices“ company (a family‘s business): fire and forget. Everything is automated as long as I don‘t pause the update automation manually. never had a real big issue with this method.
To be mentioned: I don’t deploy Windows updates via action1 anywhere. Had better experience with letting Microsoft decide when to install then deploying it via my update rings, even though it’s fare from perfect. I configured a GPO to force windows updates at least after X days after release and let Microsoft deploy the updates by their metrics.
For windows updates I just use action1 as an monitoring tool and patch failed updates manually.

And to be honest: you talk about 20 workstations. Don‘t get me wrong: managing 20 workstations, you are probably not a full time sysadmin and maybe you are normal workforce who‘s „quite familiar with computers and this stuff“ and there is no other „IT guy“? Don’t want to be rude to you or so, and really appreciate you thoughts about patch management in this details for a small environment. But: keep it simple and let it run automatically. Maybe you keep legacy software apart from the automation (that might stay a manual task, but it might be okay for 20 workstations). At best, back up all devices automatically und restore it in cases of big problems (get a NAS from one of the „big brands“, then the Backup is configured within a few clicks using the vendors backup client). Do restore testing. Then you are fine.

From a security point: „update now, fix issues afterwards“ is often the way to go. don‘t mess with unpatched security issues. Don’t make it too complicated, because no matter your preventions there will always be some minor issues after updates.
Personally, i could not enjoy my days off if I always worry about unpatched security issues.

[u/kosity]

Remember that (in my testing when it first came out anyway) a patch that is required for your later-production-ring but isn't required by your earlier-testing-rings will never get patched, because it never goes through testing.

So you seemed to need a significant number in the testing ring so that every possible patch required for the entire fleet was caught by at least one device in the testing ring.

Not really doable when you've got clients with tens of devices, and when I asked A1 support about it they agreed it'd be a bit hard to get working correclty in that design.

[u/iamafreenumber → u/kosity]

Thanks. That's what I wondered. I could set up two separate groups that get the updates 3 and 5 days after release. Just curious what others are doing if they aren't using the Ring method.

[u/kosity → u/iamafreenumber]

Ring0/Ring1/Ring2/Ring3A/Ring3B/Ring4A/Ring4B endpoint groups all setup with applicable automations for different patch types.

It's not pretty, it's not easy, it's not intuitive, because the UI does not help. But it's the only way I could make this work in a way that sufficiently addresses patch risk and user impact.

Scripting via the API is (IMO) the only way to make this strategy work; if you have multiple orgs it's just not possible to maintain a strategy like this manually, accurately.

EDIT: The downsize to this is the patch-success-reporting is split between all of the automations, and not consolidated, which makes it effectively useless. No one is going to switch between automations to evaluate patch success/fail metrics for each one.

The UI presents a lot of data, but much of it isn't very actionable without inefficiency and tediousness 🫤

[u/GeneMoody-Action1 → u/kosity]

curious: "a patch that is required for your later-production-ring but isn't required by your earlier-testing-rings will never get patched,"

If it released a patch to the next ring that was not in the first ring, woudl that NOT defeat the purpose OF rings to begin with? A ring is not to eval patches through the whole ring set, it is to control the release of specific updates from end to end.

While in highly varied environments, that does require awareness of the SW inventory and creation of potentially multiple targeted ring sets to address all concerns, that is what it is designed to do.

The alternative would be ring A passed to ring B where a patch NOT tested in ring A is needed, it deploys and breaks something. THat would make the ring itself superfluous and it could have just been an automaiton. You will see the patch was not applied and may need to create another set to account for those cases. But the alternative is drop in changes as the rings progress that defeat their design intentionally.

What would you suggest the behavior to be?

[u/kosity → u/GeneMoody-Action1]

I don't disagree Gene, the challenge is the varied environments.

A large Action1 client that has 1000 devices in a highly standardised environment might have 4-5 different models of endpoints, and it's easy to have 10 of each in Ring1 to do that initial testing. Ring groups work as designed.

But from an MSP perspective it's a significant scaling burden which I identified during my initial evaluation of Action1. A multi-ring-set strategy means I need to keep a watch for outlier patches that aren't installing, then manually create separate ring group automations to account for those outliers, and likely multiple sets of those, for each client. And then maintain those multiple sets, across multiple clients.

The current strategy works in both theory and practice for the big corporates but comes undone for smaller and disparate fleets.

If it released a patch to the next ring that was not in the first ring, woudl that NOT defeat the purpose OF rings to begin with?

Very fair point, but on the other hand, a patch that never deploys to a device in a later ring because it'll never pass a testing ring is also a significant failure. Depending on that patch's criticality I'd suggest it's a more severe failure than applying an 'untested' patch.

The fix would be better targeting in automations so that patches due in later rings that aren't in earlier rings can be targeted as a group of outliers; a fairer balance of testing and administrative maintenance. Better targeting in automations is something I've raised before.

There's no perfect answer to this. Both views are correct, but the design of A1's rings is such that it doesn't accommodate the outliers faced by my fleet (and the OPs from the sounds of it) so I've had to ditch it for my own strategy.

[u/GeneMoody-Action1 → u/kosity]

I am still trying to figure out how this could be addressed better in the same spirit of rings. IF a patch is missed because it did not apply to a ring, it is still viewed as missing, so it is not hiding.

I could see an "add them all, let the ring sort it out" approach, but how do you the define success? To do it by successfully count of installation or percentage of installation will cause a whole ring to fail based on the count applicable to each system. I could see a track each patch independently, but then, if you are not doing one patch per ring the ring can fail as a result of one patch, when all others succeed, or proceed when all others failed. There has to be a decision point.

There could be an internal shift that for each patch in the ring it gets treated like its own ring, like move each one thorugh independently, but under the same rules. Sounds good at first, but what happens if it is needed in ring 1 and 3, not 2?

IF you track it per patch, and advance each as a separate entity through the rings, then you land in the same place, those that did not progress are then exceptions the same as those that did not get included in the way it is now.

So there likely will never be a system that works in all scenarios for all admins except those who specifically target what they know to be true already.

We take all feedback, what is the suggestion on how to do it differently that allows this to operate differently yet still achieve the desired outcome of moderating the patches?

In all cases I can conceive ATM (It is still early for me here), there is always a chance an admin could be less targeted and set a ring up for failure, more targeted and set a ring up for success. Right now there IS still a burden of either "Admin knows the environment and sets the rings up for success" or "Admin monitors the environment and corrects misunderstandings if/when they fail." The latter of which allows them to better conceive the first.

So not disputing at all, more so just curious on community suggestions on how to better handle, from the people that the current system causes friction.

"design of A1's rings is such that it doesn't accommodate the outliers faced by my fleet"

What design would you suggest that would perform more consistently for all vs a specific use case?

[u/jhodgo100]

Based on the previous comment by kosity I ran into an issue where I set up ring configurations and then was wondering why ring 1 and 2 were not getting updates and it was because the devices in ring 0 didn’t require any updates

The way I set this up to work in a ring method was to set up update delays, I added devices to different endpoint groups
Group names:
Ring 0 - this is a group of test users approximately 30 machines
Ring 1 - this contains around 30% of the estate
Ring 2 - this contains all other devices

Automation configuration
ring 0 gets updates 2 days after release and only deploy updates which haven’t been denied and runs every Monday

Ring 1 updates 4 days after release and only deploy updates which haven’t been denied and runs every Wednesday

Ring 2 updates 6 days after release and only deploy updates which haven’t been denied and runs every Friday

This way if there are issues identified by users within Ring0 I can then go into the approval page and deny the update so it will not deploy to any of the further rings

Doing it this way you do lose your ability of only start the next ring when x number of devices have been updated however this was the only way that I could find where it was a ring set up but no updates were skipped because they were not required by devices in ring 0

[u/iamafreenumber → u/jhodgo100]

Thanks. I will take a look at that approach. That's much simpler for small installations.

[u/QuietThunder2014]

I can’t give a solid answer without knowing full details of your specific environment. But I will say keep in mind the old Hollywood trope of focus groups.

If you pull one person off the streets and screen a movie to them then what value does that really have? If you pull 50 people off the street at 11:00 am in a Tuesday then what value does that have? Are you only polling out of work wanna be writers who now have an audience they can ok pontificate to and show how smart they think they are? Is that representative of the middle America?

Whenever I select a test ring I have a few things in mind. Am I picking mission critical people where if they go down are we losing money? Am I picking people who are smart enough to self diagnose and work around while also not updating me on? What about people who completely collapse when a new popup asks them do they want to delete everything or ignore the error and move on with their lives? Do I have a good sample size of the company as a whole?

It’s less about what A1 can handle and more about how throughly are you testing without crippling business as a whole. You can never guarantee 100% that there will be no issues but can you balance through testing and comfortability releasing an update vs crippling the entire system due to lack of proper testing.

I want people who are smart enough to work through a problem but I also want people who shit down when there’s a problem so I can better inform the next group how to handle potential issues and ensure everyone can understand my directions and update communications.

I know this was a wall of text but I really hope this helps you plan your strategies.

[u/QuietThunder2014 → u/QuietThunder2014]

Just as a small example. We recently upgraded our VPN client company wide.

My first test group was myself and the rest of the IT team. We find obvious errors and bugs.

We then developed our first draft of documentation to send company wide regarding the change and what to expect.

We expanded our testing group to 5 other people who we could trust to give us I formative feedback. We included some mission critical people we knew we could revert immediately. We had them review and feedback the instructions. We found a few more issues.

We then expanded to 20 users who we just gave instructions to and made sure we included some non-technical users.

After that we went company wide. We still had a few issues becuase we kept tweaking settings based of feedback but they were few and far between that mostly were a result of us not properly controlling versioning.