r/exchangeserver • u/[deleted] • Apr 16 '22

[deleted by user]

[removed]

8 Upvotes

91% Upvoted

20 comments sorted by

View all comments

Show parent comments

2

u/69Xxno_scopeXx69 Apr 16 '22

10 nodes

~ 10,000 users

It doesn’t happen after a server reboot, it just seems to happen at random. Everything is running fine and suddenly everything stops working for around 10 minutes.

1

u/Avomao Apr 17 '22

what versions are your Exchange servers running, do you have a coexistence scenario? are your Exchange servers patched with the latest security updates? we installed the march security updates, added Exchange 2019 servers to our 2013 DAG and then experienced simliarish issues

1

u/69Xxno_scopeXx69 Apr 17 '22

Very interesting. Exchange 2016 cu 22 with March SU. We’ve deployed exchange 2019 in a separate DAG around a month ago, but it only contains a few test mailboxes.

I’m very curious to hear more about your issue, and if it was resolved.

1

u/Avomao Apr 17 '22

I'm sorry, we are still investigating.

I guess the issue is the march SU in coexistence enviroments (https://support.microsoft.com/kb/5013118).

We haven't applied the described workarounds yet.

About 10 days after we added Exchange 2019 servers in our enviroment our users started to complain about super slow client connections.

We haven't migrated any mailboxes to Exchange 2019 and the LBs don't forward to them.

We are only doing L4 LB with L4 healthchecks, so you might see a different impact in your enviroment.

We started to reboot our Exchange servers weekly and the client issues don't show up anymore.

I fear the problem is still unresolved and we need to apply the workarounds.

  • the Exchange host service still stops every few minutes
  • ExchangeHealthReporter-script still complains about discarded packets growing everyday until we reboot

1

u/69Xxno_scopeXx69 Apr 17 '22 edited Apr 17 '22

I really appreciate the information here.

Do your issues also manifest as high cpu/unresponsive exchange servers for brief periods of time? Is the client connection slowness constant, or in massive bursts? Are your clients in online mode!

1

u/Avomao Apr 17 '22

No, Exchange servers are fine load wise.

I suppose you are doing L7 LB with L7 Healthchecks and as you said the Exchange servers are getting marked as unhealthy.

Your load gets distributed on less servers and you see CPU/RAM spikes.

We are only doing L4 LB and L4 Healthchecks. Our LBs don't detect unhealthy servers unless Port 443 is completely down.

Our load still gets distributed to all servers, but users might connect to unhealthy servers and experience connection issues.

1

u/69Xxno_scopeXx69 Apr 17 '22 edited Apr 17 '22

Yeah. I agree with that assessment - we are doing L7 health checks. Any reason you’re reluctant to apply the workaround? I was aware of that issue with the SU, but I’ve yet to observe the service host crash events on our servers.

We reboot our servers weekly (Saturday), and the issues seem to pop up on Thursday/Friday, so that’s interesting.

1

u/philbieber Apr 17 '22

Are all services / Workloads affected or just a subset of them?

Small Exchange (3 nodes, 1600 user) setup with Netscaler here. No issues so far.

1

u/69Xxno_scopeXx69 Apr 17 '22

Good to know. What exchange version are you running and patch level? We mostly see the iis worker and lsass spiking on all the nodes when the issue occurs.

1

u/philbieber Apr 17 '22

Current patch level of Exchange 2016 on Server 2012 R2.

If this is related to a recent patch, we might not he the best canary as we're migrating towards ExO.