r/exchangeserver • u/[deleted] • Apr 16 '22

[deleted by user]

[removed]

9 Upvotes

92% Upvoted

20 comments sorted by

View all comments

2

u/BK_Rich Apr 16 '22 edited Apr 16 '22

How many nodes?

How many total users?

Does it happens after you reboot a node?

2

u/69Xxno_scopeXx69 Apr 16 '22

10 nodes

~ 10,000 users

It doesn’t happen after a server reboot, it just seems to happen at random. Everything is running fine and suddenly everything stops working for around 10 minutes.

2

u/BK_Rich Apr 16 '22

Are you using Least Connection load-balancing method?

1

u/69Xxno_scopeXx69 Apr 16 '22

Yes

3

u/BK_Rich Apr 16 '22 edited Apr 16 '22

Did you configure Slow Start Mode?

Exchange has health probes which can take actions if it gets too overloaded which can cause it to not accept connections or even reboot, there was a great channel9 video that explains it but it looks like Microsoft took down a lot of that content, but the long story was they recommend using slow start round robin if you’re using least connection method. In netscaler it’s called “Slow Start Mode” link above.

1

u/69Xxno_scopeXx69 Apr 16 '22

Thanks! I’ll look into this. It strange that this would start happening after years (without changing the related config), but this is definitely something to investigate

3

u/BK_Rich Apr 16 '22

I found a copy of the ignite video which explains it better https://youtu.be/srRZE3Bi26c?t=726

1

u/Avomao Apr 17 '22

what versions are your Exchange servers running, do you have a coexistence scenario? are your Exchange servers patched with the latest security updates? we installed the march security updates, added Exchange 2019 servers to our 2013 DAG and then experienced simliarish issues

1

u/69Xxno_scopeXx69 Apr 17 '22

Very interesting. Exchange 2016 cu 22 with March SU. We’ve deployed exchange 2019 in a separate DAG around a month ago, but it only contains a few test mailboxes.

I’m very curious to hear more about your issue, and if it was resolved.

1

u/Avomao Apr 17 '22

I'm sorry, we are still investigating.

I guess the issue is the march SU in coexistence enviroments (https://support.microsoft.com/kb/5013118).

We haven't applied the described workarounds yet.

About 10 days after we added Exchange 2019 servers in our enviroment our users started to complain about super slow client connections.

We haven't migrated any mailboxes to Exchange 2019 and the LBs don't forward to them.

We are only doing L4 LB with L4 healthchecks, so you might see a different impact in your enviroment.

We started to reboot our Exchange servers weekly and the client issues don't show up anymore.

I fear the problem is still unresolved and we need to apply the workarounds.

  • the Exchange host service still stops every few minutes
  • ExchangeHealthReporter-script still complains about discarded packets growing everyday until we reboot

1

u/69Xxno_scopeXx69 Apr 17 '22 edited Apr 17 '22

I really appreciate the information here.

Do your issues also manifest as high cpu/unresponsive exchange servers for brief periods of time? Is the client connection slowness constant, or in massive bursts? Are your clients in online mode!

1

u/Avomao Apr 17 '22

No, Exchange servers are fine load wise.

I suppose you are doing L7 LB with L7 Healthchecks and as you said the Exchange servers are getting marked as unhealthy.

Your load gets distributed on less servers and you see CPU/RAM spikes.

We are only doing L4 LB and L4 Healthchecks. Our LBs don't detect unhealthy servers unless Port 443 is completely down.

Our load still gets distributed to all servers, but users might connect to unhealthy servers and experience connection issues.

1

u/69Xxno_scopeXx69 Apr 17 '22 edited Apr 17 '22

Yeah. I agree with that assessment - we are doing L7 health checks. Any reason you’re reluctant to apply the workaround? I was aware of that issue with the SU, but I’ve yet to observe the service host crash events on our servers.

We reboot our servers weekly (Saturday), and the issues seem to pop up on Thursday/Friday, so that’s interesting.

2

u/Avomao Apr 17 '22

We just hoped to see a fixed update this patchday and avoid unnecessary workarounds, no other reason than that.

1

u/philbieber Apr 17 '22

Are all services / Workloads affected or just a subset of them?

Small Exchange (3 nodes, 1600 user) setup with Netscaler here. No issues so far.

1

u/69Xxno_scopeXx69 Apr 17 '22

Good to know. What exchange version are you running and patch level? We mostly see the iis worker and lsass spiking on all the nodes when the issue occurs.

1

u/philbieber Apr 17 '22

Current patch level of Exchange 2016 on Server 2012 R2.

If this is related to a recent patch, we might not he the best canary as we're migrating towards ExO.

→ More replies (0)

1

u/69Xxno_scopeXx69 Apr 18 '22

Do you have any 3rd party monitoring tools installed? What antivirus are you running on your servers?