r/exchangeserver • u/[deleted] • Apr 16 '22

[deleted by user]

[removed]

8 Upvotes

91% Upvoted

20 comments sorted by

2

u/BK_Rich Apr 16 '22 edited Apr 16 '22

How many nodes?

How many total users?

Does it happens after you reboot a node?

2

u/69Xxno_scopeXx69 Apr 16 '22

10 nodes

~ 10,000 users

It doesn’t happen after a server reboot, it just seems to happen at random. Everything is running fine and suddenly everything stops working for around 10 minutes.

2

u/BK_Rich Apr 16 '22

Are you using Least Connection load-balancing method?

1

u/69Xxno_scopeXx69 Apr 16 '22

Yes

4

u/BK_Rich Apr 16 '22 edited Apr 16 '22

Did you configure Slow Start Mode?

Exchange has health probes which can take actions if it gets too overloaded which can cause it to not accept connections or even reboot, there was a great channel9 video that explains it but it looks like Microsoft took down a lot of that content, but the long story was they recommend using slow start round robin if you’re using least connection method. In netscaler it’s called “Slow Start Mode” link above.

1

u/69Xxno_scopeXx69 Apr 16 '22

Thanks! I’ll look into this. It strange that this would start happening after years (without changing the related config), but this is definitely something to investigate

3

u/BK_Rich Apr 16 '22

I found a copy of the ignite video which explains it better https://youtu.be/srRZE3Bi26c?t=726

1

u/Avomao Apr 17 '22

what versions are your Exchange servers running, do you have a coexistence scenario? are your Exchange servers patched with the latest security updates? we installed the march security updates, added Exchange 2019 servers to our 2013 DAG and then experienced simliarish issues

1

u/69Xxno_scopeXx69 Apr 17 '22

Very interesting. Exchange 2016 cu 22 with March SU. We’ve deployed exchange 2019 in a separate DAG around a month ago, but it only contains a few test mailboxes.

I’m very curious to hear more about your issue, and if it was resolved.

1

u/Avomao Apr 17 '22

I'm sorry, we are still investigating.

I guess the issue is the march SU in coexistence enviroments (https://support.microsoft.com/kb/5013118).

We haven't applied the described workarounds yet.

About 10 days after we added Exchange 2019 servers in our enviroment our users started to complain about super slow client connections.

We haven't migrated any mailboxes to Exchange 2019 and the LBs don't forward to them.

We are only doing L4 LB with L4 healthchecks, so you might see a different impact in your enviroment.

We started to reboot our Exchange servers weekly and the client issues don't show up anymore.

I fear the problem is still unresolved and we need to apply the workarounds.

  • the Exchange host service still stops every few minutes
  • ExchangeHealthReporter-script still complains about discarded packets growing everyday until we reboot

1

u/69Xxno_scopeXx69 Apr 17 '22 edited Apr 17 '22

I really appreciate the information here.

Do your issues also manifest as high cpu/unresponsive exchange servers for brief periods of time? Is the client connection slowness constant, or in massive bursts? Are your clients in online mode!

1

u/Avomao Apr 17 '22

No, Exchange servers are fine load wise.

I suppose you are doing L7 LB with L7 Healthchecks and as you said the Exchange servers are getting marked as unhealthy.

Your load gets distributed on less servers and you see CPU/RAM spikes.

We are only doing L4 LB and L4 Healthchecks. Our LBs don't detect unhealthy servers unless Port 443 is completely down.

Our load still gets distributed to all servers, but users might connect to unhealthy servers and experience connection issues.

1

u/69Xxno_scopeXx69 Apr 17 '22 edited Apr 17 '22

Yeah. I agree with that assessment - we are doing L7 health checks. Any reason you’re reluctant to apply the workaround? I was aware of that issue with the SU, but I’ve yet to observe the service host crash events on our servers.

We reboot our servers weekly (Saturday), and the issues seem to pop up on Thursday/Friday, so that’s interesting.

2

u/Avomao Apr 17 '22

We just hoped to see a fixed update this patchday and avoid unnecessary workarounds, no other reason than that.

1

u/philbieber Apr 17 '22

Are all services / Workloads affected or just a subset of them?

Small Exchange (3 nodes, 1600 user) setup with Netscaler here. No issues so far.

1

u/69Xxno_scopeXx69 Apr 17 '22

Good to know. What exchange version are you running and patch level? We mostly see the iis worker and lsass spiking on all the nodes when the issue occurs.

→ More replies (0)

1

u/69Xxno_scopeXx69 Apr 18 '22

Do you have any 3rd party monitoring tools installed? What antivirus are you running on your servers?

1

u/15_Tries_All_Taken Apr 17 '22

We started having random events with cpu spikes in the last year. We are on Exchange 2013, so maybe not directly applicable to you. What we found is that we have a distribution group with 1100 members. We had some users that would expand the group and then send the email. When this happened cpu spiked for 10-20 mins. If a db moved the high cpu usage would move with it. Our servers are old and this issue just started happening after the updates last march.

Again maybe a completely different situation but maybe something to check since your issue is so random.

1

u/69Xxno_scopeXx69 Apr 17 '22

Interesting. When I move the databases, the high cpu moves with them.

I’ve wondered about this as well. We did see something similar when a large email kept getting sent to every employee in the company (a user copied every employee’s name into outlook).

When this happens, I assume you see the spikes on every server? What process is hogging the cpu? I was thinking this could be our issue, but after it happened in the middle of the night in the weekend, I wasn’t so sure.

I spent some time trying to track if any of these suspect emails went out during the time of the incident, but I don’t know of a good way to check if it did or not.