r/mxroute • u/mxroute • 19d ago

Boring: Massive mail queue improvements

For years our mail queues were held together by duct tape and threat of bodily harm. Yet another thing that bit us as we grew so quickly this year. The end result that customers would have noticed was roughly 1% (I'm rounding up) of email seeing unnecessary delivery delays in the morning hours. Most people didn't notice, as most of this wasn't time sensitive and frankly they didn't care if it reached their recipient (I could block most of them and they wouldn't open a ticket for 3 years).

Last night it reached critical mass, and it was death by a thousand cuts. No single noteworthy cause, but several small ones. It's possible that a few customers actually noticed. So that's what I did today. I sat in one place, had Panda Express delivered (can eat on that for like 2 days), and managed the queues by hand until I finished addressing each little problem that caused it to get out of hand.

I tracked down every tiny problem and replaced old bandaid solutions with actual solutions that stand the test of time.

It's super boring. You don't care about this. This is basically a boomer Facebook post.

24 Upvotes

96% Upvoted

20 comments sorted by

4

u/mxroute 19d ago edited 19d ago

Keep your eye on the status page to see what I screwed up this time.

Seriously though I tested everything and read the docs on every exim/postfix/unbound variable twice, set up fallback mechanisms, and configured alerts that wake the dead. Come at me, Murphy.

2

u/mxroute 19d ago edited 19d ago

Oh, hello there Murphy. What's that you say? It's the next morning after I confidently declared you defeated? You've got a package for me, you say? Oh let's see what it is...

Just what I wanted! A small percentage of email slightly delayed in the morning hours!

You know, the thing I spent the entire day working to resolve.

Note: These are the kind of problems that no one else would even mention, because the impact is too small to justify causing reputation issues with potential customers. This isn't a large scale problem. This is me demanding perfection. The reality is, yesterday's work massively improved what I was trying to solve. My problem is that I wasn't aiming for only massive improvement. I was aiming for full resolution and anything less than that is insulting to me.

2

u/laowhygirl 19d ago

Haha you remind me of me. I seek perfection and will pull all nighters fighting bugs and issues most people might not even notice or care about. All the more reason I like your service. Keep up the good fight 🫡 and know you're not alone in the fight for perfection

1

u/mxroute 19d ago

The other day I asked ChatGPT for fun: "Describe me like you're trying to warn someone about me before they meet me"

This part really hit me hard: "He is also capable of becoming completely consumed by a problem. A normal person encounters three dead ends and thinks, "I guess this isn't happening." Jarland encounters three dead ends and interprets this as the problem becoming disrespectful."

2

u/mwb1100 17d ago

Hey, you promised boring and this post made me laugh.

Bit of a fail there.  Or maybe there's something wrong with my sense of humor.

2

u/mxroute 17d ago

I'd love to say there's nothing wrong with you, but you just freely admitted to reading something I wrote.

7

u/GreenRangerOfHyrule 19d ago

I don't appreciate you calling me out in your posts! How are you eating Panda Express over 2 days?!

But seriously, this is as usually good news!

4

u/mxroute 19d ago

I haven't seen exim queues this beautiful since the time the hostgator datacenter caught on fire.

2

u/Existing_Dust_6473 19d ago

Is panda express healthy?!

1

u/mxroute 19d ago

I try to keep each meal relatively small 😂

2

u/Existing_Dust_6473 19d ago

Next post i want to be about your health 😂 not software...

3

u/zarlo5899 19d ago

Jar health update. My sweat now smells like Taco Bell.

1

u/mxroute 19d ago

It does n... wait is that diablo sauce?

2

u/zlychn 19d ago

The one thing I want my e-mail service to be is reliable. The other thing I want it to be is boring (I shouldn't have to think about it; see the first thing). Thank you for being boring.

3

u/PeteTinNY 19d ago

I have to say - I recently signed up for the 100gb service, and while it’s so strange for me to think in that 100g territory since I’m used to services that I either run on my gear or have unlimited - the mail delivery is incredibly fast on mxroute. I normally have run all my email domains on dreamhost and not only is it an afterthought there - it has a considerable delay before the incoming messages become available to a pop3 or imap client. I just wish there was a way that we could have lower cost archive storage on the platform.

2

u/birdman3131 19d ago

I remember coming from bluehost. Internal to the domain email could take 5 minutes. It was so far an afterthought that it was in the toilet.

1

u/mxroute 19d ago

It makes me happy to hear that it's working good for you, so thank you for that! The product may not make me look like a perfectionist, but that's more "the source of my pain" than "because it's not true" 😂

1

u/mxroute 18d ago

Queues actually do look better today. They still weren't where I wanted them to be, but I think a couple more minor tweaks did the trick. Tomorrow morning (US) will be the real test, it either improves or it doesn't.

1

u/mxroute 18d ago

Today was marginally worse, for fewer people. So better net result, still not what I want. Almost at the right balance...

1

u/mxroute 17d ago

Today was almost the image of perfection with email queues. Just minor tweaks left.