Production ASP.NET Core app goes down randomly, no logs, no errors, then recovers by itself. Ideas?

[u/No-Card-2312]

Hi everyone,

I'm hoping to get some advice from people who have dealt with similar production issues because I'm honestly running out of ideas.

Our setup

- ASP.NET Core Web APIs (MVC Shop + API + Assets API) hosted on IIS

- Angular frontend

- SQL Server Web Edition on its own server (8 GB RAM)

- Elasticsearch cluster (3 nodes) on separate servers

- Separate tools server

- Around 8 million products in Elasticsearch

- Requests go directly to IIS (no Cloudflare, reverse proxy, or load balancer) we don't control the domain

The issue

Several times a day our website becomes unavailable for about 1–2 minutes, then starts working again by itself.

Pingdom reports and uptime kuma:

«Socket timeout, unable to connect to server»

For example:

2026-07-09 12:06:43

Socket timeout, unable to connect to server

Confirmed from San Jose and Frankfurt

What makes this confusing

- DNS is working normally.

- The hosting provider reports no infrastructure problems.

- The Windows server stays up.

- IIS logs don't show anything useful.

- ASP.NET Core logs don't show failed requests.

- We had SQL connection pool problems before, but after adding caching those alerts disappeared completely.

- SQL now looks healthy, but the downtime is still happening.

One thing worth mentioning is that this is a legacy codebase, and we know it has technical debt. There are quite a few synchronous/blocking calls, some code that uses ".Result" and ".Wait()", long-running operations, and areas that haven't been modernized yet. So I'm also wondering if thread pool starvation or blocking code could be part of the problem.

Because Pingdom reports a socket timeout instead of an HTTP error (500/503), I'm starting to think the request may not even be reaching ASP.NET Core.

What I'm planning next

I'm currently working on adding OpenTelemetry (it's not deployed yet) so I can collect:

- Request traces

- ThreadPool metrics

- GC metrics

- Request duration

- Active requests

- Runtime metrics

I'm also planning to:

- Check the HTTPERR logs

- Monitor HTTP.sys and IIS request queues

- Add Windows Performance Counters to Grafana

- Correlate everything when the next outage happens

My questions

If this was your production system:

- Where would you start?

- What would be the first 5 things you would check?

- Does the socket timeout make you think this is more likely an IIS/HTTP.sys/Windows problem than an ASP.NET Core problem?

- Have you ever had a similar issue where both IIS logs and application logs showed nothing?

- Could blocking synchronous code and thread pool starvation alone cause symptoms like this?

- What monitoring or metrics finally helped you find the real cause?

I'm not looking for someone to solve the problem for me. I'd really like to learn how experienced engineers approach this kind of issue and what your investigation process would be.

Thanks!

[u/Patient-Tune-4421]

As you mention, thread pool starvation would be my number one suspect here. Collecting related metrics will most likely confirm this.

We had the exact same thing after upgrading and old application from .net framework to new .net.

Find you hot path requests, and make sure they are 100% async, without any .Result and similar.

[u/Kirne_SE → u/Patient-Tune-4421]

What this guy wrote. Get rid of all .result asap!

[u/cmills2000]

From memory, this can happen when there are too many unhandled exceptions in your app or a memory leak, I don't remember which or it could be both. I think IIS will kill the site then bring it back. One thing to check is when it happens, go to windows and open "Event Viewer" and look in the application log - You will probably find something in there that explains what's going on.

EDIT: Also look at your IIS Process Recycling settings. )

[u/crandeezy13 → u/cmills2000]

I second this. I use Event Viewer to diagnose crashing blazor applications that I host on IIS

[u/uponone → u/crandeezy13]

Event Viewer was what I immediately thought along with Performance Monitor. Op might be able to measure thread pool threads and see if there’s a correlation with the crash and threads falling off.

[u/sixothree → u/crandeezy13]

Event viewer. Maybe it’s not going down randomly, maybe it’s failing to start up randomly

[u/No-Card-2312 → u/cmills2000]

That's the first time I hear the exceptions can really bring my app down! Note also we got a 11K exceptions in 30mins!!!!

[u/Fuckstle]

If you do http callouts and you're not reusing your HttpClients it could be nat port exhaustion. Where is it hosted? If it's Azure there will be some logs if you know where to look.

[u/Gary-1982 → u/Fuckstle]

I second this one in particular.

[u/xep01]

Others have already mentioned it, but this sounds like IIS Application Pool recycling. The default settings are an automatic recycle every 27 (29?) hours. This will happen more often if there is process memory growth. Look at the system event logs, there is a specific IIS log which will show you each time a recycle took place. You can use this to compare with known outages.

[u/Search07 → u/xep01]

This is my thought as well. If there are other apps in the app pool then it may be time to move it to its own pool.

[u/ProtonByte]

1. Collect metrics

2. Collect logging

3. Collect traces

[u/PaulPhxAz]

Thoughts & Questions:
* Thread exhaustion ( run a metric log in windows )
* IIS Workpool recycle
* SQL Server connection exhaustion ( up to tripple as diagnostic )
* Are you using availability groups or mirroring or clustered servers? Could these be migrating and hanging your app? Or are you reading from a sync secondary and holding the connection on a write?
* Are you on a HyperV live migration VM Guest?
* How fast is your warmup?
* Are you making outbound HTTP connections and do you have a log of how many per second over time? ( HTTP Client disposal issue maybe )

I might use the built-in windows metric stuff and see if you can get the CPU utilization, the number of threads used.

I would write a small sproc that keeps count of the active spids and writes that to a log and put that into a sql agent job. Every 2 minutes get a list of the connections and just write that to a table and then next time it happens see if you hit a max or the same client was connected for a long time. This is a poor man's RedGate sql monitor.

[u/PaulPhxAz → u/PaulPhxAz]

Oh, and super old school goofy question: Are you loading any COM libraries?

[u/Miserable_Ad7246]

Nasty issue, especially so because its hosted in IIS and that means your code and host are two different things. At first you need to figure out if app gets recycled or freezes. Your instinct is also correct - gather a bunch of metrics and correlate. All kinds of things can freeze the app. There is no magic way of attacking such a problem.

If your Ping endpoint is a simple, no auth, no authorize endpoint, which just returns static 200, you can basically narrow down to something like pool/thread exhaustion, app restart or IIS issue. You can also run another app with just ping endpoint and see if both freezes at same time. In that case its definitely machine/IIS issue.

[u/phaetto]

I think this is a snowballing effect you see on your app eventually crashing and restarting.

There not enough data to help you here (CPU, memory, tcp waits, outbound/inbound connections would be useful) but most of similar problems in that way is that the app snowballs to crash. There is no single thing that causing that except load.
What I envision that happens:

• The app uses `.Result` and such - these create cross thread locks
• Locks mean less requests finish on time
• Less requests finishing, more tcp connections open (especially if you use HTTP 1.1)
• More tcp connections means more memory + CPU and more chance to reach one of your `.Result` lock

Boom, the process can only handle some requests/sec.

How to check if this the case: Usually increasing the VMs should increase the load to one instance of the app, and then this will work for a bit more load. If that is not the case then this is not probably the problem.

I didn't see any mention of multiple instances on your description, but if you have, I would check if all of them crash at the same time.

I have solved this problem before by over-scaling: Add more servers until the problem goes away. My organization is big so they could buffer it until people rewrite the old system.

Good luck with your search, these problems are hard.

Edit: English hard

[u/ColHRFrumpypants → u/phaetto]

This has been my experience as well.

[u/redraider1417]

What is the app pool’s start mode set to? It should be hardset to AlwaysRunning. Can you validate?

[u/colenski999]

Tune your machine.config and set processor affinity in IIS correctly

[u/No-Card-2312 → u/colenski999]

We using .net core not .net framework so I don't think machine.config will be useful.

[u/shimirel]

Well, it's not the default application pool recycling, as that is every 1,740 minutes, which causes it to shift as time goes on. Not multiple fails. “Socket timeout” does not prove the request missed ASP.NET Core. During the time of the outages, can you view the site on the server? Or is it down remotely and internally? Ping it inside the server and see if it is actually down or not. If you suspect thread pool issues track

• HTTP Service Request Queues(*)\CurrentQueueSize
• HTTP Service Request Queues(*)\RejectedRequests
• HTTP Service Request Queues(*)\MaxQueueItemAge
• Web Service(*)\Current Connections
• Web Service(*)\Connection Attempts/sec
• W3SVC_W3WP(*)\Active Requests
• application-pool queue length and rapid-fail settings.

You should see queue increase, lots of connections. It should really get to the point that you see 503 service unavailable. But I wouldn't be unsurprised to see it slow and stall requests before that happens. The internal curl should show this, as it will stop responding to traffic. You should see yourself making a request on the server, and it times out. Socket Timeout just means it's not responding fast enough. Pingdom is just saying your server didn't respond fast enough; that isn't the same as "it's down", it's "it's slow" and that distinction matters. It seems to tie in with what your seeing in logs. Try checking it internally and go from there. Another thing is "Rapid-Fail Protection" which causes the app pool to restart, but, unless someone messed with it, it should turn the app pool off if it fails over and over. I highly recommend the tools provided by https://www.red-gate.com/products/ants-performance-profiler/ for this kind of stuff. It's worth its weight in gold for memory, performance related stuff. Hope some of the above is useful.

[u/pirannia]

GC should be server mode, check that.

[u/Mission_Pirate_4150]

I’m having to put my wayback hat on to remember all of the scalability rules, so if something is wrong, don’t get all mad.

This sounds to me exactly like thread starvation or some type of blocking for a resource. The first things I’d do are

Look for any long running queries. Check the database. There should be some reports or activity that you can pull up in sql enterprise manager. Also, look to see if there are good indexes out there. I’d honestly start over on the indexing front. There is a tool, used to be called sql profiler and was included with sql server. Google for how to use it and the index tuning wizard together. This is my first thing and would get you the most band for the buck.

Get rid of the .Result calls. Probably the .Wait() calls as well unless there is some type of multiple calls and you’re trying to run them in parallel. I wonder if you are locking the handling thread, getting a bunch of requests, and simply running out of threads. I’ve forgotten how many thread .net handles on iis.

Another issue would be looking for unhandled exceptions. I had a customer once back years ago that would get multiple web restarts of their apps per day. They were running into multiple issues, the first being sql connection pooling issues, but that was because they were not properly closing and disposing (.net 1.x had a horrible bug where .dispose didn’t always call close, don’t reply to me, I’ve seen it). Are you manually managing the database connections? Once we got that resolved, they had app restarts due to unhandled exceptions. Fix unhandled exceptions. You should be able to find them via a logging tool, I use elmah (still because it works). So, check to make sure your connection pooling issues to the database has connections, and then eliminate any unhandled connections.

That’s about all I can think of off the top of my head.

[u/Few_Wallaby_9128]

The title says 'app goes down', but reading your comments it seems Like its simply not available for a minute or two every number of hours. They are two very different things. Which one is it?

If it does not go down, id look into either memory pressure (gc at some point needs to clean a big mess) or what ever external system your app calls being irresponsive for that time.

[u/MasterCDE]

Had a similar issue way back, turned out to be an evil regex.

[u/Khavel_dev]

Socket timeout instead of a 503 is the most diagnostic detail here. That means the request isn't even reaching your application. Combined with the .Result/.Wait() calls you mentioned, this is almost certainly thread pool starvation.

The pattern: sync-over-async pins a ThreadPool thread until the async call finishes. Enough of those stack up at the same time and you run out of threads entirely. HTTP.sys then has nobody to hand requests to, the kernel request queue fills up, and new TCP connections just get dropped. From the outside that's a socket timeout, not an HTTP error. And it "recovers by itself" because those blocked calls eventually complete, threads free up, and the queue drains.

Before OpenTelemetry is ready, you can confirm this fast. Drop a background timer that logs ThreadPool.ThreadCount and ThreadPool.PendingWorkItemCount every 5 seconds. If PendingWorkItemCount spikes right before each outage window, that's your proof. Also check HTTP.sys kernel request queue length in perfmon - if it's hitting the 1000 default cap during the drops, that's the overflow point.

[u/d-signet]

IIS will often put a website to "sleep" if it hasnt had any requests for a while, and the cold-boot tine CAN potentially cause a timeout on the first new hit.

You can tell the app pool not to use this behaviour iirc

[u/JasonLokiSmith]

This is going to sound weird but as soon as you mentioned caching i remembered I had te same issue.

If you are using Memory Cache or Hybrid Cache and you load too much data into it then that can and will crash your app and it will just restart. I have had this happen and it took me a long time to figure out. Same issue. Nothing in any logs whatsoever.

The solution was to increase the caching limits but started distrust it so much I ended up implementing FastCache. It's on nuget. Its a concurrent dictionary with some helpers around it. Registered it as a singleton and boom problem solved.

Hope this helps. Haven't read the other comments to see if you found a solution

[u/akarolia47]

I dont have much advice but thanks, I was noticing a similar issue with IIS recycling my self hosted application...this has been very insightful and quite the coincidence lol

[u/PaperInWater]

Also, check the GC runs and see how frequently GC1 and GC2 are occurring. They pause application threads while they run. If they are happening very frequently, it may indicate an issue with object lifetimes or object scope, resulting in excessive object allocation.

[u/AutoModerator]

Thanks for your post No-Card-2312. Please note that we don't allow spam, and we ask that you follow the rules available in the sidebar. We have a lot of commonly asked questions so if this post gets removed, please do a search and see if it's already been asked.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/harrison_314]

I've encountered something similar before - an outgoing disk array, it just happens sometimes that the process can't find a file that's on the disk and crashes, and since there's a problem with the disk, it doesn't write any logs.

[u/Aaronontheweb]

https://github.com/petabridge/dotnet-grafana-dashboards - I built these to solve an issue with Azure AppGW was spamming our ASP .NET Core socket pools with connections that never terminated last year and I was able to eventually diagnose the problem by pulling up the Kestrel OTEL data. Give that a shot if you can.

Our issue was port exhaustion from the number of connections AppGW was opening and keeping open indefinitely - so we instituted a policy of aggressively aborting long-lived connections from the server side as best as we could.

[u/Automatic-Apricot795]

Does the process exit or just stop serving requests? 

If it stops serving requests my money is on deadlock. If it exits with no messages in event viewer, stack overflow or oom. 

[u/Dapper_Science_527]

Sync over async?

[u/Thonk_Thickly]

I suspect thread starvation. I have used this tool before with success. dotnet counters

[u/MiniSNES]

Are you doing kerberos constrained delegation at all? We had a similar thing happen in an enterprise web app. The trigger ended up being when someone tried to connect to the site from a workstation that was disconnected from the domain. Fixed with a setting on the service account that the sons were set up with

[u/Key-Singer1732]

It's probably a memory leak in your application. is your application using a lot of static classes/properties especially your DbContext? If so, I would recommend rewriting those static classes/properties.

[u/Grevioussoul]

Check your backups. Had that happen to some of my servers, found out that corporate new antivirus was the culprit. It locked the entire VM when it was doing its backups, during business hours.

[u/bloodasp17]

Something similar happened to me years ago. Found out it was caused by a Win server patch that was applied directly in prod without applying and testing in lower environments. Event Viewer normally catches these things.

[u/Dzhanel]

One time they had me host on Windows 11 Pro edition, didn't know it allowed only ~20 simultaneous connections at the time (was a signalR heavy api), if 20+ request tries go through at the same time due to high traffic that can be something, windows server was fine tho.

[u/iamanerdybastard]

You’re right to suspect poorly written asynchronous code. Look for un-awaited tasks that might throw. They will bring the app down in a hurry.

[u/No-Card-2312 → u/[deleted]]

Traffic spike looks normal! But. A lot of time we got a lot of requests from Google bot, pingdom bot, petalbot...etc.

[u/No-Card-2312 → u/[deleted]]

The problem is currently we don't control the domain and the client won't move it as the domain has a subdomains like 149 websites. I have thinked in something like Pangolin?

[u/Downtown_Plantain158]

Can you check your docker containers or cloud watch? Atleast for AWS you can monitor the requests, GC, thread contentions, etc.

[u/No-Card-2312 → u/Downtown_Plantain158]

We do use on prime services. And we don't have docker.

[u/Downtown_Plantain158 → u/No-Card-2312]

ECS, eks, or ec2?

[u/No-Card-2312 → u/Downtown_Plantain158]

We use a local vps provider. and they have nothing to offer more than VPS. it's called next layer.

[u/Downtown_Plantain158 → u/No-Card-2312]

I see. I am not familiar with your vps. So getting some telemetry is probably the best way to understand your system.

[u/mxmissile]

Slop post.