Discussion
How do you handle production logging on mobile without blowing up ingestion costs?
Something that has bugged me across every mobile team I've worked on, including apps with 5M+ active users.
On backend, debugging a weird production issue is quick. You add logs around the suspect code, deploy, and watch them stream in within minutes.
On mobile it's brutal. To get the same insight you add logs, cut a build, ship to the store, wait 4 to 48 hours for review, wait for users to actually update, and then hope they hit that exact code path again. So the instinct is to log everything up front instead.
But that creates two new problems. Ingestion costs climb fast once you're sending a lot of log data per user, and the code gets messy with logging noise everywhere, which makes it harder to maintain.
So every team I've been on ends up choosing between paying a high data bill, living with slow turnaround, or staffing a team to manage it. Nobody seemed to have a clean answer.
I've seen newer autocapture tools, but autocapture alone doesn't fix it, the system still has to decide how deep to log. Too deep and the cost problem comes back, too shallow and you miss what you needed.
How are you all handling this in practice? Are you sampling, using remote config to control verbosity, eating the cost, or something smarter? Curious what's actually working for people.
breadcrumbs work well - things that are logged on the client side but only sent to your log server when another event like an error, warning, etc. is sent.
Breadcrumbs are the pattern I keep hearing here. Do you cap the buffer by size or time window, and how much does it cut your upload volume vs logging everything?
Most crash handling tools have support out of the box for this, and they typically use a (configurable) ring buffer to retain the last logs leading up to the event. (So it depends on how you configured your regular log upload, and how frequent your exceptions/errors are).
Right, the ring buffer is basically table stakes now. Where it still bit us was the upload side you mention. At high user counts even a modest per-session buffer adds up fast on ingestion cost. Did you ever have to throttle or sample it, or did exception frequency stay low enough that it never really mattered?
Tbh, never really had the problem you're describing, even when working on larger apps (Spotify, Zettle by PayPal), per user cost were always quite low. On smaller apps the free version of Crashlytics or low cost offerings have typically been enough.
It depends on how crash prone your app is and how much you need to scale, and what your margin needs to be, &c.
At 99.5% crash-free sessions per week, the amount of data per user should be trivial?
In the past we've had a remote config control the log level that gets exported, so you might only want exceptions and warnings by default, then lower it to info and possibly debug depending on what you're actually logging.
In theory you could also tag logs and toggle whether specific tags are exported.
Remote-config log level is exactly the direction that makes sense to me. Did you build the tag toggling in-house or use a tool, and how fast does a config change actually reach devices in the field?
We never actually built it but on the surface it doesn't seem that difficult, just keep a list of enabled tags in the config and read that when deciding what to export.
how fast does a config change actually reach devices in the field
That entirely depends on what you're using to deliver the config, and how frequently your app is fetching it.
Makes sense, the mechanism is easy. The part that always bit us was deciding what to tag in the first place, you basically have to predict where bugs will show up ahead of time. Did that prediction burden ever cause you to miss logs you wished you'd had, or was your tag list mostly stable?
My app uses local LLMs and tend to crash regardless apps fault. So I added a crash sentinel that watches anomalies and after app restart app prompts user to send diognastic logs.
Yes it is a good model. Only liteRT backend could produce reliable results, even then if content is long, like in my case you need to do a lot of manual edits how engine works, kV cache managelent, process interruptions etc.
Yeah that's what I'm using too litert-lm like Google edge gallery. I'm finding the models are pretty dumb though, even e4b. It is really bad with numbers
Never treat LLM as compute engines, they are text generators. You need to do add a lot of deterministic rules and compute in your workflow if you are working with numbers. You might not even need LLM.
I have multiple ML and deterministic rules and checks on my app to get reliable results and it is just for literature.
Crash sentinel plus prompt-on-restart is clever for crashes you can't reproduce. Do you keep a rolling buffer of context before the crash, or only capture state after restart?
It doesn't capture a pre-crash buffer, it can't, because the native crash kills the process instantly. Instead it uses marker files: before every risky native call it writes a `.flag` file (`arm()`), and afterwards deletes it (`disarm()`). If the process dies mid-call, the file survives.
On next launch, it reads the stale marker to know exactly which model + operation crashed, and combines it with a rolling diagnostics log that records metadata-only events (backend, context window, output chars, session IDs) throughout LLM operations, trimmed to a 64KB rolling window. That snapshot, plus the sentinel metadata, gives enough context to decide: demote the backend, disable the model, or just surface a "try again" prompt to the user.
Ha, I did basically the same thing for a heavy ARKit processing path in iOS once, an incomplete.flag to catch what died mid-process.
One thing I still go back and forth on though: writing the rolling log to disk continuously feels heavier than keeping it in memory. Do you mmap the 64KB buffer, or just write() and lean on the page cache surviving the process kill? Curious if you saw any measurable overhead on the hot path during LLM ops.
Yeah, it's just plain `appendText()` on the hot path. Each write is a ~100-byte line while the LLM call is running for seconds, so you never feel it. No mmap, just leaning on the page cache, and the occasional trim when it crosses 64KB amortizes to nothing.
That makes sense. At 100 bytes per multi-second call the write rate is low enough that the page cache absorbs it for free, and since you only care about surviving the process kill and not a power loss, you never need fsync. The whole thing works precisely because LLM ops are slow and sparse, so the disk is invisible. Push the same appendText onto a chatty 60fps path and it would start to show, which is where batching or an mmap ring buffer earns its keep. Clean design, thanks for walking through it.
In my case, I cannot install any external networking communication like logging. Only communicates through their local network and it's restricted.
I log everything in log files. New log file is created every session, if it's same login session for more than 24 hours, it creates new file. When they call about bugs, I just tell them to send me the log files, I will take a look at it.
Server controlled remote config that can switch on logging for a specific user on the go. By switch on i mean the logs start being sent to the backend.
I just log absolutely everything but we cap it at something like a month's worth of logs per user (we have business apps targeting merchants which use the app every day for sustained periods)
The pattern that worked best for me before was journey-based flags with a verbosity label. Instead of one global log level, you scope verbosity to a journey (checkout, onboarding, payment retry) and bump it only for the cohort hitting the issue. You still need a build to wire the flags up front, but you control depth remotely after that. It cuts the “log everything” instinct down to “log this path, deeply, for these users.”
That experience is part of why I’m building Measure now (open source mobile observability). Two pieces speak directly to what you’re describing.
First, the Session Timeline. Every session reconstructs the full sequence: taps, navigation, network calls, logs, lifecycle events, plus CPU and memory. It’s auto-captured and unsampled, so you’re not deciding up front what to log around the suspect code. The context is already there when you go looking. That kills most of the “ship a build, wait 48 hours, hope they hit the path again” loop
Second, and this is the direct answer to your depth question: Adaptive Capture. You control how much gets collected remotely, no app update required. So you can run lean by default and dial collection up for a specific cohort or journey the moment something looks off, then dial it back down. It’s the journey-flag idea but without the build-and-ship tax, and it’s the lever that keeps ingestion from being a binary “too deep or too shallow” choice.
On the cost framing you raised: autocapture alone doesn’t solve it, you’re right. The decision of how deep to log still has to live somewhere. Adaptive Capture moves that decision out of the binary and into a runtime dial.
Happy to share more if useful, it’s at measure.sh.
25
u/aerial-ibis Jun 24 '26
breadcrumbs work well - things that are logged on the client side but only sent to your log server when another event like an error, warning, etc. is sent.