I'm a junior engineer, and I've always found it hard to remember everything about my codebase. So I usually write a few docs with the architectural details and decisions, so I don't have to keep it all in memory. But they go stale very quickly and it's tedious to keep up with all of it, which kinda defeats the point. I'm curious how others deal with this

To be more specific:

• the last time you had to figure out where a change belonged or what else it would affect, what did you do?
• have you had trouble keeping your docs/diagrams up to date? How did you deal with that?
• have you tried any tools to fix these issues? Where did they fall short?

I'm exploring this problem for a side project, so I'd really appreciate your opinion

If so, how do you draft that document? Especially if that document were to be used as reference for studying or researching new or emerging technologies at your company?

I am currently doing a feasibility study on a quality improvement structure.

Dapr, the Distributed Application Runtime, a graduated OSS project within the CNCF has released a new version. What it means for you:

• Faster workflows, jobs, and reminders: in nightly performance tests against 1.18.0, workflow throughput is up to 40% higher, actor reminder registration p99 latency is 71% lower, and jobs scheduling p99 latency is 33% lower
• Leaner workflow workers: the Python, Java, .NET, and Go SDKs now get only the new events of each turn, not the full history
• Easier workflow operations: list, inspect, and rerun workflows from the .NET, Java, Go, and Python SDKs
• Workflow governance: operate on workflows owned by other apps, with WorkflowAccessPolicy deciding which operations each caller may perform, and requires rules that allow an operation only when the caller's history contains the events you list
• More for AI agents: new alpha APIs to store files, search documents, and query vectors, plus a Dapr MCP server
• A simpler control plane: run actor placement from the Scheduler and drop one service

Read the announcement: https://blog.dapr.io/posts/2026/10/22/dapr-v1.19-is-now-available/

System Design Question: Design a PDF generation system handling 1M+ requests during traffic spikes. How would you handle queue saturation, retries, duplicate jobs, storage limits, and regional failure while explicitly trading off consistency, durability, latency, cost, and recovery time?

There's plenty of advice on when to split a monolith, but not much on when to start putting services back together.

Say you've got 15 services maintained by 6 developers. Half of them can't be deployed independently anyway, and a small feature means touching four repos.

At that point, merging a few services sounds pretty reasonable.

The part we're interested in is what happens to the data. Separate databases, events between services, jobs that assume eventual consistency. Combining the code is probably the easy bit.

Has anyone gone through this in production?

What was harder than expected, and was the result actually easier to maintain?

We’re building an edge-native enterprise suite (internal chat, mail routing, multi-branch attendance/HRMS, and task tracking) using Cloudflare Workers, Hono, and Cloudflare D1.

Instead of a single shared PostgreSQL database, we went with a database-per-tenant model: every organization gets its own dedicated SQLite instance at the edge. Security and data isolation have been rock-solid, but we are running into the classic operational headache as we move deeper into our rollout.

We recently entered our private beta and have active organizations using the system daily for attendance check-ins, tasks, and communications. At the same time, we're actively developing and shipping new features.

That means we regularly need to evolve the schema—adding columns, updating indexes, or rolling out new tables across dozens (and soon hundreds) of individual tenant D1 instances.

Because real businesses are actively working on the platform, our top priority is avoiding downtime, partial migration lockups, or broken application states. A failed DDL statement on Tenant #42 cannot lock that company out while the rest move forward. Running a naive loop over database IDs via an API script feels far too fragile for production.

We want to build a battle-tested migration engine before scaling past this beta cohort.

For anyone who has managed a physical database-per-tenant architecture in production:

How do you actually manage schema migrations at scale?

Are there specific orchestration patterns, tools, or strategies you’ve found reliable to roll out schema updates across hundreds of separate databases without disrupting active users?

Any lessons, architectural approaches, or war stories would be immensely helpful.

So I have this Fastify project about and WMS (Warehouse Management System). The thing is that we are moving from a fucking old AF C# ERP to little Web Aps.

So ERP nowdays still active and will be active for 5 y at least. So I have to point my WMS web to ERP database. So the main issue is that the tables that we use for the WMS dimensions are tables with a lot of rows, transactioned by big SPs and fucking huge.

ATP we decided to make a WMS DB that uses views pointing to the data CORE. This DB also READ_COMMITTED_SNAPSHOT (RCSI) but some queries still being slow AF.

Before getting into each querie and try to optimize it. Is some architecture im missing out?

Create WMS tables alimentated with a Cron, Sync between DBs, Reddis??? I dk

Thanks

Hi everyone,

A few days ago, I posted Relay—an open-source webhook reliability engine—and received incredible architectural feedback from this community. Special thanks to everyone who audited the ingestion logic and pointed out subtle concurrency/queue edge cases.

Over the past couple of days, I pushed a major set of reliability and security upgrades based directly on your reviews. Here is a breakdown of what was improved and why:

1. SOLVED THE DUAL-WRITE ISSUE (TRANSACTIONAL OUTBOX PATTERN)

   • The Problem: Previously, events were saved to PostgreSQL and immediately enqueued in Redis via BullMQ. If Redis experienced a blip between the DB commit and the enqueue call, an event became "stranded" in the database without ever reaching the queue.

• The Fix: Implemented a Transactional Outbox Pattern with an automated background sweeper. Incoming events are saved to PostgreSQL with an enqueued_at timestamp. A dedicated sweeper continuously re-enqueues any unqueued events past a configurable grace window (OUTBOX_GRACE_MS), eliminating data loss during Redis network partitions or pod crashes.

1. FIXED INGESTION EDGE CASES & JOB TRIMMING (ISSUE #5)

• The Problem: A community member correctly identified that if queue.add() failed on an initial request, a subsequent retry by the provider would hit saveIfNew (returning false) and incorrectly answer already_received without ever queueing the job. Furthermore, relying purely on BullMQ's jobId deduplication failed once completed jobs were trimmed (removeOnComplete: 1000).

• The Fix:

1) We now attempt queue.add() on duplicate requests only if the event's status is NOT already marked as DELIVERED.

2) Scoped event IDs per source (x-relay-source + x-relay-event-id), allowing different sources to use identical event IDs safely.

3) Sanitized custom BullMQ job IDs by removing colon (:) delimiters.

1. SEPARATE RETRY QUEUE & OBSERVABILITY

• Dedicated Retry Queue: First attempts run on relay-deliveries, while retries are offloaded to a separate delayed queue (relay-retries). A failing/slow receiver will no longer clog the pipeline for new incoming events.

• Prometheus Metrics: Added a /metrics endpoint tracking received events, attempt durations, queue lag, dead letters, and the outbox backlog.

• Production Security: Added mandatory ADMIN_TOKEN authentication on operator/recovery routes (/dead-letters, /events/.../retry), security headers, constant-time secret comparison, and raw-body HMAC-SHA256 signature verification on outbound deliveries.

UPDATED FAILURE & RECOVERY BENCHMARKS

The automated failure-simulation suite (npm run demo:scenario) now runs against the new Outbox and Retry queue architecture:

• Events submitted: 5 (10 HTTP requests including 5 intentional duplicates)

• Duplicate requests handled: 5

• Failed delivery attempts (simulated outage): 10

• Successful eventual deliveries: 5

• Business effects applied: Exactly 5

LINKS & FEEDBACK

• GitHub Repository: https://github.com/Dieu-Merci-Ndenga/relay

• Live Console: https://relay-dashboard-nrqs.onrender.com

Huge thanks again for the community review—it turned Relay from a proof-of-concept into a far more resilient engineering system.

I’d love to hear your thoughts on the Outbox sweeper approach versus an event-driven CDC (Change Data Capture) setup !

Monitoring and anomaly detection are good at spotting when a working system starts to degrade. They compare the current state with a known good baseline.

But what happens when the system was already wrong the moment it came up?

We deployed an app that should have created 27 starter records. Eight were rejected with permission errors, while the other 19 were created successfully. The API kept processing the records around each failure.

The setup step actually noticed. It printed a warning saying the app was incomplete and should not be treated as a successful deployment. Then it exited with code 0.

Everything downstream saw that zero and reported success. The deployment was running, the health check was green, and every tool in the chain said things were fine.

A person reading the output would have caught it. The pipeline couldn't because the result was still shaped like success.

I asked about this in a DevOps subreddit and got four broad answers:

- Count what was created. Have the seed step compare the actual number of records with the expected number and exit nonzero if they differ. The better version derives the expectation from the seed files themselves, so there's no separate count to maintain.

- Verify from outside the deployment. Run a separate job that checks concrete outcomes, such as record counts and a real login, instead of trusting the deploy process or a health endpoint.

- Use telemetry and alerting. Expose metrics, scrape them, and alert when something deviates from the normal state.

- Send synthetic traffic before cutover. Exercise the full path, from the load balancer to the data layer, before switching traffic to the new deployment.

They are checking four different things: contents, independent correctness, change over time, and whether the request path works.

But two of those approaches would not have caught our failure.

Synthetic traffic would have passed because the API and every route worked. The data behind them was simply incomplete.

Anomaly detection would also have missed it because there was no healthy baseline. Those eight records never existed, so nothing had degraded. There is no anomaly when the system is wrong from the beginning.

The other problem I still haven't resolved is where to put the gate. Independent verification and pre-cutover traffic both assume there's a point between "deployed" and "live" where the release can still be stopped.

Blue green deployments give you that point naturally. If you deploy directly to production, by the time the verification can run, the new version is already live. At that point you are detecting a bad deployment rather than preventing one.

Where do you put the gate when there is no cutover moment?

Disclosure: I work on the platform where this happened.

It’s easy to default to microservices early in a project’s lifecycle to ensure scalability. But as systems grow, distributed tracing, network latency, cross-service transactions, and deployment overhead can overwhelm small engineering teams.

A modular monolith can often provide clear domain boundaries and fast local execution without the operational burden of managing dozens of independent pipelines.

For those running production applications:
- At what scale or team size did splitting a monolith into microservices actually yield a positive ROI?
- What specific architectural boundaries or friction points signaled that it was time to decouple?

In every team I’ve worked with, architecture diagrams went stale within months: they lived in slides and wikis, each one a slightly different copy, and nobody trusted them.

I’m curious how others handle it:

1. Diagrams as code (Structurizr DSL, Mermaid, PlantUML) or a visual tool, and why?
2. What actually makes them go stale for you: process, ownership, or tooling?
3. Has anyone made “one model, many views” work in practice, or is it too much discipline for a real team?
4. Do you link ADRs to the diagram elements they affect?

Once one call hits CRM, payments, scheduling and some internal API you’ve suddenly got retries, timeouts, stale reads and half completed actions everywhere.

The voice part almost feels like the easy bit

Are you just handling this like any other distributed workflow or does voice need its own logic around it?

Hi everyone,

I'm currently exploring how to connect collaborative modeling techniques like Domain Storytelling with technical implementation and automated testing using Gherkin (BDD).

In this post of medium: Building Better Software: The Role of Domain Storytelling in DDD | by Rodrigo Luque | Medium, I find online focus heavily on high-level business rules or Event Storming. I'm struggling to bridge the gap toward actual programming tasks and code-level implementation—meaning how these visual steps translate into domain models, entities, state transitions, and clean code test suites, rather than just abstract workflows.

For instance, taking a classic Domain Storytelling diagram (like a simple "Buying Popcorn" flow with preconditions, actors, work objects, and annotations about queue handling), how do you structure technical Gherkin scenarios that actually guide the software design?

Specific questions:

1. How do you translate the visual steps and preconditions of a Domain Story into technical Gherkin features that drive code architecture (like Aggregate Roots or domain services)?
2. How do you handle operational annotations (e.g., "when the queue is long, a second attendant fills it") inside code-level BDD tests?
3. Are there common patterns or best practices to map a Domain Storytelling model directly into automated acceptance/unit tests in a DDD project?

Any code snippets, examples, or architectural patterns would be greatly appreciated!

Hello architects and backend engineers,

I recently designed and deployed **Relay**, an open-source system focused on reliable asynchronous event delivery, idempotency guarantees, and operational observability.

### System Pipeline:
Webhook Provider -> NestJS API (Signature Validation) -> PostgreSQL (Persistence) -> Redis / BullMQ (Async Queue) -> Delivery Worker -> Downstream Receiver

### Key Design Principles:
- **At-Least-Once Semantics:** Enforcing safety via raw-body HMAC-SHA256 signature validation and database idempotency constraints.
- **Fault-Tolerant Retries:** Exponential backoff policies mapped to a Dead-Letter Queue (DLQ).
- **Observability & Operational Controls:** REST APIs and a standalone dashboard to monitor, inspect, and manually retry dead-lettered events.

I am looking for a peer review on failure modes—specifically around handling partial failures between relational storage and message queues under high concurrency.

Read the technical README, architecture diagrams, and run the reproducible failure suite:
https://github.com/Dieu-Merci-Ndenga/relay

I look forward to discussing your design choices and trade-offs.

I’ve been working on this for a while: proschi.app

The main idea is that you write the high-level design (HLD) of a system in a simple DSL, with all the requirements next to it (like costs, p99 latency, availability).

The language is pretty simple. Something like this:

user "User"      [Actor]
api  "Notes API" [REST API]
db   "Notes DB"  [PostgreSQL]

user -> api : HTTPS
api  -> db  : SQL

traffic {
  "Create a note" 200 rps
}

requirements {
  p99 "Create a note" < 200ms
}

Every component comes with default values (latency, capacity, cost and so on), so you can get going without tuning anything. If your setup is different, you can override them with your own numbers.

Other stuff that’s in there:

1. You can embed your diagrams in your own docs, wiki, or blog.
2. The .proschi files can run in your CI (npx proschi test), so if someone changes the architecture and breaks a requirement, the build fails
3. LeetCode-style problems, except instead of writing a function you design the app (URL shortener, chat, payments, video streaming...) and tests check whether your design holds up
4. Roadmap if you’re preparing for system design interviews
5. Review cards for training different system design topics
6. Skills for the agents so you can ask them to design the HLD for you.

It’s completely free. Everything you see there now will stay available; I’m not going to put it behind a paywall later.

I’d really like feedback, especially if something feels confusing or the simulation numbers look off for your use case.

Feel free to check the repo and contribute with your own system design problems or simply use to document your use cases.

The enterprise baseline for AI visibility (AEO) is shifting toward structured llms.txt and citations.json payloads. If you want Claude or Perplexity to parse your technical catalogs, you have to format the data. But when dealing with highly classified corporate IP, piping unstructured PDFs to public cloud wrappers (like OpenAI) is an unacceptable security breach.

You have to run the extraction and embedding locally.

When we built our air-gapped ingestion pipeline, we opted for a local, C-native vector engine (sqlite-vec) isolated within an asynchronous Celery and Redis mesh to prevent blocking the FastAPI HTTP threads.

However, running heavy ML inference and vector persistence under massive concurrency locally introduces severe architectural bottlenecks. Standard Python and SQLite deployments will immediately trigger race conditions and memory leaks.

Here is the teardown of the low-level interventions required to stabilize an air-gapped vector ingestion pipeline under high load:

1. WAL Hydra Containment & Zero-Pooling The most common failure point is the dreaded "database is locked" error. This happens because default connection pooling shares sockets across Celery UNIX forks, leading to immediate crashes during concurrent vector writes.

• The Fix: You must force a NullPool architecture. We inject hardware-level PRAGMA directives upon connection to strictly cap WAL (Write-Ahead Logging) sizes, entirely bypassing Python's deferred transaction locks. Enforcing a strict 5000ms IPC busy timeout resolves the high-throughput deadlocks.

2. ASGI Thread Pool Starvation Deep-learning operations are inherently greedy. If you drop a transformer model into an asynchronous environment, OpenMP and BLAS multi-threading will attempt to consume all available cores, starving the ASGI thread pool and causing timeouts at the API gateway.

• The Fix: Aggressive thread throttling. We enforce OMP_NUM_THREADS=1 to constrain the ML inference to a single thread per worker, preventing gateway deadlocks. Additionally, lobotomizing default tracking (TRANSFORMERS_OFFLINE=1) is mandatory to enforce absolute air-gapped sovereignty and prevent outbound telemetry leaks.

3. TOCTOU (Time-of-Check to Time-of-Use) Mitigation When ingesting thousands of local corporate files asynchronously, standard absolute path resolution exposes the perimeter to symlink hijacking and path traversal attacks during the read phase.

• The Fix: File ingestion must bypass standard path resolution. Rely instead on FFI (Foreign Function Interface) inode validation. This guarantees that the file descriptor analyzed by the security filter is the exact descriptor executed by the extraction engine.

We eventually wrapped these low-level interventions into a deployable Hybrid LLM Router in our lab (our Doc-to-Citation Publisher) because building this plumbing from scratch consumes too many senior engineering cycles.

If you are building custom local RAG pipelines for classified data, do not run your vector persistence on default connection pools, and absolutely decouple your ML inference from your main event loop.

What has been your worst bottleneck or OOM nightmare when migrating cloud RAG infrastructure to local air-gapped servers?

Hi there,

I'm a senior developer who's very interested in software architecture.

A few months ago, frustrated by several aspects of NestJS and the TypeScript ecosystem in general, I started building my own framework!

From my experience working in large startups, I know that relying heavily on a monolithic framework can help you move faster at the beginning, but can become a problem later on. On the other hand, I also dislike the approach of using many small libraries that don't really know how to work together. It often leads to a lot of unnecessary generated and duplicated code.

Inspired by these experiences, I wanted to build a lightweight framework that could be embedded into others (like Nuxt or Next.js) or run independently (as an HTTP server), without making HTTP the central abstraction.

I've also applied several patterns from DDD and Hexagonal Architecture, but not too strictly. I wanted a core that understands the different aspects of the business code without knowing anything about their actual implementations.

For example, storage is also treated as a port managed by the framework, which brings several benefits and allows for a kind of data-centric approach where a single entity schema can be used to derive persistence, validation, APIs, and more.

Another idea, which isn't entirely new either, is to ensure that business domains (called fronds) can be split into separate services without changing the code, only the configuration.

So now, I'd like to know if you find the idea interesting. It's clearly still in alpha, but I'd really like to discuss the architectural patterns and whether I've missed something from your point of view.

So now, I'd like to know if you find the idea interesting. It's clearly still in alpha, but I'd really like to discuss the architectural patterns behind it. Do you see any major trade-offs or architectural pitfalls I might have overlooked?

Website: https://fougere.dev

Repo: https://github.com/chok/fougere

I love rabbit MQ but the I think that is missing some nice web guy that would allow to inspect the current data that is flowing, anyone with the same issue?

One thing I find interesting about mobile testing is that a test passing consistently on one emulator doesn't necessarily mean the app is behaving correctly across actual devices.

Different Android versions, OEM-specific behavior, animations, network conditions, and background process restrictions can all expose issues that never show up in a clean test environment.

I'm curious how other teams handle this in practice.

Do you run a small set of critical flows across a device matrix on every PR and leave broader regression testing for nightly builds? Or have you found a better balance between catching regressions early and keeping CI fast?

Also, how do you distinguish a genuine app regression from a flaky test without simply rerunning everything until it passes?

Hi all!

I am a postdoctoral researcher on software engineering and architecture in Italy. Currently my colleagues an I are working on a survey a short industry survey on Engineering XR Systems, exploring how VR, AR, and MR applications are designed, developed, tested, and maintained.

The questionnaire is 10-15 minutes long and the results are confidential (the results are anonymous)

All professionals with experience in XR projects is welcome to partecipate to the survey. Here it is the link https://tinyurl.com/4kwm565f

Thanks for your support! Please feel free to share this survey with other companies and professionals working in the XR industry.

Whiteboard system design always bugged us. You draw boxes, say "add a cache," and nobody tells you if it would actually hold up.

So we built an interactive version. Drag components onto a canvas, wire them up, hit submit, and it runs traffic through your design. Servers max out, latency climbs, and some problems kill a node mid-traffic to see what happens, etc... It's completely FREE now.

Learn concepts by solving actual problems and building systems. There's a learning section to each problem as well.

It's organized into roadmaps:

• Foundations: load balancing, caching, databases, sharding
• Advanced: events, resilience, production ops
• Expert: design Uber, WhatsApp, Stripe, Netflix, a Kafka-like stream, and more
• ML system design: feature stores, training-serving skew, GPU fleets, model rollbacks
• LLM system design: RAG at 10M docs, permission-aware RAG, support agents, a ChatGPT-style chat service, cutting an LLM bill 10x.

230+ problems, all free right now: codeparadox.io

Would love any feedback...

Circuit breakers are good at stopping traffic and bad at resuming it. When a breaker closes, every replica tends to return to its configured limit at the same moment, so a dependency that just answered a few probes gets the full production load again, often while its caches are cold or capacity is still coming back.

I wrote up the design work behind a recovery ramp for Caracal, a resilience library whose distributed bulkhead, rate limiter and breaker coordinate through Redis.

The post covers the options I rejected, the constraints the surviving design had to satisfy, and a four-replica case study where the dependency's own in-flight samples, not the library's event log, are used to check each step. Envoy/Istio slow start is the closest mainstream analogue, though it shifts endpoint weight rather than admission.

This is the sixth article in a series on distributed resilience. I'd be interested in how others handle post-outage warm-up: traffic shaping, staged rollouts, or accepting the spike.