Removing Bottlenecks: Returning Fast

This is not an SMTP problem. It shows up most often as "registration is slow" or "p99 spiked when SendGrid had an incident", but the underlying mistake — making the caller wait for work the caller doesn't need — is the same shape behind webhook fan-out, inline thumbnailing, audit-log writes inside requests, and synchronous notification fan-outs. Once you see it once, you see it everywhere.

A user clicks Sign Up. The spinner spins. Four seconds later: 500 — server error. They refresh, try again, give up. Your dashboard shows p99 latency on /register: 4.1s, error rate: 8%. The first instinct is "the database is slow" — but it isn't; the user INSERT took 30 ms. The handler is waiting on SendGrid. When SendGrid is healthy, your signup feels fine. When SendGrid has a bad five minutes, your signup has a bad five minutes — and your latency budget is now coupled to a vendor's status page.

Nothing in your code looks wrong. The handler is straightforward:

app.post("/register", async (req, res) => {
  const user = await db.users.insert({ email, hashedPassword })  // 30 ms
  await emailService.sendWelcome(user)                            // 800 ms — sometimes 4 s, sometimes timeout
  res.status(201).json({ user })
})

That handler is doing one thing wrong, but it's a structural thing. The user clicked Sign Up to be signed up — not to have a welcome email confirmed delivered. The endpoint is fulfilling two contracts at once: the one the caller asked for (create the account) and one the caller doesn't care about (notify them by email). The caller is waiting on both, and the second contract's failure modes are bleeding into the first.

Watch the latency

Four patterns, one client, one server. Toggle below to see how the same business outcome — user created, welcome email sent — produces wildly different caller experiences depending on where the work runs.

In Inline (sync), the handler awaits every step; the response lands at 850 ms. The work is durable in the trivial sense — the pod kept running through it — but the user paid for it. In Inline + retry, on a day SMTP is flapping the email step retries inside the request and the spinner stretches to ~3 s. Your endpoint p99 traces the vendor's status page, not your own RPS. In Fire-and-forget, the handler hands the email send to a goroutine (or a Promise nobody awaits) and returns at 51 ms. Fast — but if the pod restarts before the SMTP ack, the welcome email is gone with no record it should have been sent. In Durable queue, the handler enqueues the job to Redis (or Postgres-as-queue, or SQS) and a separate worker picks it up moments later. Caller still returns at ~50 ms; if the worker crashes mid-send, the job stays in the queue and is retried.

The lesson lives in the second metric the visualization tracks — survives a crash? The first two modes are slow but durable. The third is fast but loses work on restart. Only the fourth gives you both — fast caller and durable side effect — and the cost is one new piece of infrastructure.

Why the request path is the wrong place for slow work

The cost has three parts and they all bite at the same time.

• Latency coupling. Your endpoint's p99 is now your_work + downstream_p99 + downstream_jitter. Every retry on the email service shows up on the user's spinner. You don't control the variance; you've inherited it.
• Connection-holding. A handler await-ing for 4 seconds holds an HTTP connection, a worker process, sometimes a DB row lock, all for work that has nothing to do with the response body. Concurrent registrations fall over the moment SMTP gets slow — and the failure cascades to the rest of your API because the workers serving /register are also serving everything else.
• Failure-surface mismatch. "Welcome email failed to send" should not be a 500 on a registration that already succeeded. But if you await the send and it throws, that's exactly what happens. You either swallow the error (and lose the email silently) or propagate it (and tell the user their account creation failed when it didn't). Both are wrong; the framing is wrong.

The framing fix is simple: the response is for confirming what the caller asked you to do. Anything triggered by that action but not part of the answer goes on a different timeline.

Same shape, different surface

Anywhere your handler is await-ing on work whose result isn't in the response body, the same anti-pattern is hiding. A short tour:

• Webhook fan-out on save. A "save post" handler that POSTs to three integration endpoints inline. Each integration's downtime is now your save endpoint's downtime — and a slow integration penalizes a customer who doesn't even use it.
• Search index update on write. INSERT user then es.index(user) in the same handler. ElasticSearch GC pause = signup pause. The search team's incident becomes the signup team's incident.
• Image processing on upload. Resize / thumbnail / transcode inside the upload handler. A 200 MB video uploaded by one user makes everyone else's /upload slow even when their file is tiny — because the worker pool serving uploads is busy in ffmpeg.
• Audit logs on every action. Synchronous write to a logging service that's "usually fast." The day it isn't, every action in the app is.
• Cache warming inside requests. "While you're here, let me also recompute these five aggregates." The user paid for somebody else's eventual fast response.
• Notification fan-out (push, SMS, in-app, email) inside the originating handler. One "post a comment" hits four downstream services; the comment endpoint p99 is the worst of the four.

The substrate changes — SMTP, ElasticSearch, ffmpeg, syslog, push providers — and the verb changes: send, index, transcode, log, notify. The diagnosis doesn't: a request handler is waiting on work whose result the caller doesn't need to see.

Three ways to fix it

Each of the three fixes solves a problem the previous one left behind. Pick the lightest one whose failure mode you can live with.

1. In-process async (go func() / setImmediate / Promise.resolve().then(...))

The caller returns fast; the work happens in a background goroutine or on the next event-loop tick. One keyword change:

app.post("/register", async (req, res) => {
  const user = await db.users.insert({ email, hashedPassword })
  res.status(201).json({ user })

  // fire-and-forget — caller doesn't wait for this
  emailService.sendWelcome(user).catch(err =>
    log.error("welcome email failed", err)
  )
})

Latency win: real. Durability: gone. If the pod crashes — or gets SIGTERM-ed during a deploy — between res.send and the SMTP ack, the email is never sent and there's no record that it should have been. There's also no retry, no visibility (no queue depth, no failed-job count), and no backpressure — goroutines pile up faster than they drain and OOM the process with no observable signal until the heap explodes.

Use this only for genuinely-disposable work — best-effort cache priming, optional metric emissions, idempotent analytics events with a separate replay mechanism. Welcome emails fail this test. So do password resets, payment confirmations, and basically anything a user might later ask "where's my X?" about.

2. Durable job queue (Sidekiq / River / BullMQ / SQS + worker)

The handler pushes the job onto a durable queue — Redis with persistence, Postgres + advisory locks à la River, or a managed broker like SQS — and a separate worker process pulls and executes:

app.post("/register", async (req, res) => {
  const user = await db.users.insert({ email, hashedPassword })
  await welcomeEmailQueue.add("send-welcome", { userId: user.id })  // ~2 ms — Redis RTT
  res.status(201).json({ user })
})

// elsewhere, in the worker process:
welcomeEmailWorker.process("send-welcome", async ({ userId }) => {
  const user = await db.users.findById(userId)
  await emailService.sendWelcome(user)
})

Caller pays one cheap Redis round-trip instead of an SMTP call. Email-service downtime no longer surfaces as user-facing 500s — it surfaces as queue depth, which the operator can monitor and alert on. Retries with exponential backoff and dead-letter destinations come for free with every mature queue. Worker pool size becomes a knob — cap concurrent SMTP connections at four regardless of incoming registration RPS, so a viral signup link can't DDoS your downstream provider.

The cost is operational: a new system in your stack to provision, monitor, upgrade, and page someone for. There's also a subtle correctness gap — db.users.insert and welcomeEmailQueue.add are two separate writes to two separate systems. If the DB commit succeeds but the Redis enqueue fails (network blip, Redis OOM), you've got a user with no welcome-email job, and you'll discover it later via support tickets. Most teams ship with a try/catch and "we'll backfill manually" — fine until the day it isn't.

Default to this fix. It's the right answer for almost every "this needs to happen reliably but the caller shouldn't wait" case.

3. Transactional outbox

Solves the dual-write gap fix #2 leaves behind. The trick: write the job into the same database transaction as the business write, then have a separate process relay outbox rows to the real queue.

await db.transaction(async tx => {
  const user = await tx.users.insert({ email, hashedPassword })
  await tx.outbox.insert({
    aggregate: "user",
    type: "send-welcome-email",
    payload: { userId: user.id },
  })
})
res.status(201).json({ user })

// separate process: poll the outbox table, ship rows to the queue, mark them dispatched.
// CDC (Debezium etc.) does the same thing without polling.

Both rows commit or neither does. A separate relay process reads outbox and forwards entries to the real queue, marking them dispatched. Now the invariant is: the welcome-email job exists if and only if the user row exists. No more "user created but the side effect was lost."

The cost is the highest of the three: an outbox table, a relay process (poller or CDC consumer), idempotency end-to-end (the relay may re-deliver after crashes), and one extra row written on every business transaction. Time-from-write-to-side-effect grows too — a polling relay running every 1 s adds ~500 ms of jitter before the queue even sees the job.

Reach for this only when "the row exists but the side effect was lost" is a bug a user would file — billing events, password-reset emails, GDPR-relevant audit logs, financial reconciliation, anything regulated. For welcome emails, fix #2 is fine; for the email that says "here is your new password", you want fix #3.

Three signals you have one

1. Endpoint p99 correlates with a downstream's health, not with your own load. A login spike doesn't move it; a SendGrid incident moves it. Your latency dashboard is reading from someone else's status page.
2. An async side-effect failure surfaces as a 5xx on a request whose primary action succeeded. "The user was created but /register returned 500" — the row is in the DB, the email send threw, the throw propagated up the handler.
3. Latency budgets that don't add up. You measured every operation in the handler, summed them, and got a number much smaller than your actual p99. The variance is hiding in the slowest dependency, not in averages.

Rule of Thumb

Your response is for confirming what the caller asked you to do. Anything else triggered by that action — emails, indexing, fan-out, thumbnails, audit logs, cache warming — belongs on a different timeline. Push it to a durable queue and step out of the way; if the side effect must track the business write atomically, write it through an outbox.

The practical heuristic — the one-second test: ask whether the caller would be happy if this work happened one second from now. One minute. One hour. If yes, it doesn't belong in the response timeline. If no, it belongs in the response.

Don't pick the heaviest fix on principle. Pick the lightest one whose failure mode you can live with. Fix #1 if losing the work on a crash is honestly fine. Fix #2 for almost everything else. Fix #3 only when "the row exists but the side effect was lost" is a bug a user would file.

The pattern generalizes. Anywhere a request handler is await-ing on work whose result isn't in the response body, the caller is paying for latency they don't need. Move the work; keep the response.

That's the fourth shape in Removing Bottlenecks: round-trip ceremony (N+1), worker-pool skew (hot partition), control-plane data work (large file upload), and request-path slow work (returning fast). Four flavors of the same family of bug — the shape of how things move through your system, not the speed of any one thing.