Alerting: from a report to a notification

There is no rules engine and no message queue in this path. It's worth being direct about that up front, because "alerting" suggests more machinery than exists. What follows is the real path, and where it falls short.

Two triggers, not one

1. Something happens right now — an incident or hazard gets assigned to someone, a corrective action is created, an interview is scheduled. The handler that processes the request does its main write, then fires off a notification in a separate goroutine so the HTTP response doesn't wait on an email round-trip.

2. Time passes — a background job runs once every 24 hours and scans for things nobody has acted on: incidents still open with an assignee, corrective actions past their due date, and actions due within 48 hours. Each one it finds gets a notification, same as trigger 1.

Neither trigger involves "detection rules" in the sense of configurable thresholds or a rules DSL. Trigger 1 is a handful of hardcoded call sites, one per event type. Trigger 2 is three hardcoded SQL conditions in one function.

The path for an immediate event

sequenceDiagram
    participant Client
    participant Handler
    participant Service as Domain Service
    participant DB as PostgreSQL
    participant Goroutine as go func() (detached)
    participant NotifSvc as Notification Service
    participant Mail as SMTP

    Client->>Handler: POST /incidents/:id/assign
    Handler->>Service: AssignIncidentToUser(...)
    Service->>DB: UPDATE incidents SET assigned_to = ...
    DB-->>Service: ok
    Service-->>Handler: updated incident
    Handler->>Client: 200 OK (response sent — does not wait below)

    Handler->>Goroutine: go NotifyIncidentAssignment(incident, assignee)
    Note over Handler,Goroutine: fire-and-forget: no error returned to the caller,<br/>no retry, no record that this even started
    Goroutine->>NotifSvc: SendNotification(...)
    NotifSvc->>DB: INSERT INTO notifications (...)
    NotifSvc->>Mail: send templated email
    alt SMTP succeeds
        Mail-->>NotifSvc: 250 OK
    else SMTP fails or times out
        Mail-->>NotifSvc: error
        NotifSvc->>NotifSvc: log the error, return nil anyway
        Note over NotifSvc: the notification row already exists;<br/>nothing marks the email as failed
    end

The notification row is always written first — that part is durable. It's the email send, and the entire goroutine itself, that has no safety net: if the process restarts between the go func() call and the SMTP round-trip completing, that notification simply never gets emailed, and there is no record that it was supposed to be.

The path for the daily reminder scan

sequenceDiagram
    participant Ticker as 24h ticker (goroutine, started at boot)
    participant NotifSvc as Notification Service
    participant DB as PostgreSQL
    participant Mail as SMTP

    loop every 24 hours
        Ticker->>NotifSvc: CheckAndSendReminders()
        NotifSvc->>DB: SELECT unresolved incidents with an assignee
        NotifSvc->>DB: SELECT overdue corrective actions
        NotifSvc->>DB: SELECT actions due within 48h
        loop for each row found
            NotifSvc->>DB: INSERT INTO notifications
            NotifSvc->>Mail: send email
        end
    end

This job also tests the SMTP connection once at startup and logs if that fails, but that check isn't wired to anything — a broken SMTP connection at boot doesn't stop the ticker from trying anyway.

Where the queue actually is (and isn't)

There is exactly one real, bounded, worker-backed queue in the system: the request-logging middleware that feeds the analytics rollup. It's a fixed-size channel with a small pool of worker goroutines, and it drops a new entry outright if the queue is full — a deliberate choice, because the job it's protecting is analytics, and losing an occasional API-usage log line is an acceptable cost for never blocking or slowing down a real request.

That queue has nothing to do with alerting. Notifications don't go through it, don't have a queue, and don't have a drop policy — they either send inline in a detached goroutine, or they don't send at all with no trace of the failure.

What this costs, concretely

• No retry. An SMTP timeout is caught, logged, and discarded. The notification row exists; the email doesn't go out; nothing revisits it.
• No durability across a restart. The notification-send goroutine holds no state anywhere except in memory between the DB write and the SMTP call. A deploy or a crash in that window loses the send silently.
• No backpressure or rate limiting. A single action that fans out to many recipients (for example, closing an incident that many people are watching) opens one goroutine and one SMTP connection per recipient, unthrottled.
• No visibility. There's no "failed notifications" view, no metric, no alert-on-alerting-failure. The only trace of an SMTP failure is a log line.

See decisions.md for the fire-and-forget notification design as a decision that didn't hold up, and architecture.md for what a durable-outbox replacement would look like.