The Event Bus That Keeps Work Alive
DeskcommCRM’s event architecture answers a specific operational problem: once a WhatsApp message has become a durable demand, AI, automation, analytics, media processing, and compliance may all need to react, but none of them should be allowed to make the original request carry every side effect. This chapter explains the relationship between event_log and the durable job queue, then uses crashes, retries, and permanent failures to show how the system keeps a demand moving without pretending that delivery is automatically reliable. The reader should leave able to predict who owns each piece of work and what evidence remains when a worker fails.
The distinction begins with two different questions. An event asks, “What happened?” A job asks, “What work must one worker now perform?” Confusing those questions creates either a brittle request path or an untraceable pile of tasks. DeskcommCRM stores the fact first, then lets independent consumers decide which work follows.
Facts before effects
When a message arrives, the ingest layer can record a domain event such as a received message, a created lead, a changed stage, or a blocked contact. The event_log row is organization-scoped and carries a constrained event name, the relevant entity, a payload, metadata, attempt state, and the time at which another attempt may occur. It is a durable description of something that has already happened. It is not itself the WhatsApp call, the model invocation, or the outbound webhook.
That separation protects the transaction that records the customer’s message. A database trigger can insert a line into event_log in the same local transaction as the state change. It should not call an external service from inside that transaction. An HTTP request can be slow, fail after the database has locked records, or return an ambiguous result. If a trigger makes that call, the system has coupled the integrity of its local record to the availability and behavior of a third party. Under load, a simple insert becomes a distributed operation with no clean recovery boundary.
Instead, the event is a handoff point. The current registry includes consumers for AI response and sentiment, RAG indexing, LGPD export and redaction, automation rules, follow-up reactivity, media persistence and derivation, push notifications, and other domain work. Several consumers may observe the same event, and they do not all have the same urgency or failure policy. An AI response may need a durable turn. A sentiment calculation can fail without blocking the reply. A redaction failure is a compliance issue. An external conversion report can be retried independently of the customer-facing timeline.
The consumed_by marker makes this fan-out explicit. A consumer records its own stable key, so “the event was seen by the automation handler” does not imply “the RAG indexer also processed it.” The dispatcher selects handlers whose event types match and whose keys have not already been recorded. This is not pure event sourcing: the CRM keeps current tables such as conversations and leads, while the event log supplies a durable bridge for asynchronous consequences. That lighter model fits a product that needs reliable side effects without turning every query into a reconstruction of the entire past.
The agent path adds another layer of separation. The canonical agent-engine drain claims pending ai_agent.dispatch_requested events and converts them into inbound_turn jobs in the same Postgres-backed harness. A long-running worker then claims those jobs and runs the AI turn. The event says that an agent may need to respond; the job owns the attempt to process that turn. Keeping the two layers distinct lets other consumers react to the same inbound without racing to impersonate the agent.
Consider what happens when one customer sends a message about a delayed order. The ingest operation can emit a single fact: a message was received in a particular organization, conversation, and channel. The follow-up consumer may use it to wake or cancel a waiting enrollment. The AI consumer may turn it into an inbound job. A sentiment consumer may classify frustration. An automation rule may add a tag or notify another system. None of these consumers needs to be called from the webhook itself, and none should infer that another consumer has finished merely because the original request returned 200. Their independent consumer markers let each path answer a different question: was this fact processed for my responsibility, and if not, when should I try again?
This also explains why Realtime is not the internal bus for critical work. A live subscription is excellent for showing a new message in the inbox or patching a Kanban card. It can reconnect, refresh, and update a screen. But a UI subscription does not provide a durable acknowledgment or a reliable retry contract when a browser is closed or a network drops. A customer-facing send, an LGPD redaction, or a knowledge reindex cannot depend on a screen being connected at the right moment. DeskcommCRM uses live updates for observation and durable events/jobs for consequences. The distinction is not that one technology is modern and the other is old; it is that observation can tolerate a refresh while an irreversible side effect needs ownership and recovery.
The “outside the original request” label is the important boundary. The customer’s message is not held hostage while every downstream service responds. At the same time, the system does not erase the work merely because the request has returned. The event and the job provide different kinds of continuity: one preserves the fact, the other preserves responsibility for the next action.
A queue that admits failure
The durable queue is organized around a contact lane. DeskcommCRM allows work for different contacts to proceed in parallel, but it avoids running two conversational jobs for the same contact at once. That is a semantic constraint, not just a throughput setting. If two inbound turns for one person are processed concurrently, both can read the same prior state, generate inconsistent replies, or send messages in the wrong order. Serializing the lane gives the system a coherent sequence while retaining useful concurrency across the customer base.
The claim operation has several defenses. It first avoids selecting multiple pending jobs from the same lane in one batch. It then locks eligible rows with FOR UPDATE SKIP LOCKED, which lets competing workers skip work another worker has already claimed instead of waiting on every row. The claimed job changes to running, receives a worker identity and lock time, and increments its attempt count. A partial unique index provides a final belt: at most one running job may exist for a contact. If two workers still collide in the small interval between selection and update, the uniqueness violation costs one claim attempt, not the integrity of the whole queue.
A global concurrency cap is handled in the database as well. Before claiming, the worker counts running jobs while holding a short advisory lock, calculates available capacity, and claims no more than that amount. The lock covers the reservation decision, not the model call or network request. This distinction matters: serializing all processing would make the queue safe by making it needlessly slow; serializing only the capacity calculation preserves parallel work while preventing several workers from collectively exceeding the configured limit.
A second form of idempotency connects an event to a job. When an at-least-once delivery tries to enqueue the same source event again, the unique organization/source-event constraint returns the existing job. Replaying the event therefore does not create another turn. The same principle appears at the send boundary. The send ledger records the logical sequence number of an attempted message and its body hash without storing the body as a log secret. If the worker dies after the channel accepts a message but before the database records the result, the next attempt can recognize the accepted operation and avoid sending the same customer message again.
This is why “exactly once” needs careful wording. The system cannot force an external provider to deliver exactly once merely by naming a database constraint. What it can do is make the local intention idempotent and make the effect recognizable. A duplicated webhook can become a no-op. A duplicated job can return the existing row. A repeated send can consult the ledger or an external idempotency key. The guarantee is strongest at the boundary where the system controls identity; it becomes a recovery strategy, not a metaphysical promise, once a third party is involved.
Leases handle the opposite failure: the worker disappears while holding a job. A lock timestamp and visibility timeout allow a reaper to return abandoned work to pending, or move it to dead when attempts are exhausted. Completion is guarded by the worker identity and the claim timestamp. If the job has been reclaimed by another worker, the old worker cannot commit its late result. A completion transaction can also write the state change and enqueue the next event together. If that transaction fails, neither half is presented as complete. This avoids a particularly damaging split: a worker might claim that a stage changed while the event that should notify the next subsystem was lost.
A useful failure scenario makes the boundary clear. A worker claims the job for the delayed-order message, calls the channel or an AI provider, and then the process is killed before it updates the queue. The database still has a running job, but the lease eventually expires. The reaper can make the work claimable again. On the retry, the worker does not blindly repeat every effect: it checks the durable state and the send ledger, where an accepted message or an existing CRM message can be recognized. If the earlier attempt never reached the external boundary, the logical send can proceed. If it did reach that boundary, the retry records or reuses the outcome instead of creating a second customer-facing message. The system is not eliminating the crash; it is making the crash a state with a defined recovery path.
The same logic protects the transition from one module to the next. Suppose a lead-stage update succeeds but the process dies before the notification or automation event is acknowledged. An at-least-once event may be delivered again. The consumer must therefore make its own effect idempotent, often through a unique key or a state check. A replay that finds the stage already recorded should not create a second lead activity merely because it was invoked twice. This is why the event log stores both a durable fact and consumer-specific progress: the fact may be replayed, while each consumer decides whether the desired effect has already happened.
Retry is appropriate only when another attempt could change the result. Network errors, timeouts, and temporary upstream failures fit that category. The queue applies exponential backoff rather than retrying in a tight loop. A provider that is already rate-limiting the installation should not receive five immediate copies of the same request. Backoff gives the upstream room to recover and gives operators a more interpretable timeline.
A permanent business decision is different. If a customer has opted out, sending again is not a transient failure. If the system has no valid target or a payload violates an immutable contract, repeating the same operation will not repair it. The agent engine distinguishes such a veto from an infrastructure incident: a blocked send can cancel the job without producing a stream of critical dead-letter alerts, while a worker or provider failure can retry and eventually create a visible job_dead item. The status describes what a person should do next, not merely what a function happened to throw.
The same principle governs automation. Before executing a rule that includes a WhatsApp action, the engine can discover that the channel is outside its sending window or that a protection requires postponement. It records an “adiado” outcome and retries at the permitted time instead of running earlier actions and leaving the rule in a partially changed state. After execution, skipped and failed actions prevent the aggregate result from being called success simply because no exception escaped. A message that was never attempted is not a green delivery.
Visible failure is the final part of the design. A normalized error in a worker log may help a developer, but it is not an operating surface for a business owner running a VPS. The queue therefore turns exhausted jobs into Central items with a kind, severity, reference, and a bounded reason. Follow-up flows use the same idea: a flow that becomes dead creates a visible notice before its terminal state is written, so a crash between the two does not leave an invisible failure. The ordering favors a duplicate notice over a dead demand with no notice because the latter cannot be found or repaired.
For the customer whose message started the chain, this architecture changes the meaning of “the request succeeded.” Success at the webhook means the event was accepted and recorded. Success at the queue means a worker completed its owned work. Success at the channel means the relevant send outcome was accepted. Each boundary has its own state and evidence. The next question is what the agent does inside its job: what context it reads, which action it is allowed to take, and what durable memory it must leave when the model session ends.