Early Warnings, False Alarms, and the Limits of Proof

The proposed detector makes a strong claim: preserving conversational state can catch slow-burn scams earlier and more reliably than inspecting a narrow slice of messages. This chapter diagnoses what the reported evidence actually supports. It follows the results through their denominators, baselines, ablations, and failure cases, then adds the human, privacy, and deployment conditions that separate a promising research system from a production-ready safeguard.

What the comparisons establish

The first result is reported on LoveFraud02, a corpus of 83 romance-fraud conversations. The conversations were truncated to their pre-consummation portions so that the system had to detect the fraud before the later aftermath. The proposed system detects all 83, with a mean first-detection latency of 9.06 turns and a median of 6. That is meaningful evidence that the detector can recognize a pattern early in this naturally occurring, long-form corpus.

It is not evidence about false alarms. LoveFraud02 contains scams but no benign conversations. A system could warn on every conversation and still achieve perfect recall there. The result answers a narrow question—can the system find the scams in this corpus, and how early does it first trigger?—but it cannot answer whether the warning is selective when the same system sees ordinary conversations.

ConScamBench-278 supplies that missing class, though in a controlled form. It contains 128 scam conversations and 150 non-scam conversations. The proposed system detects 126 of the 128 scams and leaves 146 of the 150 benign conversations unflagged, yielding reported accuracy of 97.8%, scam recall of 98.4%, precision of 96.9%, and a false-positive rate of 2.7%. The counts are more informative than the headline percentage: two scam conversations were missed, and four benign conversations were flagged.

The benchmark is broader than the romance-only corpus, but it is not a slice of live traffic. Most scam conversations are real-sourced, while 18 are LLM-generated adversarial simulations. The benign conversations are drawn from general dialogue sources plus authored hard negatives, and the paper separately constructs a scam-adjacent stress test. This makes the benchmark useful for controlled comparison and for exposing particular errors, but it does not establish a deployment base rate or a real-world false-alarm rate.

The recent-window comparison helps diagnose why state matters. On LoveFraud02, a classifier using only the last six messages detects 48.2% of scams, whereas the proposed system detects all 83 in the reported evaluation and does so early. On ConScamBench-278, the six-message window reaches 60.9% recall, far below the full system’s 98.4%. A full-transcript classifier performs much better than the narrow window, but it does not beat the proposed system on the reported scam recall: it reaches 96.9% on ConScamBench and 97.6% on LoveFraud02. The pattern is consistent with the architecture’s claim that a running summary can retain distant evidence without requiring the model to reread every turn at every decision.

The comparison does not prove that every summary implementation will work. It compares one proposed state representation, cadence, backend, and decision rule with particular alternatives on particular corpora. A different summary might omit the decisive identity claim.