Overview

Process-aware detection has two distinct targets: rolling Termination Prediction, which estimates whether an interaction is nearing decisive extraction, and Scammer Action Prediction, which forecasts future actions and techniques. This chapter explains why timing, false-positive tolerance, action structure, and benchmark-specific metrics matter more than a single accuracy score.

Learning outcomes

Termination Prediction estimates rising proximity to extraction; Scammer Action Prediction forecasts future actions, so they are not one task.
AUC, AUPR, and Alert Time capture ranking, rare high-risk performance, and lead time rather than one universal notion of accuracy.
Fluent continuations can miss the action or technique that advances a scam, and reported scores remain tied to the benchmark setup.

Keywords

forecastingterminationAUCAUPRAlert Timecalibration

What Should a Detector Predict?

This chapter asks what process-aware detection should actually predict when a conversational scam unfolds over time. It separates two tasks that are easy to collapse into one: estimating whether an interaction is approaching a decisive extraction phase, and forecasting what the scammer will do next. That distinction lets the reader interpret benchmark scores as evidence about particular capabilities rather than as a single verdict on whether a model “understands” scams.

Two forecasts, two decisions

The first task is real-time Termination Prediction. At each point in a partial conversation, the system receives the history so far and produces a rolling risk score. The score represents the model’s estimate that the next turn is about to enter the Termination phase—for example, that the interaction is approaching a request for payment, credentials, or another decisive form of compliance. The output is continuous rather than simply yes or no. Risk can rise gradually, remain uncertain, or fall when new evidence changes the interpretation.

This is an early-warning problem. A system does not need to wait for the explicit request that makes the scam obvious. It can ask whether the trajectory is becoming dangerous soon enough for a person or platform to intervene. The usefulness of the score therefore depends on more than its final classification. A detector that raises an alert only after the request may identify the phase correctly but offer little preventive value. A detector that warns very early may create unnecessary interruptions. Timing and false alarms are part of the task, not secondary details.

PreScam evaluates the risk trajectory with three complementary measures. AUC summarizes how well the predicted scores rank higher-risk turns above lower-risk turns over the course of a conversation. AUPR is particularly important because turns near Termination are sparse; it asks how much precision the detector retains as it tries to find those relatively rare high-risk points. Alert Time measures how many turns before Termination the system first crosses a threshold calibrated to a fixed 10% false-positive rate. Together, the measures distinguish broad ranking quality, performance on the rare positive region, and practical lead time.

These measures answer different questions about the same score stream. AUC can say that one system usually places a risky turn above a safer turn, but it does not say that a score of 0.8 means the same thing across conversations. It is a ranking measure, not a promise that the output is a well-calibrated probability. AUPR makes the minority region more visible, but a model can still achieve a useful area under the curve while raising its first alert too late for a real intervention. Alert Time brings the decision back to the timeline: after fixing a comparable false-positive tolerance, when does the system first become willing to interrupt?

The fixed false-positive rate is useful for comparing methods because it prevents one model from appearing earlier simply by warning on everything. It is not a universal operating point. A bank, messaging service, or individual user may accept different levels of interruption depending on the cost of a missed scam, the cost of a false alarm, and whether a human can review the warning.

Published by DeepMox

FAQ

What is Termination Prediction?

Termination Prediction is a rolling estimate of whether the next turn is nearing the Termination phase, such as a request for payment, credentials, or compliance.

How is Scammer Action Prediction different?

Scammer Action Prediction forecasts the sequence of future scammer actions from an observed prefix, including their psychological techniques.

Why are AUPR and Alert Time useful for scam detection?

AUPR focuses attention on sparse high-risk turns, while Alert Time measures how early a threshold is crossed at a fixed 10% false-positive rate. That operating point is useful for comparison but is not universal.

Previous9 minThree Layers of Scam Detection

Next13 minScam Detector Memory

Scam Risk and Action Prediction

What Should a Detector Predict?

Two forecasts, two decisions

FAQ