From Messy Reports to a Model of Progression
This chapter examines how a process benchmark can be built when the available evidence is not a clean chat log but a noisy victim report. The focus is PreScam’s construction funnel and its structured instance format, so the reader can see both what the benchmark makes measurable and why its headline scale must not be mistaken for a census of conversational scams.
The funnel selects a particular kind of evidence
A benchmark about progression needs more than a large pile of texts. It needs enough chronological information to say what happened first, what the scammer did next, how the victim responded, and whether the interaction moved toward a decisive outcome. Public scam reports are valuable because they contain real incidents, but they are not normally written for machine learning. A victim may summarize weeks of interaction in a paragraph, omit messages that seemed unimportant, mix dialogue with reflection, or describe the result without preserving the turn-by-turn exchange.
PreScam addresses that mismatch through a sequence of filters. The source corpus begins with 177,989 publicly available BBB Scam Tracker reports published between February 2024 and November 2025. An LLM-based extraction step identifies 25,402 reports that appear to contain usable multi-turn interactions. Cases with fewer than two engagement rounds are then removed, leaving 13,007 candidates for structuralization. After null or malformed entries are cleaned, the final benchmark contains 11,573 structured instances across 20 scam categories.
Those numbers describe a funnel, not a prevalence estimate. Each step changes the population. The first stage selects reports that appear to contain multiple interactions. The engagement filter selects cases in which a model can identify enough psychologically labeled rounds. The cleaning stage removes outputs that do not fit the expected structure. A report that disappears at one of these stages has not thereby been proved irrelevant to conversational-scam research. It may be too short, too ambiguous, too poorly recorded, or simply outside the representation’s requirements.
The funnel is a methodological choice. It makes a difficult research problem tractable by concentrating on reports from which a process can be reconstructed. Without such a choice, a benchmark might be larger but unable to support the tasks it claims to measure. With it, the benchmark becomes more coherent while also becoming less representative of every report in the starting corpus. A responsible reader needs both facts at once.
The extraction step is also a reminder that the benchmark is mediated. The source material is written by victims, but the decision that a report contains multi-round interaction information is made by a language model. A second modeling step then organizes the selected narrative into phases and actions. The result is not an untouched transcript recovered from a database. It is a structured interpretation of a report, with a traceable relationship to the report but with additional decisions about what counts as a round and where a phase begins or ends.
Quality control reduces some of the risk in that mediation. The construction process uses reflection and verification checks, and the paper reports an independent human evaluation of a random sample of 200 generated structures by three PhD-level reviewers. That evidence supports the claim that review improved the quality of the generated structures in the sampled comparison. It does not turn every label into an independently observed fact. A review score can reveal common errors, but it cannot restore turns that the original report never recorded.
This distinction matters for what a model trained on the benchmark learns. It may learn useful regularities about how victims describe scam progression and how actions can be aligned with psychological techniques. It may also learn the habits of the extraction procedure: which narratives are easy to structure, which categories are common, and which kinds of persuasion fit the available taxonomy. A high score can therefore reflect competence on the benchmark’s evidence regime without proving that the model would recover the same structure from a raw conversation on an unfamiliar platform.
What a structured instance preserves—and what it changes
Each PreScam instance keeps several layers of information together. It retains the scam category and original victim narrative, then adds an Initial Contact summary, an Engagement sequence, and a Termination summary. Engagement is represented through scammer actions and victim responses, with supporting verbatim spans and psychological-technique labels. The instance also records whether the scammer succeeded and a reason for that outcome. The design turns a retrospective account into something a progression model can inspect: it can ask what the scammer did, what the victim did in response, which phase the interaction occupied, and what evidence supports the description.
The original narrative is important because it preserves the source context. A summary alone might remove uncertainty, emotional detail, or a qualification that changes the interpretation. Verbatim spans provide a second anchor: instead of accepting a generated action because it sounds plausible, a reviewer can check whether the report actually supports it. This is particularly important when the system is assigning a psychological technique. “The scammer used fear” is a conclusion; a quoted description of a threat or impending loss is the evidence from which that conclusion should be evaluated.
The structured fields also make comparisons possible. Researchers can compare whether a particular kind of action tends to appear near the end of an interaction, whether scam categories differ in their use of techniques, or whether a partial history contains enough information to estimate proximity to Termination. Without a common representation, each study might use a different idea of a “turn,” a “next action,” or an “early warning.” The benchmark’s value is not only its number of examples. It is the shared vocabulary that lets separate systems be tested against a similar progression question.
But every added field is an interpretation. The extraction procedure must decide where one interaction round ends and another begins. It must keep scammer and victim roles separate even when a report moves quickly between quoted speech and retrospective description. It must decide whether an action belongs in Initial Contact, Engagement, or the final outcome. These choices can make the story easier to analyze, but they can also make it look more orderly than the underlying report.
The requirement that every Engagement round contain at least one psychological technique is a useful grounding rule with a side effect. It prevents the structured sequence from filling up with actions that have no supported persuasive function. At the same time, it means that neutral or unclear turns may be merged, moved, or discarded. The final Engagement sequence is therefore not a complete transcript of everything that happened. It is a representation of the actions that the construction process judged to be psychologically meaningful and supportable.
That side effect is not a reason to reject the benchmark. It is a reason to ask a precise question of every result: is the model learning progression in the reports, progression in the structured representation, or the decisions used to create the representation? These are related capabilities, but they are not identical. A system may be good at matching the labels produced by the benchmark while struggling with raw, ambiguous dialogue. Another may identify a suspicious direction that does not correspond neatly to a labeled Termination event.
A useful validity check therefore has three layers. First, does the structured action have support in the original narrative? Verbatim spans help answer that question, but they do not guarantee that the span was assigned the right role or phase. Second, would another careful reviewer make a similar structural judgment? Independent review and agreement studies can test consistency, though a sample cannot expose every systematic error. Third, does the benchmark task represent the decision a detector is meant to support? A label that says a later turn entered Termination is useful for studying progression, but it may not capture the earlier moment when a user could have been helped. A benchmark can be internally consistent and still measure a narrower capability than deployment requires.
These layers also clarify what a model comparison means. If two systems receive the same structured prefixes, their scores can show which one better handles that benchmark’s definition of risk or action. They do not by themselves show which system is safer for a person receiving incomplete, multilingual, platform-specific messages. Transfer requires evidence that the representation survives changes in channel, writing style, scam family, and available context. It also requires benign conversations that share some of the same social patterns, because a detector that only sees clearly documented scams may learn to be decisive where real users face ambiguity.
The construction process makes another trade-off visible. Filtering for recoverable multi-round cases improves the signal-to-noise ratio for progression research, but it can remove the cases in which a detector would have to work hardest: a single alarming request, a conversation that ends before the demand, or a report whose chronology is too thin to reconstruct. If those cases are absent, a high score can coexist with poor performance at the edges of the process. The benchmark should therefore be paired with tests that deliberately ask what happens when evidence is incomplete, the outcome is unresolved, or the conversation stops at an ambiguous point.
This is not a demand for a perfect dataset before any research begins. It is a demand for a precise claim. PreScam can support the study of structured progression because it provides a common representation with links back to report evidence. It can also reveal whether a model can use partial histories rather than only isolated keywords. The reader should simply preserve the chain of interpretation: the quality of a prediction depends on the quality of the labels, and the quality of the labels depends on what the original report made possible.
The same caution applies to the outcome fields. A binary scammed label and a reason make it possible to study whether an interaction ended in loss, recognition, disappearance, or another reported result. They do not provide a complete measure of harm. Reports differ in what victims know, what they choose to disclose, and when they submit the account. A reported outcome is evidence about that case, not a complete account of the scam’s consequences.
The benchmark’s category distribution adds another layer of interpretation. Employment scams make up 31.6% of the corpus and phishing 12.3%, while the remaining categories are much smaller. Engagement sequences range from three to fifteen rounds, with most concentrated in the shorter part of that range. These figures help describe the dataset, but they also show why an aggregate score can conceal uneven performance. A model that handles the dominant categories well may still be weak on less represented scam families. A model tested on short reconstructed sequences may not have demonstrated memory for long, naturally occurring conversations.
Narrative noise creates a related boundary. A report can be extremely short, miscategorized, missing a timeline, or written as a single retrospective event. Such cases are not merely inconvenient outliers. They are part of the environment in which real people report scams. Excluding them may be necessary to create a tractable progression task, but it means the benchmark answers a narrower question: how well can a system model cases with enough recoverable structure for the chosen representation?
That question is still valuable. It supports experiments that a raw report collection could not support, including rolling risk estimation and forecasting future scammer actions from partial context. It also creates a shared test for whether models can recover more than topical cues. The important discipline is to keep the layers separate: a report is observed evidence; a structured instance is an annotated interpretation; a model output is a prediction evaluated against that interpretation.
The boundary between measurement and reality
The construction funnel makes PreScam suitable for studying progression, but it limits what the benchmark can establish. It cannot tell us how common conversational scams are in the world, because the starting reports are not a random sample of all interactions and the final instances are selected for structural recoverability. It cannot guarantee that a reconstructed sequence preserves every turn or every adaptation. It cannot show that the same phase boundaries will transfer unchanged across languages, platforms, or scam families.
The benchmark is best understood as an instrument. An instrument can be well designed for a particular measurement while still failing to capture everything outside its range. PreScam’s range is the study of structured, multi-round scam reports and the progression questions that can be asked from them. Its evidence is strong enough to motivate process-aware evaluation, but its boundaries must travel with every result. The next useful step is to place observed evidence, structured labels, and detector predictions on one map, so that a later score can be read as a claim about a specific layer rather than as a verdict on conversational-scam detection as a whole.