How a Moderation System Sees a Message
HPAA's central failure does not sit inside one model layer; it emerges as a submitted message crosses several system boundaries. This chapter traces those boundaries—from intake and automated classification through escalation and release—and shows what the black-box threat model can observe at each point. The goal is to make a safe detector decision and a human-recognizable harmful message understandable as one control-flow failure, not as a contradiction.
The pipeline is a chain of decisions
An online moderation system is not simply a model waiting for text. It is a sequence of decisions about what enters the system, what representation is analyzed, what a score means, and who receives the result. The USENIX Security paper describes a common tiered arrangement: a user submits a post, comment, or message; an automated moderator classifies it as safe or unsafe, or returns a toxicity score; uncertain or contested cases may be sent to a human reviewer; and the platform ultimately releases, blocks, or otherwise handles the content.
Each stage owns a different kind of state. The submission interface owns the material a user sends and the formatting that the platform chooses to accept. The automated detector owns a classification signal, not the final truth about the message. The policy layer interprets that signal through a threshold or rule and selects an action. A human reviewer, when involved, owns a later judgment about a case that the automated path could not resolve confidently. The recipient sees the result of all those earlier choices, usually as a rendered message rather than as the detector's internal input.
That division is easy to miss because platforms often present the result as one simple status: allowed or removed. Internally, a continuous safety score may sit between the model and the action. A threshold converts the score into a category, and another policy may decide whether an uncertain category is held for review. The threshold is therefore an operating decision, not a measurement of objective harm. A lower threshold may catch more questionable content but create more false positives; a higher threshold may reduce friction while allowing more false negatives. The detector can only make that trade-off over the information it receives.
Uncertainty is supposed to trigger another control path. When the system cannot confidently decide, it may escalate the message to a human moderator. That reviewer can examine context that a binary model output does not contain and can apply a more nuanced policy. But escalation is conditional. It occurs only when the automated path recognizes enough uncertainty or risk to invoke it. If the detector misses the relevant signal and returns a benign decision, the message does not enter the review queue merely because a person would later interpret it differently.
The pipeline can therefore be read as a set of transitions:
- Intake: the platform accepts a submission and determines which characters, spacing, or formatting features it will preserve.
- Representation: the platform prepares information for automated analysis. In the setting studied by HPAA, moderation systems primarily operate on tokenized text and may not retain every visual cue that shaped the displayed message.
- Classification: the detector returns a binary decision or a score.
- Policy action: the platform blocks, allows, holds, or escalates according to that signal and its operating rules.
- Human exposure: a reviewer or recipient encounters the content in the form the platform displays.
The point of this list is not to describe every platform implementation. It is to show where responsibility changes hands. A failure at intake can remove a cue before the detector sees it. A failure in representation can flatten information that remains visible later. A policy can prevent escalation even when the score is ambiguous. A release path can expose content before a human reviewer has an opportunity to reinterpret it.
A useful audit asks three different questions at every crossing. Fidelity: did the detector receive the features that the recipient will rely on? Timing: did the safety decision occur before the platform made the relevant presentation available? Feedback: if the outcome was uncertain, did the system retain enough information to route the case to a reviewer? These questions describe interfaces rather than model brands. A detector can be accurate over its input while the end-to-end pipeline remains unsafe if fidelity or timing fails. Likewise, a human-review program can be well staffed yet ineffective for cases that never trigger its queue.
This separation also gives operators a way to classify incidents. A representation failure occurs when the display carries information the detector did not receive. A decision failure occurs when the detector's signal was available but the policy sent the case down the wrong path. An observability failure occurs when logs cannot show which representation, score, or policy version produced the release. HPAA is primarily evidence for the first class, but the harm becomes visible only when the first class interacts with the second and third. A message can escape because a visual cue was lost, because a threshold treated the remaining evidence as safe, and because the system cannot later reconstruct what the recipient saw.
The sequence has two important asymmetries. The render layer and the detector are both downstream of intake, but they need not receive equivalent information. And human review is a branch, not a guaranteed final step. The message reaches a recipient on both the direct-safe path and the reviewed-release path, while only one of those paths necessarily has a person inspecting it before delivery.
This is why a later human safeguard cannot automatically repair a first-line miss. Human review is valuable when the system routes the case there. It does not reconstruct all content that the automated gate has already treated as ordinary. In a high-volume service, the platform cannot ask reviewers to inspect every allowed item without giving up the scale that motivated automation. The architecture consequently depends on the first screen to identify which cases deserve more attention.
The first screen also shapes what the platform can observe later. If it stores only a score and a normalized text string, an incident investigator may not be able to reconstruct the exact presentation that a recipient saw. If it stores the rendered form but not the detector input, the reverse problem appears. A reliable account of moderation behavior therefore needs to know which representation crossed each interface and which component owned the decision at that moment. HPAA's contribution is to make that often-hidden distinction the object of measurement.
The boundary that decides what gets seen
The most consequential boundary is between a message's presentation and its machine representation. A person can use spatial arrangement, spacing, emphasis, and other visual cues while reading. A token-based moderation system generally reasons over a sequential representation. Those descriptions do not imply that all models are blind to all layout, or that all platforms discard formatting. They establish a conditional vulnerability: when the displayed message retains a cue that the detector does not represent with comparable strength, the human and the classifier can reach different conclusions.
It helps to distinguish three forms of “the message.” The submitted form is what the user sends through the platform's interface. The detector form is what the moderation component receives after any platform-side preparation. The displayed form is what a recipient sees after the platform renders the content. In a simple plain-text service, the three may be close. In a service that supports emphasis, special character forms, or spatial organization, they may differ materially. The security question is not whether the strings are technically related; it is whether the detector's representation preserves the features a human relies on to reconstruct meaning.
Human interpretation is sensitive to relationships among visible parts. A reader may notice that fragments are grouped, highlighted, or arranged in a regular pattern and infer that they belong together. A sequential detector input may instead place those fragments beside benign material, represent the styling weakly, or treat unusual characters as ordinary tokens. The reader and the model are then solving different reconstruction problems. One is interpreting a presentation; the other is classifying the representation selected for it.
That difference does not require a model to be unintelligent. A detector can be highly capable within its input contract and still be poorly matched to the platform's display contract. If the moderation service promises to judge what users will encounter but passes only a reduced text view to the classifier, it has created hidden coupling between the rendering layer and the safety decision. Any formatting feature added for product reasons can become a new moderation dependency unless the detector's input is updated at the same time.
The paper's threat model tests this boundary with limited knowledge. The adversary is a non-privileged user who submits content through an ordinary platform interface. They do not have access to the target model's parameters, architecture, training data, moderation logic, or gradients. They observe only the outcomes the platform exposes, such as a safe/unsafe response or, where available, a confidence score. Their objective is not to control the detector internally. It is to find a submission that retains harmful meaning for a human while receiving a benign moderation outcome.
That black-box condition matters for two reasons. First, it makes the issue relevant to deployed interfaces rather than only to researchers who can inspect a model. Second, it makes feedback itself part of the attack surface. If one submission returns a safe result and another returns an unsafe result, the difference provides information even when the platform reveals nothing about the model's reasoning. The paper therefore treats the number of attempts as a practical constraint. Repeated submissions can trigger rate limits, account restrictions, or human review, so one successful submission and a success after many trials have different operational significance.
The ordering of the pipeline amplifies the mismatch. Consider a message whose display preserves a visual relationship that a person can interpret, while the detector receives a less informative sequence. If the classifier returns safe, the platform may release the rendered form directly. The recipient then supplies the perceptual work that the first-line filter failed to perform. If the platform instead routes borderline cases to reviewers, that route helps only when the detector's output crosses the escalation boundary. A blind spot below that boundary is not an “uncertain” case from the platform's perspective; it is an allowed case.
Several deployment conditions can remove or change this pathway. A platform may strip formatting or transform the message before display, so the human no longer sees the cue HPAA depends on. It may feed a rendered representation or image-like view to a multimodal detector, giving the classifier access to information that a token-only system lacks. It may require human approval before release, eliminate informative black-box feedback, or impose controls that make repeated attempts impractical. These possibilities do not invalidate the paper's result. They define the conditions under which the result should or should not transfer.
The reverse conditions matter as well. A platform may preserve a visual treatment for accessibility or ordinary communication but send a different form to a detector. Different devices, fonts, clients, and text-rendering libraries can also affect whether a cue is apparent to a recipient. A defense that looks effective in one interface may fail when another client renders the same content differently. Conversely, a blanket formatting sanitizer may suppress the channel while damaging legitimate emphasis, language-specific shaping, or users' ability to communicate clearly. The system-level question is therefore not “can formatting be removed?” but “which presentation semantics must remain, and how will the safety layer see them?”
This framing keeps the result narrow and useful. HPAA does not show that every automated moderator will miss every human-perceptible cue. It shows why a moderation architecture that treats tokenized text as a sufficient proxy for rendered communication can have a structural blind spot. The next question is which controllable presentation choices create that divergence—what gets grouped, where it is placed, and how it becomes visually salient—and how HPAA measures whether human recognition survives.