How to Test Whether Unlearning Holds

The subject here is evidence: what a test can actually establish about an unlearned vision-language model, and what it leaves unknown. The focus is the gap between suppressing a target answer under an ordinary prompt and resisting recovery through a different query or renewed training. Readers should come away able to match each test to the claim it probes, read forgetting and retention together, and avoid turning a finite benchmark into proof that a model has universally forgotten.

That discipline matters because a model’s interface is persuasive. If the expected name no longer appears, the system seems to have complied. But the same result can arise from several causes: a targeted association may have been removed, a language pathway may have been blocked, the visual representation may still recognise the target, or general capability may have degraded so badly that the model no longer answers reliably. Evaluation has to separate those possibilities rather than reward the most convenient surface behaviour.

Change the access route, not just the question’s wording

The supplied study begins with three prompt settings. An original generative prompt asks the model to identify the person in an image. A paraphrased prompt asks the same thing in different language. A discriminative prompt changes the task: instead of asking the model to generate a name, it asks whether the person is a specified target identity. These tests appear similar, but they exercise different parts of the system.

An original prompt mainly measures whether the model will produce the familiar response in the familiar format. A paraphrase checks whether success depends on memorising one wording or whether the behaviour is stable across semantically equivalent instructions. The discriminative version is more revealing when the target is an identity or concept because it can test recognition without requiring the model to generate the target name freely. A model that has learned to avoid a particular token may still answer “yes” when asked directly whether the image is that person. Conversely, a model that has lost broad answering ability may produce a non-affirmative response for almost everything.

The prompt change therefore changes the claim. Passing the original prompt supports a narrow statement: under that query, in that configuration, the target was not returned according to the scoring rule. Passing the paraphrased prompt supports a somewhat broader statement about wording robustness. Passing the discriminative prompt supports a claim about a different access path, including recognition rather than only verbal generation. None of the three alone establishes that the target influence is absent from the parameters or representations.

The study then introduces three robustness attacks designed to make a supposedly forgotten association easier to expose. The first is an in-context attack. The evaluator prepends auxiliary context that is related to the target but does not explicitly name it. For a public identity, the context might refer to a profession, role, or broad career characteristic. The original image and question remain, but the language context now supplies semantic cues that can interact with the visual representation.

This is not simply giving the answer to the model. The context is intentionally indirect. If the model succeeds only when the name is written in the prompt, the test has not shown much about residual knowledge. If related clues make the target reappear, the result suggests that the unlearning procedure blocked a direct retrieval route while leaving enough of the visual, textual, or cross-modal association for context to reactivate it.