What the Measurements Actually Prove
FIRMENSTEIN’s evaluation asks whether semantic roadblock diagnosis produces more than a convincing execution trace. This chapter focuses on how to read that evidence: what the connect and interact metrics measure, what the comparisons with specialist systems and generic coding agents support, and how rehosting gains carry into security analysis. The practical judgment is conditional. The results are substantial on the authors’ benchmark, but their meaning depends on denominators, selection procedures, validation boundaries, and the failure modes that remain.
That discipline matters because “rehosting success” can compress several different achievements into one word. A service may launch, accept a connection, handle a stateful interaction, expose more basic blocks, survive fuzzing, or reproduce a known vulnerability. Those are related outcomes, but each tests a different layer of the environment contract. A stronger result at one layer is evidence for the next question, not an automatic answer to it.
Read the benchmark from connect to interact
FIRMENSTEIN is evaluated on 450 unique Linux-based firmware samples. The set contains distinct vendor-model combinations, spans routers, switches, and IP cameras, includes ARM, MIPS, and x86 architectures, and covers release dates from 2018 through 2023. It is derived from the deduplicated LFwC corpus and augmented with 30 recent Tenda images. The authors selected 60 device models from each of seven major vendors, rather than allowing a web crawler to dominate the benchmark with many historical versions of the same model.
That construction addresses a real threat to interpretation: a large sample count can conceal low diversity. A corpus filled with near-duplicates may reward a repair rule for recognizing a family resemblance rather than for generalizing across firmware. The curated set is a stronger test of variation than such a crawl, while still remaining a sample chosen by the authors. It should be read as evidence about the evaluated population of Linux-based firmware, not as a prevalence estimate for every IoT device or every firmware architecture.
The paper uses two progressive service-level metrics. Connect means that a user can establish a connection to the rehosted service without abnormal termination. Interact requires the service to handle more complex exchanges correctly, assessed through the content it returns. The second measure is deliberately closer to the runtime contract described earlier. A port that opens proves that some initial path is available; it does not show that configuration, authentication, request processing, or stateful behavior still works.
On connect, FIRMENSTEIN reaches 194 samples. Greenhouse reaches 74, FirmAE 91, and Penguin 134. On the stricter interact measure, FIRMENSTEIN reaches 169, compared with 67 for Greenhouse, 55 for FirmAE, and 79 for Penguin. Those interact counts produce the paper’s reported 2.5-times, 3.1-times, and 2.1-times comparisons. The underlying counts are more informative than the ratios because they retain the 450-sample denominator and show that the result is not simply a relative improvement over a tiny successful set.
The gap between connect and interact is itself evidence. FIRMENSTEIN moves from 194 connected services to 169 interactive ones, so not every service that becomes reachable remains usable under a more demanding exchange. The baselines also lose services between the two thresholds, sometimes sharply. That drop is not a defect in the metric; it is what prevents a launch or connection from being mistaken for faithful behavior.