From Research Signals to Candidate Ideas
This chapter follows AutoResearch’s intake path: how it gathers heterogeneous signals, removes obvious noise and duplication, and decides which items deserve deeper model judgment. The practical question is narrower than “what is important in AI research?” It is how a workflow can turn public attention into a manageable candidate set without mistaking popularity, availability, or fluent classification for evidence. Understanding that distinction lets a reader see both the usefulness and the bias of the project’s discovery machinery.
Eleven channels, one intake contract
Research signals do not arrive in one shape. A Reddit discussion may have a title, score, comment count, and self-written text. An arXiv record may offer a title and abstract but no meaningful community score. A news feed may provide a headline and summary, while an open-review page may expose a different set of fields again. If every downstream stage had to understand each source separately, the system would spend its complexity budget on adapters rather than research judgment.
AutoResearch addresses that problem with a declared channel roster and a common internal item shape. The roster spans three kinds of signal. Community sources include Reddit communities, Hacker News, and GitHub Trending. Academic sources include arXiv, Hugging Face Daily Papers, Emergent Mind, Paper Digest, researcher blogs and conference spotlights, and OpenReview. Media sources include several RSS feeds and a Chinese-media route. The point of the roster is not to claim that these channels are a complete map of research. It is to gather different forms of early visibility: technical discussion, formal papers, project activity, conference attention, and reporting.
Each channel carries a lookback window and a declared unit, such as a post, paper, or project. It also has a failure policy. Reddit and Hacker News are treated as the required community signals in the current design; if one of them fails, the collection run stops rather than quietly presenting an empty community-first result. The other channels are optional. A timeout, missing dependency, or missing configuration on one of them is recorded and the pipeline continues with the remaining signals.
That asymmetry is a precise trade-off. It protects the project’s chosen intake thesis—that real discussion is an important discovery signal—while preventing a secondary source from becoming a single point of failure. It also makes absence interpretable. If a media collector fails, the run has fewer media signals. If the required community routes fail, the system does not pretend it has completed the same kind of discovery. A resilient pipeline is not one that always returns a result; it is one that says when the result was produced under a narrower view of the world.
Once collected, items are normalized into fields such as title, URL, comment count, score, summary, source, and source type. The normalized shape does not make the underlying sources equivalent. It only gives later filters a stable interface. The source type remains attached so the pipeline can apply different rules to a community post and an academic record rather than erasing the distinction during preprocessing.
The next protection is deduplication. AutoResearch removes repeated items globally by URL or by a normalized title. Title normalization strips common labels such as “Research” or “Show HN,” removes punctuation and spacing noise, and compares a bounded lowercase form. This catches a familiar failure of multi-source collection: the same paper may appear as a community discussion, a trend item, and a media summary. Without deduplication, the system could interpret repeated visibility as several independent signals and spend its deeper judgment budget on one work multiple times.
Deduplication is not the same as identity resolution. Two different projects can have similar titles, and two records of one work can have different URLs. The implementation uses practical URL and title safeguards, not a universal scholarly graph. That is appropriate for a first-stage intake layer, but it is also a boundary: a surviving item is not guaranteed to be unique in the world, only not matched by the rules the pipeline currently applies.
Discussion quality is a triage rule, not a value judgment
After normalization, the pipeline separates community material from academic and media material. Community items must meet a default minimum of 15 comments and a score of 50, have a nontrivial title, and avoid a list of markers associated with memes, rants, jokes, or purely conversational prompts. Academic and media items pass through a more lenient version of the same filter because their usefulness is not defined by the discussion metrics that community collectors provide.
The thresholds encode a hypothesis about discovery: sustained, positively received technical discussion is more likely to contain a research insight than an item nobody engaged with. That hypothesis can be useful operationally. It focuses expensive model calls on material that has already attracted some attention and avoids spending time on obvious low-signal entries. But it does not say that an unpopular paper is unimportant, or that a highly commented thread is scientifically sound. It is a recall-and-budget decision made before the system has performed substantive research evaluation.
This is why the source type matters twice. It controls the intake requirements, and it remains available for later interpretation. A community score is a social measurement. An academic item’s presence in a curated paper feed is a different kind of signal. Treating both as one numerical scale would create false precision. AutoResearch instead lets them enter a common candidate pool while retaining the fact that they arrived through different routes.
The filter also excludes titles that look like pure product announcements, generic opinions, or entertainment. This is not a claim that an engineering release or application demo can never contain insight. It is a deliberately conservative first screen aligned with the project’s aim to find a concrete research method, phenomenon, or mechanism rather than another broad platform story. A project that fails this screen may be useful for a different purpose; it simply does not fit the current path’s next question.
From attention to insight
Surviving items enter an LLM-based insight filter. The prompt asks the screener to retain work with a novel and specific technical insight, a deep improvement to an existing method, an unintuitive phenomenon, or a compact but powerful approach. It rejects broad engineering systems, product releases, leaderboard-only claims, pure application demonstrations, and news. The filter is therefore not trying to summarize everything collected. It is trying to identify which items contain a mechanism that could support a research question.
That distinction is the pipeline’s most important conceptual narrowing. “People are discussing this” answers a visibility question. “This contains a mechanism worth testing” is a research triage judgment. The second judgment is still supplied by a model and can be wrong, but the prompt at least asks it to look for a type of object that later experimentation can interrogate.
There is a cost-saving exception. Text that receives a sufficiently high influential-person boost can pass the LLM screen directly. The project records the boost and the mentioned person, so the bypass is not invisible. It is also not a neutral shortcut: authority becomes a reason to skip one layer of model screening. That may be sensible when the goal is to avoid missing work associated with a known researcher, but it introduces a path where reputation can substitute for the same initial test applied to ordinary items.
The daily entrypoint adds another budget boundary when the insightful pool is large. It can score batches of titles for innovation value, retry malformed or missing score responses, and select a bounded number of seeds. The ranking helper records how many items actually received scores. If only part of a batch is scored, those items are placed after scored items without being assigned a fabricated middle value. If every scoring batch fails, the run stops instead of producing a plausible-looking top list with invented defaults.
That behavior illustrates an important engineering principle for discovery systems: an unknown score is not a medium score. An unscored item may be excellent, poor, or simply unavailable to the evaluator. Giving it a default value would turn an infrastructure failure into a claim about research quality and could push genuinely evaluated items out of the budget. AutoResearch preserves the gap so an operator can distinguish “not evaluated” from “evaluated and ranked low.”
Why comments influence the order
The candidate pool is sorted by a simple popularity-weighted key: twice the number of comments plus the score. Comments receive extra weight because the project is trying to find technically substantive discussion, not merely approval. A post with many comments but few upvotes can therefore outrank a post with a high score and no discussion. The choice is transparent and easy to audit, which is preferable to an opaque learned ranking at this early stage.
It is also easy to misread. The key does not estimate novelty, reproducibility, causal importance, or correctness. It estimates which items should be examined first under the project’s community-first discovery policy. The test suite protects one practical consequence of sorting: when a cap is applied, hot academic items must not be crowded out merely because community items were concatenated earlier in the list. The whole insightful pool is sorted before truncation, and the system announces how many candidates the cap dropped.
That announcement is more than a logging nicety. A silent cap makes a resource limit look like an observation about the world: “these were all the useful candidates.” An explicit truncation tells the researcher that the pipeline found more material than it chose to examine deeply. The difference is essential when comparing runs or interpreting a sparse set of final seeds.
Popularity can improve prioritization and still distort the search. Communities cluster around fashionable topics, well-connected authors, and projects that are easy to explain in a headline. The pipeline’s own thresholds and ranking formula cannot correct that bias. They can only make its influence visible enough to discuss. The later cross-domain and experimental stages must supply the kinds of evidence that attention cannot.
Pro judgment and the unparsed middle
The final discovery step applies a deeper Pro judgment to the leading insightful candidates. The model is asked to assess the core technical insight, why the community is discussing the work, whether the method is simple or an engineering accumulation, whether the mechanism can cross into another domain, and whether an academic group could reproduce it with realistic resources. It then returns one of three controlled judgments: strong recommendation, worth investigating, or unsuitable.
This stage adds structured reasoning to the earlier binary screen. A candidate can be socially visible and apparently insightful but still be a poor seed because its contribution is vague, its method is too broad, or its resource requirements make an experiment unrealistic. Conversely, an item need not be a clear winner to be retained: “worth investigating” preserves a middle category for ideas that deserve a closer look without being promoted as strong recommendations.
The parser is intentionally stricter than a casual keyword search. It looks for the final-judgment segment, handles common headings and qualifiers, and limits the portion of text in which a verdict can be recognized. That matters because a reason may contain words such as “not suitable” or “worth” that should not be mistaken for the actual conclusion. The raw model judgment and the parsed verdict are stored separately, so a later reader can inspect what the model said and what the pipeline extracted from it.
If the response is ambiguous, truncated, or missing the required judgment, the result becomes unparsed. It is not turned into “unsuitable” merely because the parser failed, and it is not admitted into Idea Forge as though the model had approved it. This is a narrower form of fail-closed behavior: the system refuses to manufacture a positive research signal from an unreadable answer, while preserving the candidate and the failure reason for inspection.
The same rule applies to the earlier ranking stage. A malformed score array, a missing score, or a failed endpoint is an event about evaluation coverage, not a measurement of the candidate. Keeping those categories separate may reduce the number of seeds that reach the next stage, but it gives the resulting set a more honest interpretation. It consists of candidates that passed specified screens and received a parseable deeper judgment—not candidates that happened to survive an opaque chain of defaults.
What this intake can and cannot establish
By the time an item becomes a candidate seed, AutoResearch has performed meaningful narrowing: it has checked source-specific intake conditions, removed likely duplicates, applied an insight-oriented screen, prioritized items under an explicit budget rule, and required a parseable deeper judgment. That is useful because later stages can spend more context and model calls on a smaller set of concrete possibilities.
It is not yet evidence that the idea is novel or true. The intake is optimized for finding promising questions, not answering them. A score and comment count can be manipulated or culturally uneven; an influential-person bypass can overvalue authority; an LLM can mistake a polished announcement for a mechanism; and a parser can only tell whether a response followed the expected structure. Even a correct Pro judgment remains a model-assisted recommendation.
The right interpretation is therefore conditional: AutoResearch has built a candidate pool under a stated discovery policy. That policy favors visible technical discussion, concrete insight, manageable resources, and parseable decisions. The next stage must add a domain-specific test. A filtered signal becomes more informative only when its mechanism is placed against the actual pitfalls, constraints, and opportunities of a research field, and when more than one model is asked to challenge the mapping.