The Scheduler That Stops Being a Black Box
SECT, the Sched-Ext Concurrency Tester, is a Linux kernel concurrency-fuzzing framework built around a simple but consequential idea: the scheduler can help search for bugs instead of merely trying to keep the machine responsive. This opening focuses on the problem SECT is designed to change—rare thread interleavings—and on the practical judgment that follows from it. A syscall program may reach the right kernel code and still miss a bug if the operating system never lets the relevant operations occur in the necessary order.
That distinction matters because concurrency testing has two search spaces. One is familiar: which inputs, system calls, arguments, and sequences reach new code? The other is the ordering of events among tasks that are already running. SECT treats the second space as a first-class testing target while retaining the first. Its scheduler chooses among runnable tasks according to exploration policies, so a schedule is no longer only an accidental consequence of fairness, load, and timing. It becomes part of the experiment.
The bug is not just in the input
A sequential bug can often be exposed by finding an input that reaches a bad operation. A concurrency bug needs one more condition: another task must be allowed to intervene at a particular moment. The faulty code path and the faulty ordering are separate requirements. A fuzzer that is excellent at generating system-call programs can therefore remain weak at exposing races, stale state, deadlocks, or ordering-dependent null dereferences if it leaves the interleaving to the native scheduler.
The paper behind SECT illustrates this with a Linux JFS case. One thread, in dbUnmount, frees a bitmap object and then clears the stored pointer. Another thread, in jfs_ioc_trim, obtains and uses that object. A previous fix added a read-write semaphore to prevent the original use-after-free ordering, but it omitted a null check. The repaired code could still fail if the unmount path ran first: the second function would see a null pointer and dereference it. This was not a matter of inventing a more exotic syscall argument. It was a matter of making a particular ordering happen.
Under the default operating-system scheduler, the paper reports that the sequence did not reproduce the problem in 10,000 executions. With SECT controlling the relevant tasks, it reproduced the issue in fewer than ten. Those figures are evidence about that case and that experimental setup, not a universal conversion rate for concurrency fuzzing. Their importance is conceptual: repeated execution is not the same as repeated exploration. Ten thousand nearly identical schedules can provide less information about an ordering bug than a much smaller number of deliberately varied schedules.
This is why ordinary coverage can be misleading in concurrent code. Branch coverage says that execution reached certain control-flow edges; it does not say that all important cross-thread orderings around those edges were exercised. A syscall sequence can become a valuable seed for sequential fuzzing while remaining a poor test of concurrency unless the tasks derived from it are given meaningful opportunities to interleave.
The missing dimension is not simply “more randomness.” A race may depend on one task reading state after another task has invalidated it, or on two critical sections becoming adjacent in one direction rather than the other. If the scheduler usually lets a task run to completion, or if the relevant wakeups happen in a stable order, repeated executions can explore the same broad path while leaving the decisive transition untouched. Conversely, forcing arbitrary switches everywhere would spend most of the campaign on orderings that cannot communicate through shared state and would add cost without adding useful evidence. The testing problem is therefore to control the moments that can matter, while preserving enough ordinary execution to keep inputs valid and campaigns productive.
That distinction also explains why a concurrency failure is not synonymous with a data race. Some failures arise from unprotected conflicting accesses; others come from an unexpected ordering of operations that are individually synchronized. A schedule explorer must be able to expose both kinds of behavior, and the method used to recognize the failure must match the behavior being sought. The JFS null dereference is valuable here precisely because it shows how a schedule can reveal a defect left behind after one synchronization problem has been addressed.
Changing what the scheduler is for
Linux’s native scheduler is built to solve a different problem. It must share CPUs, respond to wakeups, preserve useful progress, and balance performance with fairness and latency. Those are essential operating-system goals, but they make the scheduler an imperfect instrument for systematically exposing rare interleavings. Natural timing variation can produce useful accidents; it does not give a tester a reliable way to request a particular class of ordering.
Earlier concurrency fuzzers have moved the control point outward or weakened the ambition. Delay injection can perturb timing with little machinery, but it may miss orderings that require a precise sequence of events. Hypervisor-level approaches can suspend and resume virtual CPUs or use hardware breakpoints, gaining stronger control at the cost of heavy setup, high latency, restricted parallelism, or dependence on virtualized hardware. Tools that patch the kernel directly may gain access to internal state while becoming expensive to maintain as the kernel evolves. These are not failures of engineering so much as different points in a control–cost–portability trade-off.
The location of control changes what the tester can know as well as what they can change. An external mechanism sees the kernel from across an abstraction boundary and must translate a desired schedule into delays, traps, or virtual-machine actions. A kernel-native mechanism can participate closer to the task lifecycle, but it inherits the kernel's rules about preemption, state transitions, and recovery. That makes the design more precise, not automatically simpler. The benefit is the possibility of expressing exploration policy where dispatch decisions are already made; the cost is that the policy must respect the same invariants as the system it is testing.
SECT changes the location of the control surface. It uses the Linux sched_ext scheduler class and eBPF to place a programmable scheduling policy in the kernel’s dispatch path, rather than treating the scheduler as an opaque obstacle to be manipulated from outside. The policy can select which eligible task runs next for a test domain. SECT then uses that ability to explore interleavings as a sequence of scheduling decisions.
Calling this a “scheduler-as-explorer” does not mean that SECT throws away the kernel’s scheduling machinery or makes every instruction independently selectable. It means that scheduling choices are given an explicit testing purpose. A policy can favor a different runnable task, preserve or change a priority, or react to an event that may interfere with a task already executed. The kernel still owns the task lifecycle and the boundary between runnable, running, sleeping, and exiting states. SECT supplies an exploration intent within that boundary.
The diagram separates responsibilities that are easy to conflate. Input fuzzing searches for executable behavior and new code coverage. Concurrent mutation creates a set of tasks that can interact. SECT controls part of their schedule. The kernel executes the result, and coverage or a bug detector supplies evidence for the next round. The native scheduler remains a valid execution mechanism; it is the comparison case in which the ordering is not intentionally explored.
Control also has a boundary in the diagram. SECT does not manufacture a bug merely by selecting a task, and it cannot infer that an execution is correct because no crash appeared. Its contribution is to make an otherwise unlikely execution reachable often enough for an observation to become useful. The observation may be a new branch, a sanitizer report, a deadlock, or a failure that must be compared with an unmodified execution. In each case, scheduling control increases the quality of the experiment; it does not replace the experiment's input, detector, or human interpretation.
That boundary is important for the claims made later. A higher rate of reproducing a known failure is strong evidence that schedule exploration addresses a bottleneck in that workload. It is not evidence that every schedule has been covered, that every report is caused by a real kernel defect, or that the same policy will behave identically on another kernel version. The scheduler becomes a testing instrument, and instruments are judged by their resolution, calibration, and operating conditions.
A map of the questions that follow
The scheduler-as-explorer idea is only useful if its control surface is understood precisely. A programmable policy must know when it can influence a task, how a task moves through dispatch queues, and when the kernel has taken the task out of the policy’s custody. Those are the mechanics provided by sched_ext. They determine what SECT can safely ask the scheduler to do and where the kernel retains authority.
A second question concerns the shape of exploration. Making a task selectable does not tell us which task should run next. A uniform random choice, a persistent priority, an interference-aware policy, and a policy that concentrates a small number of priority changes make different bets about which schedules are informative. Their benefits cannot be separated from starvation, fairness assumptions, switching overhead, and the behavior of the kernel code under test.
A third question concerns visibility and trust. Existing scheduler boundaries may be too coarse to expose the ordering that matters, which motivates additional scheduling points near shared-memory and synchronization events. But an extra preemption can be unsafe, and a missed or unsampled point can leave a bug invisible. Even a successful campaign therefore produces evidence under conditions, not a proof that the entire schedule space was searched.
This last distinction also places SECT alongside, rather than above, tools such as KCSAN. A race detector can inspect selected executions for conflicting memory accesses; it does not, by itself, decide which interleaving the kernel should run next. Conversely, a scheduler explorer can produce a revealing ordering without automatically classifying every failure as a data race. The strength of a testing system comes from matching control, input generation, and observation to the failure being sought.
The central claim of this series is consequently narrower and more useful than “the scheduler finds bugs.” SECT shows how Linux scheduling can become a programmable exploration resource for concurrency testing. Whether that resource is effective depends on the dispatch interface beneath it, the policy making each choice, the points at which control is reclaimed, the oracle judging execution, and the kernel version and workload in which the experiment runs.
The next step is to examine that dispatch interface. Before asking how SECT turns scheduling into search, we need to see what sched_ext actually allows a BPF scheduler to observe and control—and what the kernel still refuses to hand over.