The Control Surface: How sched_ext Lets a BPF Program Schedule Tasks
The Linux sched_ext scheduler class is the control surface that allows SECT to turn scheduling into a testing mechanism. The key question here is not whether a BPF program can run inside the kernel, but what authority it receives over a task and where the scheduler core retains ownership. Following one task from wakeup through dispatch, queueing, execution, and release makes the later SECT design intelligible: the framework can supply an exploration policy because sched_ext exposes the right decisions without handing over the entire operating system.
A scheduler policy inside the kernel, not a replacement for the kernel
sched_ext lets userspace load BPF programs that implement a scheduler through struct sched_ext_ops. The interface is deliberately broad: a scheduler can define how it chooses CPUs, enqueues tasks, dispatches work, and reacts to lifecycle events. The operations are optional apart from the scheduler’s name, so a minimal policy can rely on core defaults while a more ambitious one takes responsibility for more of the path. That is the opening SECT needs. Its exploration logic can be supplied as a policy rather than welded into the kernel’s fixed scheduling classes.
The boundary matters. Loading a BPF scheduler does not mean that BPF now owns every task or every CPU in an unconstrained way. The kernel scheduler core still determines when callbacks are valid, maintains task state, and mediates the transition between a task being runnable and actually running. A BPF program proposes scheduling behavior through the interface; the core enforces the interface’s lifecycle rules. This division is what makes “programmable” different from “unmanaged.”
Activation is also dynamic. sched_ext can be turned on and off while the system is running, which is valuable for a testing framework that should be loaded at test time rather than baked into every kernel boot. The documentation describes several paths back to ordinary scheduling: terminating the BPF program, detecting an internal error, finding that a runnable task has stalled, or invoking the documented administrative abort sequence. In those cases, the BPF scheduler is aborted and tasks are returned to the fair-class scheduler. Diagnostic information is emitted to help investigate the failure.
That recovery behavior is part of the testing model, not merely an operational convenience. An exploration policy is intentionally unusual: it may reorder runnable work, hold tasks in a custom queue, or apply priorities designed to expose bugs rather than maximize fairness. If a policy violates an internal condition or stops making progress, the system needs a way to leave the experiment without leaving the machine dependent on the failed policy. The fallback limits the blast radius of a scheduler implementation error. It does not certify that the policy made a sensible testing choice, nor does it tell us whether a crash observed during an experiment is a real kernel defect or a consequence of an unsafe intervention.
The interface also has a version boundary. The kernel documentation explicitly gives no stability guarantee for the BPF scheduler callbacks, constants, and scx_bpf_ helper functions across kernel versions. That warning does not cancel the value of a kernel-native control surface, but it changes the engineering claim. A framework can be less invasive and easier to migrate than a large patch set while still requiring maintenance when the scheduler interface evolves. Portability is therefore an observed design advantage under particular versions, not a permanent property of every future kernel.
How a task crosses the dispatch boundary
The most useful way to understand sched_ext is to follow the task state rather than memorize callback names. When a task wakes, ops.select_cpu() is invoked first. Its return value is a CPU-selection hint and an opportunity to wake an idle CPU; it is not the final, binding decision about where the task will run. The core can reject an invalid choice, such as a CPU outside the task’s allowed mask, and the eventual dispatch decision is made later.
select_cpu() can do more than suggest a CPU. It can directly insert the waking task into a dispatch queue. If the task is inserted into the local queue associated with the selected CPU, the later ops.enqueue() callback is skipped. That fast path is useful when the scheduler already knows that direct dispatch is appropriate. It also creates a lifecycle distinction: code that stores state as though enqueue() will always follow a CPU hint can become inconsistent when direct insertion takes a different path.
If the task was not directly dispatched, ops.enqueue() receives it. The BPF scheduler can insert it into a local queue, the global queue, or a custom queue; it can also keep the task in BPF-managed data structures for a later decision. These choices are not interchangeable. A terminal local or global queue hands the task back toward ordinary core dispatch. A custom queue or BPF-side structure means that the policy has taken responsibility for deciding when the task leaves that holding area.
Dispatch queues, or DSQs, are the bridge between those two forms of ownership. There is a global FIFO and a local DSQ for each CPU by default, and a BPF scheduler can create additional DSQs. A CPU first looks at its local queue. If it is empty, it tries to move a runnable task from the global queue. If that also fails, the core invokes ops.dispatch(), giving the BPF scheduler an opportunity to populate the local queue from a custom queue or another BPF-side structure. The CPU then runs the first available task from its local queue.
This ordering creates two distinct opportunities for a custom policy. It can make an early placement decision while a task wakes, or it can hold a task and decide later when a CPU actually needs work. The first path can be efficient because it avoids another callback and puts work directly where it may run. The second path gives the policy more context: it can compare several eligible tasks, apply a queue-specific rule, or wait for a scheduling event that changes its priorities. SECT relies on this distinction conceptually. The scheduler’s exploration policy is not just a replacement FIFO; it needs a place to retain candidate tasks and a later point at which one candidate becomes the next local execution.
The core’s fallback sequence also prevents a custom queue from becoming an invisible dead end. If a local queue is empty, the core tries the global queue before asking the BPF scheduler to dispatch. If the BPF scheduler fails to move a usable task, the core can retry the dispatch path rather than assuming that a single policy decision must succeed. A race with a task property change can likewise make a previously planned move invalid, causing dispatch to be retried. For an exploration framework, these are not incidental implementation details: the set of runnable candidates can change between policy observation and actual execution.
The diagram’s important boundary is not the number of queues but the direction of responsibility. The core decides when to ask for a choice and when a task can execute. The BPF scheduler decides where to place a task or how to populate the next local queue. A task may move through a custom queue before it is made runnable on a CPU, but the CPU ultimately executes from its local DSQ. That arrangement lets a policy maintain a high-level ordering while the core retains a concrete execution path.
The term “BPF scheduler custody” names the state consequence of that choice. A task enters BPF custody when it is dispatched to a user-created DSQ or stored in BPF-managed data. When it later leaves that custody—because it is dispatched to a terminal queue, selected for execution by core scheduling, dequeued for sleep, or affected by a scheduling property change—the core invokes ops.dequeue() under the documented conditions. For a task in a custom queue, the callback is expected exactly once when custody ends. By contrast, a task sent directly to a terminal local, local-on-CPU, or global queue does not enter or leave BPF custody in the same way, so a dequeue callback is not the universal “task has run” signal.
This distinction becomes especially important when a task is re-enqueued. The documentation allows ops.enqueue() to be called multiple times without an intervening ops.dequeue() when a task remains in BPF custody. A policy that treats every enqueue as a new task, or every property change as an ordinary dispatch, can lose track of whether its metadata still describes the kernel’s task. Conversely, once a task has left custody, later changes to its properties do not trigger a dequeue through the BPF scheduler because the policy is no longer managing it. Correct exploration therefore depends on keeping shadow decisions aligned with the core’s actual ownership state.
For a testing scheduler, custody is therefore an ownership contract. Metadata such as “eligible,” “waiting,” or “next priority” is meaningful only while the BPF side is responsible for the task. When the task is handed to a terminal queue, the policy should stop acting as though it can still reorder that particular instance. When the task remains in a custom queue, repeated enqueue calls do not mean that the policy has received a fresh object; they are further transitions of the same managed task. The distinction prevents stale entries from surviving after execution and prevents a scheduler from selecting a task that the core has already removed for sleep, migration, or a property change.
The same contract explains why state changes can be more important than the nominal callback sequence. A task may be runnable when a policy examines it, then become unavailable before dispatch. A task may change affinity or scheduling properties while still under BPF control, requiring it to leave custody and re-enter through the core’s rules. A scheduler that ignores those transitions could produce an ordering that its own metadata describes but the kernel can no longer honor. SECT’s later shadow state must consequently follow the core’s lifecycle rather than pretending that its policy map is authoritative by itself.
The lifecycle is broader than wakeup and dispatch. Tasks can be enabled for BPF scheduling, become runnable, be placed in a queue, run until a slice expires or they block, become quiescent, and eventually exit. A property change—such as an affinity, priority, or scheduling-policy change—can force a task out of custody and back through a different callback path. A running task can also be preempted by a higher-priority scheduling class. These transitions are why SECT needs a disciplined scheduler layer rather than a single function that picks the next thread: the policy must update its view as the core changes the task’s state.
There is a second safety limit in the callback model. The CPU selected during wakeup is an optimization hint; it does not give a custom scheduler permission to ignore task affinity or other core constraints. Direct insertion can skip later callbacks, but it cannot bypass the task’s valid execution conditions. A custom DSQ can delay a task for exploration, but it must eventually participate in the lifecycle expected by the core. In this sense, sched_ext exposes a programmable dispatch boundary, not a private miniature operating system inside BPF.
The abort path completes that boundary. If a scheduler stalls a runnable task or triggers an internal error, returning to the fair-class scheduler protects system progress, while the debug dump preserves information about why the experiment stopped. For a fuzzing campaign, that creates a useful separation between infrastructure failure and target failure. A run that ends because the policy itself violated a scheduler condition cannot be counted automatically as evidence that the kernel code under test is defective. The recovery path keeps the machine usable, but the campaign still needs an observer and a triage decision.
SECT’s distinctive contribution comes after this boundary is in place. The general interface tells a BPF scheduler how to participate in dispatch and how to recover when it fails; it does not decide that one runnable task should be favored because its next event may expose a rare interleaving. That is the work of SECT’s temporal-isolation policy. The next question is how the same queues and lifecycle transitions can be used to serialize choices inside a test domain while independent test instances continue to run.