How Open vSwitch Turns Packets into Shared State
Open vSwitch (OVS) is the subject here, but the useful question is architectural: where does a packet go inside a virtualized host, and which parts of that path retain state shared by multiple virtual machines? Tracing that path gives the reader a practical model for understanding why a virtual switch cache can later become observable. The important result is not a catalogue of components. It is the relationship among a VM's virtual interface, the hypervisor's software switch, the physical uplink, and the forwarding state that OVS owns on behalf of all of them.
The path from a VM to the physical network
A virtual machine looks like a computer to the application running inside it. It has processors, memory, storage, an operating system, and a virtual network interface card, or vNIC. The vNIC is not a physical socket on the server. It is an interface presented by the virtualization platform and connected to a software-defined network inside the host.
That network is built from virtual switches. A virtual switch can connect several vNICs on the same host, allowing one VM to reach another without first leaving the server. It can also connect those virtual interfaces to a physical network interface card, or pNIC. Once traffic reaches a pNIC, the packet enters the data-center network and may travel to another virtualized host before reaching its destination VM. The virtual network is therefore a meeting point between two kinds of interface: virtual endpoints owned by guests and physical links owned by the host and surrounding network.
OVS is one implementation of the virtual switch in that meeting point. Its role is to decide what to do with an incoming packet. The packet arrives from a vNIC or a pNIC, OVS examines the packet headers, and a forwarding action sends it to another virtual port, an uplink, or another place defined by the network configuration. The packet may remain inside the host, or it may cross the boundary into the physical network. What crosses that boundary is ordinary network traffic, but the decision about where it goes is made by software running in the virtualization layer.
The OpenFlow model gives OVS a flexible way to make that decision. An OpenFlow controller supplies rules, and flow tables match incoming packet headers against those rules. A matching rule can direct traffic to a particular port or apply another action. This separation between rule definition and packet forwarding is useful in environments where VMs move, tenants share infrastructure, and network policies change more often than they would in a fixed physical network.
The same host can carry several kinds of traffic. Management traffic carries commands used to administer the hypervisor. Infrastructure traffic includes operations such as VM migration. Inter-VM traffic comes from applications running in VMs and may stay within the host or move through the data-center network. The categories have different operational roles, but they can all pass through software-defined networking components. NIST's guidance on VM protection treats the virtual network as part of the security design precisely because VMs are its end nodes.
The following map shows the ordinary relationships without assuming that every packet takes the same branch:
The diagram separates the virtual-switch role from the cache hierarchy only to make the flow legible. In operation, the caches are part of OVS's forwarding machinery. A hit can send a packet directly to the forwarding action, while a miss continues to another layer. The physical network is one possible destination path, not a mandatory step between every pair of VMs.
Why OVS does not treat every packet as a new decision
Walking all OpenFlow tables for every packet would make forwarding flexible but expensive. Many packets belong to flows that OVS has already seen, and their appropriate action is unlikely to change between adjacent packets. OVS therefore keeps remembered forwarding state in software caches stored in the host's main memory. The cache hierarchy is a performance shortcut: it lets common traffic avoid repeating the full rule lookup.
The first shortcut is the microflow cache. It is small and fast, and it looks for an exact match to a packet flow. Exact here has a practical meaning: the incoming packet must match the cached flow's relevant header fields closely enough for OVS to use the stored action. If that lookup fails, OVS consults the megaflow cache.
The megaflow cache is larger and uses wildcarded rules. Instead of representing only one fully specified flow, a megaflow entry can match a family of packets under a particular mask. The cache is organized into subtables whose entries share the same mask. OVS checks those subtables in order and stops when it finds a matching entry. If neither cache can resolve the packet, OVS follows the slower OpenFlow table path. Once the miss is resolved, the result can be retained for later packets, which is how a one-time rule decision becomes continuing forwarding state.
This arrangement creates three useful paths through the system:
- A microflow hit is the fast exact-match case.
- A microflow miss followed by a megaflow hit uses a broader cached rule.
- A miss in both caches reaches the slow path, where the OpenFlow pipeline performs the fuller lookup.
These labels describe processing paths, not three security zones. A VM does not automatically receive a private microflow cache merely because its network policy is separate from another VM's policy. The cache belongs to the OVS instance and is used by traffic that reaches that instance. The host's forwarding layer owns the state; guest VMs generate traffic that can create, reuse, or displace it.
That ownership distinction matters. A flow rule may express a policy boundary between two tenants, and a VLAN or overlay may keep their packet traffic logically separated. Yet OVS still has to process both sets of traffic. If the processing shortcut is shared, activity from one flow can change the state encountered by another flow. Network isolation and state isolation are related, but they are not identical properties.
Three interfaces, one shared decision point
It helps to follow the packet across the interfaces rather than imagining the virtual network as a single black box. At the guest boundary, an application writes data to its operating system, and the guest sends packets through a vNIC. The VM controls the application and guest-side packet generation, but it does not own the host's OVS cache. The packet crosses from guest software into the virtualization layer.
At the virtual-switch boundary, OVS receives the packet and applies its forwarding logic. The switch owns the lookup state and decides whether a cached result is sufficient. If the destination VM is on the same host, the action may deliver the packet to another vNIC without using a pNIC. If the destination is outside the host, the action sends it toward a pNIC and the physical network. In the reverse direction, a packet arriving from the physical network enters through a pNIC and is mapped back into the virtual network.
At the physical-network boundary, the pNIC and external switches carry packets according to their own configuration. VLANs, trunk links, overlays, and physical routing can determine which traffic is allowed to travel where. But those physical controls do not replace the software decision inside OVS. The packet still needs an OVS action when it enters or leaves the virtualized host.
The control plane adds another relationship. The OpenFlow controller defines or updates rules, while OVS applies them to arriving packets. This division improves flexibility, but it also means that forwarding state can change for reasons that are not visible to a guest application: a rule may be inserted, a VM may migrate, or a network profile may be updated. The cache is consequently a living part of the host's system state, not a static copy of configuration.
For the same reason, a service VM is more than an example application in the OVS side-channel study. A query to a service creates a real packet path through OVS. A remote client may not be able to talk directly to another VM, but it may be able to query a service that shares the switch. That service path gives the client a way to generate traffic and measure the response. The service does not need to cooperate with the victim for the shared forwarding layer to matter.
The distinction can be stated simply:
- Traffic state answers which packets are allowed or where they should go.
- Forwarding state records enough of past decisions to process later packets quickly.
- Isolation state determines which guests, networks, or tenants should be kept apart.
A secure design needs all three to line up. If traffic policy separates two workloads but forwarding state remains shared and measurable, an observer may learn something without violating the policy in the ordinary sense. That is the implementation boundary the OVS study investigates.
Policy isolation is not the same as state isolation
A useful test of the system model is to imagine two application tiers that should not communicate freely. An administrator might put one tier on a separate virtual switch, assign traffic to different VLANs, place a virtual firewall between them, or use an overlay that spans several hosts. Each choice changes the allowed traffic path, the physical scope, or the way the administrator manages the segment. None of those descriptions, by itself, tells us whether the forwarding state is held in separate cache structures or in one shared OVS instance.
This does not make segmentation meaningless. Segmentation can stop packets from reaching an unintended destination, and firewall rules can inspect traffic that crosses a boundary. It can also reduce which workloads are co-located. The narrower point is that the packet-policy boundary and the state-owner boundary answer different questions. “Can this VM send a packet to that VM?” is a policy question. “Can this VM's traffic change the processing state used by another flow?” is an implementation question. A design can answer the first with no while leaving the second shared.
The distinction is especially important when the network spans hosts. A distributed virtual switch can make virtual-switch configuration consistent across multiple hosts and support VM migration, while VLANs and overlays can carry logical segments beyond one host. Those features solve real management and mobility problems. They may also make it harder to infer where a particular forwarding decision is held, because the logical network can be broad while the performance state is maintained in host-level components. The security analysis therefore has to follow the actual OVS instance and datapath, not just the label attached to a segment.
Forwarding state also differs from configuration state. A controller may define a rule once, but cache contents reflect which packets have recently arrived and which cached results remain useful. Two hosts with the same intended rules can have different cache histories because their workloads differ. A VM migration can preserve an application role while changing the host-level forwarding context. Conversely, a host can retain shared structures even when the visible tenant configuration has not changed. The cache is a history of use layered beneath the more stable policy description.
Consider a simple sequence. A packet from one VM arrives and OVS must consult its fuller rules before it knows the forwarding action. Later packets from that flow can use a cached result. A packet from another VM then arrives through the same OVS instance. It may use a different result, miss one layer, or alter which cached entry is available to a later request. Nothing in this sequence requires the second VM to break a firewall rule or enter the first VM's address space. The shared system has changed the cost of processing a future packet, which is enough to create a possible observation if the cost difference survives the network.
The security meaning of a cache hierarchy
The cache hierarchy is not suspicious because it is a cache. It is security-relevant because it is shared, stateful, and on the critical path of traffic. Each packet can find an existing entry, miss and populate another layer, or alter which entries remain useful. Those state changes are normally invisible to the VM that caused them. They can still affect the time taken by a later request.
This creates an asymmetry between the owner of the state and the user of the network. OVS, running in the host, knows which cache path a packet took. A guest or remote client does not receive that fact as a field in the response. But if the paths have different processing costs, the client may estimate the hidden state from timing. The result is an observation channel built from normal forwarding behavior.
That possibility does not mean that the cache stores a readable copy of every packet or that a VM can inspect another VM's memory. It means that the implementation may expose indirect evidence about past traffic. The strength of that evidence depends on the difference between paths, the noise added by the network, the attacker's ability to send repeatable probes, and the state that other traffic shares.
This is why the system model must come before the attack details. Without the model, “microflow,” “megaflow,” and “slow path” sound like interchangeable technical labels. With it, they describe different owners, transitions, and forwarding costs inside one shared decision point. The next question is how those transitions occur: how an exact entry is replaced, how megaflow subtables are ordered, and how long each kind of state persists. Those details determine whether the shared decision point leaves a reliable trace.