Inside the Handshake: Where Agility Succeeds or Fails
Post-quantum migration becomes a live systems problem inside the security protocol handshake. The specific question is how two endpoints identify a mutually acceptable cryptographic path, protect that choice, and combine traditional and post-quantum mechanisms without mistaking key exchange for complete authentication. Following that sequence gives you a practical way to locate interoperability failures, downgrade risks, and the boundary between a hybrid connection and a quantum-resilient one.
The handshake is a negotiation, not an algorithm call
A security protocol does not simply invoke whichever algorithm an engineer selected at design time. Communicating parties first need a common description of the cryptographic services they will use. A cipher suite can cover several of those services: integrity protection, authentication, key derivation, key establishment, encryption, and digital signatures. The protocol then needs a way to identify the choices, discover what the other party supports, apply local policy, and confirm that the resulting selection was not altered in transit.
A simplified TLS-like exchange makes the dependencies visible. The client begins by advertising supported protocol versions, cipher suites, and key-establishment groups. The server chooses an option from the intersection of the two sides' capabilities and policies. The server presents credentials so the client can authenticate the service. The parties then establish a shared secret, derive traffic keys, and protect application data. Each step has a different owner and a different failure mode. A client can lack the group that the server prefers; a certificate can fail validation even when the key exchange succeeds; a policy can reject the common choice; or an attacker can try to influence the selection before the session is protected.
The first design choice is how the protocol names what is happening. Some protocols carry one identifier for each algorithm; others use a cipher-suite identifier that names a combination of algorithms. Explicit identifiers make it easier to add a new choice without changing the entire protocol version. If the identifier does not specify every important parameter, the remaining parameters—such as a key size—must be negotiated or firmly defined elsewhere. Otherwise, two implementations may believe they support the same suite while making different assumptions about the material it carries.
Neither identifier model is free of cost. A suite identifier can prevent an incomplete combination from being specified, because the suite selects the full set of cryptographic services. But if many algorithms and parameters can be combined, the number of suites can grow rapidly. Separate identifiers avoid some of that combinatorial growth, but they move the responsibility to implementations and local policy: the endpoints must decide which combinations are acceptable. Crypto agility therefore requires more than a large menu. It requires rules that keep the menu coherent as new algorithms appear and old ones disappear.
Mandatory-to-implement algorithms show the tension between interoperability and change. A standards organization may specify many permissible algorithms while requiring every implementation to support at least one common set. That baseline helps two independently built systems connect. Over time, however, the mandatory choice may become too weak. NIST recommends that standards organizations be able to revise the required algorithms without rewriting the base protocol specification, for example by maintaining an updateable companion document. This separates the relatively stable mechanics of the protocol from the changing set of algorithms it carries.
That separation only helps if the ecosystem moves before the old choice reaches its breaking point. A new algorithm must be specified, implemented, deployed, and recognized by enough peers while the existing choice still works. Protocols also need a way to learn whether a peer has adopted the new option. Capability information can be exchanged during negotiation or through a management mechanism. Without that visibility, an administrator faces a dangerous choice: disable the old path and risk breaking an unknown dependency, or keep it enabled and leave a vulnerable option available.
A useful way to see the problem is to follow one proposed upgrade through a connection. The protocol specification can define an identifier for a new key-establishment method, and a library can implement that identifier. The server can then list it as supported. None of those facts tells the operator whether the client will advertise the same method, whether local policy will permit it, or whether the selected parameters fit the certificate and message limits on the path. The connection succeeds with the new method only when the choices intersect at runtime. A migration plan that measures only the server's feature set is measuring a possibility, not the state of the communication system.
This runtime intersection also explains why standards and deployment need to move together. A standards organization can publish a method before every implementation supports it, which creates a window for testing and rollout. If it waits until the older method is already unsafe, the ecosystem may have to choose between a rushed deployment and a prolonged exposure. Early support is useful because it creates room for interoperability evidence; it is not evidence by itself that the system has completed the transition.
Key and certificate lifetimes make the timing less local than a single session. A cryptographic key is associated with a particular algorithm. Certificates place a validity period around a public key, and revocation can tell a relying party to stop using it before expiration. These controls can support transition, but only if the parties that issue and validate the credentials understand the new algorithm and if old credentials are handled deliberately. A protocol change can therefore be technically correct while the surrounding identity system remains unable to carry it.
The most important protection in the negotiation is integrity. If an attacker can modify or suppress the algorithms that a client advertises, or the selection that a server returns, they may force both sides onto a weaker option that they would not otherwise choose. This is a downgrade attack: the attacker does not need to break the strongest algorithm if they can make the endpoints avoid it. A protocol that uses one fixed integrity algorithm to protect negotiation creates another dependency, because a weakness in that algorithm could undermine the choices it was supposed to protect. The integrity mechanism must be strong enough for the futures the protocol is meant to support and must itself have a path to change.
The downgrade boundary is easy to miss because the visible result can still look healthy. The client and server may complete a handshake, encrypt the application data, and report no transport error. If an intermediary has removed the stronger option before the endpoints make an authenticated decision, however, the session can be confidential only against an attacker who respects the original offer. The protocol needs a way for the parties to detect that the negotiation was altered or that the resulting choice violates policy. Otherwise, availability becomes the reason the weaker path is accepted, and the transition mechanism quietly preserves the old dependency.
This does not mean that every fallback is an attack. During coexistence, a server may need to speak with an older client, and an administrator may knowingly accept a narrower security level for a bounded population. The architectural requirement is that the exception be visible, authorized, and measurable. A controlled fallback can be a temporary bridge; an unobserved fallback is an unbounded part of the attack surface. The same distinction applies when an old choice remains available for decrypting existing data but is prohibited for creating new protection.
More agility can make this boundary harder to defend. Each optional algorithm creates another code path to implement, test, configure, monitor, and eventually remove. Complex negotiation can increase the chance of an implementation error or an overlooked downgrade path. Yet too little choice can make a transition impossible when the mandatory option weakens or a peer needs a different security level. The design problem is not to maximize alternatives. It is to provide the alternatives that are necessary, make the selection explicit and protected, and retire unused or disallowed paths instead of letting them accumulate.
Hybrid key exchange changes confidentiality first
Hybrid mechanisms are a practical response to uncertainty during the move from traditional public-key cryptography to PQC. A hybrid key-establishment method combines a traditional exchange, such as elliptic-curve Diffie–Hellman, with a post-quantum key-encapsulation mechanism (KEM), such as ML-KEM. The parties combine the resulting components to derive a shared secret. The security rationale described by NIST is that at least one component should remain strong over time, allowing the hybrid to preserve protection while the new mechanism is studied, implemented, and deployed.
The sequence of a hybrid exchange is still subject to the protocol rules above. Both endpoints must recognize the identifier for the combined method. Their implementations must be able to carry the larger key share or ciphertext. Their policies must permit it. The negotiation must be protected so an attacker cannot remove the hybrid choice and induce a classical fallback. And the parties still need an authenticated identity mechanism. A hybrid key exchange can improve the protection of the shared secret, but it does not by itself change the signatures on the certificates that tell the client which server it is talking to.
The fallback branch is not automatically a vulnerability; compatibility may require a classical path while peers are being upgraded. It becomes a security problem when the fallback is invisible, unauthenticated, weaker than policy permits, or retained after the organization has decided that the older mechanism is no longer acceptable. The protocol must make the choice observable and protect the negotiation that produced it. Otherwise, a server may appear PQC-capable in a product description while a real client continues to receive a classical session.
Hybrid signatures expose an additional layer. A hybrid signature combines a traditional signature algorithm with a PQC signature algorithm. It may require two public keys to be certified. One design can place both keys in a single certificate so they are always used together; another can use separate certificates and separate roots of trust. The first can simplify the conceptual relationship between the keys but creates a combined trust object that every relying party must understand. The second can make it easier to stop issuing certificates under the traditional root later, but it can require validation of two certification paths during the transition.
This is why key exchange and authentication should be treated as separate migration decisions even when a product presents them as one feature. Key exchange protects the derivation of a session secret. A certificate signature and its trust chain establish who is allowed to participate. A service can have a hybrid answer to the first question and a wholly classical answer to the second. If a future quantum adversary can forge the classical certificate signature, they may impersonate the service and conduct a man-in-the-middle attack. Confidentiality and authenticity are related, but they do not move through the system on the same schedule.
The trust decision has its own deployment boundary. A certificate is not useful merely because it contains a public key of the desired kind; relying parties need a validation path and a trust anchor they recognize. If a service presents two credentials, clients and operators must know whether both paths are required, which one governs acceptance, and how failure is reported. If the roots are changed, the new roots must reach all relevant relying parties without making the old trust relationship impossible to manage during coexistence. These are operational questions about who issues, distributes, validates, and retires identity material. They cannot be inferred from the fact that a key-exchange group appeared in one handshake.
The hybrid approach also creates a second-transition question. If the traditional component remains in the hybrid after PQC is broadly deployed, the organization may eventually need to remove it when policy disallows that algorithm. Keeping it may be operationally attractive because its overhead is small or because it preserves compatibility with older peers. Removing it may be necessary because its security no longer meets the requirement. Either way, the choice should be deliberate. A hybrid design postpones one decision; it does not make the decision disappear.
The security analysis can also be harder than analyzing either component alone. The protocol must define how outputs are combined, how failure is handled, which component's parameters are negotiated, and how certificates and trust anchors represent the result. More code and more paths create more opportunities for defects or side-channel leakage. The value of a hybrid mechanism is therefore conditional: it can provide a useful bridge when its construction is sound, its negotiation is protected, its resource demands fit the path, and the organization has a plan for authentication and eventual retirement.
The practical lesson is to inspect a handshake in layers. First ask what the endpoints advertised and what they selected. Then ask whether the selection was allowed and integrity-protected. Next ask which key-establishment components contributed to the shared secret. Finally ask how the endpoint identity was authenticated and which trust store accepted the certificate. A successful encrypted session answers none of these questions by itself. It only proves that the endpoints found some path they were willing to run.
That layered inspection supplies the right checkpoint for the next stage. The question is no longer whether protocols have a place to carry PQC. It is whether a real client-to-server path is supported, negotiated, authenticated, observable, and controlled from end to end. The difference between those conditions is where partial migration becomes a systems judgment rather than a feature announcement.