Encrypted Is Not the Same as Authenticated
RustDesk's transport logic can produce a stream, but a stream is not yet a trusted remote session. This chapter follows the security lifecycle that gives the stream meaning: what the rendezvous public key authenticates, how the endpoints bind a peer ID to an ephemeral key, how ordinary frames are encrypted, and why WebRTC adds a separate fingerprint check. The useful judgment is to ask three different questions—who authenticated the coordinator, who authenticated the peer, and what protects the bytes—instead of treating “secure connection” as one undivided property.
The rendezvous key authenticates infrastructure, not the whole session
The rendezvous server participates before the endpoints have a peer-to-peer stream. In a TCP rendezvous exchange, RustDesk can receive a signed key-exchange message and verify it against the configured rendezvous public key. If verification succeeds, the client has evidence that the key material came from the infrastructure identity it trusts. That is important: an attacker who can redirect a connection to an arbitrary server should not be able to substitute a new rendezvous key without being detected.
But this first check answers a limited question. It authenticates the coordination service or the server identity represented by the configured key. It does not, by itself, prove that the machine answering a peer request is the intended device. A trusted coordinator can still coordinate a connection to the wrong endpoint if a later peer-identity step is absent or fails open. This is why RustDesk keeps infrastructure trust and peer trust as separate stages in the code.
It helps to imagine the public key as a seal on the meeting place, not as an identity card for every participant. The seal tells the client which rendezvous service it is speaking to. The endpoint handshake must still answer which device is on the other side of the route, and the user-authorization stage must still answer whether that device will accept the requested session. Confusing those stages would make a compromised or misdirected endpoint look trustworthy merely because the coordinator was familiar.
After a route is established, the endpoint handshake carries signed identity material. Each side has a RustDesk ID and an ephemeral public key. The signed structure binds those values together; the receiving side verifies the signature using the trusted public key material obtained through rendezvous, then checks that the claimed ID is the peer the controller requested. In plain terms, the coordinator supplies a trust anchor, while the endpoint proves which named peer is participating in this particular session.
The use of an ephemeral key matters because a peer's stable ID and its session's encryption key serve different purposes. The ID lets the controller select the intended device and compare the claim with its stored expectation. The ephemeral key is created for the connection's cryptographic exchange. The source supports this separation, but it does not turn it into a claim that every deployment has perfect key lifecycle management; key persistence, configuration, updates, and administrator practice remain operational responsibilities.
That division also explains what self-hosting changes. An operator may choose a different rendezvous and relay environment, but the client still needs the corresponding public key and the endpoint identity still has to pass the peer handshake. Moving the coordinator changes the trust anchor and the operational owner. It does not collapse the two authentication questions into one.
A failure at this stage is therefore meaningful. An invalid server signature, an invalid peer signature, a peer ID mismatch, or malformed key material is not equivalent to a slow route. It means the evidence needed to connect the route to the intended identity is missing. Some older or key-less paths have compatibility behavior, so the repository's own status distinction matters: a path can be encrypted without being displayed as peer-authenticated.
From a signed peer to a protected stream
Once an endpoint has verified the other side's signed identity, the code creates an ephemeral public-key exchange for a symmetric session key. Public-key operations are useful for establishing that key without placing the long-lived symmetric secret in the rendezvous message. Symmetric encryption is then practical for the continuing session because remote desktop traffic is a stream of frequent, varied messages: video frames, pointer movement, keyboard events, audio packets, clipboard changes, and file blocks.
The resulting lifecycle has a clear ownership change. Rendezvous helps the endpoints obtain trusted identity material; the endpoint handshake checks that material and produces the session key; the stream framing code owns the next-message encryption and decryption state. A relay may carry the resulting ciphertext, but it is not thereby the component that decides whether the peer ID is valid. This is the architectural reason to discuss coordination and payload separately rather than calling the entire path “the server connection.”
RustDesk's TCP framing code installs the symmetric key on the stream and encrypts each outgoing frame with a secretbox construction. The frame sequence number contributes to the nonce, so successive messages do not reuse the same nonce under the same key. On receipt, the stream reverses that operation and rejects a decryption failure. The framing layer is therefore not merely a wrapper around protobuf messages; it is part of the state machine that decides whether the next bytes can be admitted as a valid session message.
A nonce is a value that must be used safely with a cryptographic key. RustDesk's sequence-derived approach gives the two stream ends a progressing counter for ordinary framed traffic. The practical consequence is that encryption is attached to the life of this stream, not applied as a single blanket operation to a file or a screenshot. If the stream is reset, a new connection must perform the relevant setup again rather than inheriting an unexplained encryption state from an old route.
This is where cryptography meets ordinary product behavior. A remote desktop is not one request followed by one response. It is a long-lived exchange in which messages arrive while the display is changing and the controller is sending input. A sequence-aware framing layer gives the transport a continuing notion of message order and key state. It also provides a defined failure surface: a malformed or undecryptable frame is a stream problem, not an invitation to reinterpret the bytes as an unauthenticated control message.
The rendezvous server's role in the described peer handshake is to provide or forward signed coordination material; the endpoint code then verifies identity and establishes the session's keying state. That is different from saying that every deployment has the same visibility or logging behavior. The client repository does not include the complete server implementation and cannot, by itself, settle what an operator records or how an external relay is administered. It does show that transport choice and endpoint protection are separate design layers.
The distinction is also visible when WebSocket Secure is used. RustDesk can skip an additional encryption step for the rendezvous link when wss already supplies transport-layer protection, but the code comments explicitly preserve the separate end-to-end session path between clients. Avoiding redundant encryption on one hop is not the same as disabling the peer session's security logic. The word “secure” must always be attached to a particular link or to the peer session as a whole.
The security lifecycle can be read as a set of state transitions:
The diagram has two important asymmetries. First, ordinary TCP, UDP, or relay paths use the RustDesk session-key exchange to install secretbox framing, while WebRTC already has DTLS at the transport layer. Second, WebRTC still needs a way to connect the DTLS channel to the signed RustDesk identity. Encryption protects a channel; identity binding tells RustDesk which endpoint that channel terminates at.
The transitions also have different owners and different ways to fail. The rendezvous side can reject or fail to verify a server exchange before the peer handshake begins. The controller can receive signed identity material but reject it if the claimed ID is not the requested peer. The endpoints can verify each other's identities and still fail to install or use a session key. A stream can then decrypt ordinary frames successfully while a later WebRTC-specific binding check refuses the channel. Treating all of these as one “handshake” would hide which trust assumption actually broke.
The same layered model explains why a relay does not automatically become a cryptographic exception. A relay can be the place where two sockets meet, but the endpoint-side code still receives the peer's signed identity and performs the relevant exchange over that stream. The relay's presence changes the route and the operator responsible for availability; it does not by itself establish the peer identity or grant user permissions. Conversely, a direct socket does not remove the need for those checks. The path is a delivery decision, while the handshake is an identity decision.
There is also a practical difference between what the code intends and what a deployment proves. The client can be built with the right verification branches and still be configured with the wrong rendezvous key, an outdated endpoint, an untrusted custom server, or a compromised operating-system environment. A cryptographic audit would need to examine the algorithms, libraries, key provisioning, downgrade behavior, error handling, and server implementation together. The repository gives a reader a strong map of the intended transitions, but it does not turn that map into a certification.
Why WebRTC can be encrypted and still rejected
WebRTC negotiates a DTLS certificate as part of establishing its data channel. That gives the channel transport encryption, but the certificate fingerprint arrives as part of the WebRTC negotiation. A rendezvous or relay component that could swap the SDP fingerprint could try to make the controller establish an encrypted channel to a different endpoint. RustDesk addresses this by putting the local DTLS fingerprint into the signed peer identity and checking the remote fingerprint against the identity that the controller already verified.
When the fingerprint matches, the WebRTC channel can be treated as the verified peer's channel. When a trusted peer identity exists and the fingerprint is empty or different, the code fails closed. It does not quietly downgrade the mismatch to “probably fine.” The client can coordinate a fresh relay connection instead, so a WebRTC setup problem does not necessarily destroy an otherwise usable session. The new relay coordination matters because the first relay association may already have been consumed by the failed WebRTC race; a fresh route prevents a stale pairing from being mistaken for a clean recovery.
The most important boundary is the case in which no trusted peer identity can be verified. The source distinguishes this from a concrete mismatch. In that situation, RustDesk cannot bind a fingerprint to a verified peer, so the WebRTC channel can remain DTLS-encrypted while being reported as unsecured from the application's peer-authentication perspective. That is an uncomfortable but precise distinction. The bytes may be confidential against a passive observer, yet the application has not established the identity claim needed to present the route as authenticated.
This distinction is not academic. Suppose a user sees a connection indicator and assumes that encryption alone proves the intended machine is answering. The RustDesk model rejects that shortcut. The coordinator's signature says something about the coordination service. The signed peer material says something about the endpoint identity. The frame key says something about confidentiality and integrity of the stream. A WebRTC fingerprint comparison says whether the negotiated DTLS channel is the channel associated with that signed identity. Each statement has a different object and a different failure condition.
The same reasoning applies when a direct path becomes a relay. The route can change without changing the question RustDesk must answer about the peer. A direct WebRTC pair still has to pass the identity-binding check. A relayed TCP or WebRTC path still has to use the appropriate session handshake. The route determines how the endpoints exchange bytes; the security lifecycle determines what those bytes mean and whether the application is willing to continue. A failed direct attempt is therefore not permission to skip identity verification on the fallback.
The repository gives strong evidence for these intended mechanisms and for the explicit failure behavior. It does not provide a cryptographic audit, prove that every key is distributed correctly, guarantee endpoint integrity, or establish that external dependencies are flaw-free. It also cannot settle server-side storage or logging behavior outside the declared client evidence. The right conclusion is therefore neither “the relay can read everything” nor “encryption solves the session.” RustDesk's design separates coordinator trust, peer identity, transport encryption, and application authorization. A careful deployment judgment must keep those distinctions intact.