Learning to Back Off

OmniGet's downloader cannot make the network predictable, but it can avoid confusing different symptoms and repeating the same mistake. This chapter diagnoses the main failure signals around a running transfer—slow progress, rate limits, denied requests, expired links, and non-media responses—and explains the feedback mechanisms that respond to them. The practical goal is not maximum speed at any cost; it is a bounded choice among continuing, reducing pressure, validating the result, asking for new context, or stopping.

A stalled-looking download is an observation, not a diagnosis. A host may be slow but healthy, refusing too many parallel connections, returning an access page instead of media, or serving a link whose token has expired. OmniGet's direct-transfer and adaptive-concurrency mechanisms are useful because they separate those possibilities instead of treating every failure as “try again.”

Symptoms point to different mechanisms

A slow transfer that continues to advance is different from a host that responds with HTTP 429. In the first case, the current number of connections may be acceptable but the path is simply limited by distance, server capacity, or local conditions. More parallel segments might help, but they might also add contention. In the second case, the host has supplied a stronger signal: it is asking the client to reduce pressure or wait.

The direct downloader keeps throughput history per host rather than treating the whole internet as one machine. That distinction matters because the useful concurrency for one CDN or platform can be harmful for another. A host that performs well with several segments may rate-limit at the same setting later, while a different host may tolerate more. Per-host history does not predict the future perfectly, but it avoids applying one source's behavior to an unrelated source.

OmniGet also keeps a global connection budget shared among active downloads. Even if each host appears tolerant on its own, several simultaneous jobs can overload the local machine or the network path. The per-download ceiling is reduced when more downloads are active, while the design keeps a lower bound so that a crowded queue does not make every item stop completely. This is a coordination rule, not a benchmark claim that the chosen ceiling is optimal everywhere.

A 403 is handled differently from an ordinary fatal status in the direct path. The implementation gives it a small, bounded header ladder: it can retry with a range request and then with additional browser-like fetch hints. The reason is narrow. Some CDNs reject a request that does not resemble a media-element request, then accept the same resource when the request shape changes. The ladder has a fixed end; after its steps are used, the 403 is fatal again. It is a limited interpretation of one signal, not a general invitation to evade access controls.

A 401, 404, 410, or other selected permanent response has a different meaning. It may indicate missing authorization, a resource that no longer exists, or a link that should not be retried automatically. The user may need to pair the correct browser account, obtain a fresh link, or accept that the resource is unavailable.