Models Become Replaceable Capabilities

Dify treats a model choice as a runtime capability, not as a permanent property of a workflow. The specific problem is how the platform resolves a tenant's provider configuration, credentials, model schema, quota policy, and transport without making the graph itself provider-specific. Following that boundary helps the reader distinguish what a workflow asks for from what the surrounding system must authorize and operate.

Configuration belongs to the tenant; invocation belongs to the run

A workflow node can say that it needs a language model, an embedding model, or a reranker. That statement is not enough to make an invocation safe or possible. Dify first has to assemble the provider configuration available to the relevant tenant. Its provider manager combines database records with provider schemas supplied through the plugin system, then accounts for system-hosted and custom configurations, preferred provider type, model settings, credentials, quotas, and optional load-balancing entries.

This division matters because a provider is not the same thing as a model. A provider identifies a family of integrations and supplies a schema for its models. A model is one selectable capability inside that provider. Credentials may belong to the provider or to a particular model, and the active credential can vary by tenant. A model schema describes what that model can accept and return, including capabilities that affect how a node or retrieval planner may use it. Keeping the concepts separate allows a workflow to request a capability while the tenant decides which concrete configuration is available.

Credential handling is part of the boundary, not an afterthought. Dify supports a configurable key-provider mechanism for protecting tenant secrets, with local per-tenant key material or an Azure Key Vault path described in the current configuration. Provider configuration rows can be cached, but runtime-aware objects and decrypted values remain scoped to a manager or request path. That distinction prevents a cache of stable database information from being mistaken for a universal, process-wide model session. It also makes credential rotation a real operational concern: a long-lived cache can become stale, which is why the model manager documents bounded-scope use for credential caching.

When a workflow resolves a model, ModelManager asks the tenant-bound provider manager for a provider/model bundle and returns a ModelInstance. The instance carries the selected provider and model identity, credentials, model-type implementation, invocation parameters, and any load-balancing manager. For system-hosted models, Dify can use a quota-managed instance. That instance reserves usage for an invocation and settles or releases it according to the outcome, including the different timing needs of streaming calls. For custom configurations, the same high-level model interface can be used without pretending that hosted quota policy applies in the same way.

The support-assistant example makes the value concrete. The workflow can keep its “explain policy” step while an administrator changes the active model or credential configuration for the tenant. The workflow's logic has not changed, but its request now carries a different capability. If the new model lacks a needed feature, has no valid credentials, is disallowed by a hosted-model policy, or has exhausted its quota, the failure belongs to capability resolution rather than to graph control flow.

This assembly is also why Dify's model management is more than a dropdown. The provider manager chooses whether the effective configuration is system or custom, checks whether a quota remains usable, builds model settings, and can attach multiple credential configurations for load balancing. It must reconcile product policy with runtime availability. A model that appears in a provider schema is not automatically a model that the tenant may invoke.

Configuration assembly has a temporal dimension as well as a permission dimension. Stable provider records can be reused from a Redis-backed cache, but the assembled object still has to bind a fresh, request-aware runtime. That is a sensible compromise: repeated requests avoid rebuilding every database-shaped record, while the model call retains tenant and caller context. It also explains why an operator should not assume that changing an API key is instantly visible to every long-lived process or cache. The source documents the default model-manager path as not retaining a credential cache across unrelated work; when a cache is deliberately enabled, its scope and invalidation become part of the deployment's correctness story.

The same distinction helps with model migrations. Replacing a provider is not just changing a string in a workflow. The new provider must expose a compatible schema, the tenant must have valid credentials, the relevant model type must be enabled, and any downstream assumptions—such as tool calling, token counting, or output parsing—must still hold. Dify can keep the orchestration layer stable while these checks move to configuration and capability resolution. That is the architectural benefit, but it also means migration testing belongs at the boundary where the model is actually used.

The boundary between stored configuration and a live call can be summarized like this:

classDiagram class TenantConfiguration class ProviderManager class ModelManager class ModelInstance class PluginModelRuntime class PluginDaemon class LLM class Embedding class Reranker class ModerationSpeech TenantConfiguration --> ProviderManager : supplies policy and credentials ProviderManager --> ModelManager : resolves tenant capability ModelManager --> ModelInstance : creates request instance ModelInstance --> PluginModelRuntime : invokes through adapter PluginModelRuntime --> PluginDaemon : dispatches provider call ModelInstance --> LLM : exposes ModelInstance --> Embedding : exposes ModelInstance --> Reranker : exposes ModelInstance --> ModerationSpeech : exposes

The diagram is intentionally layered. Tenant configuration owns the decision about what is available. The model manager turns that decision into an instance for a bounded use. The runtime adapter carries the request to the plugin daemon. The final model-type interfaces are not separate application architectures; they are different contracts on the capability side of the same execution path.

One runtime boundary serves several model types

Dify's model abstraction includes more than text generation. An LLM can generate or stream language, but an embedding model turns text or multimodal material into vectors for indexing and query, a reranker changes the order of retrieved candidates, a moderation model evaluates content, and speech or text-to-speech models connect other input and output forms. The application can therefore use different model types at different stages without putting every provider's SDK directly into workflow nodes.

That separation is especially important for retrieval. A knowledge pipeline may use an embedding model when documents are indexed, the same or a compatible embedding model when a query is searched, and a reranking model after candidate retrieval. Those calls are related, but they are not one generic “LLM call.” They have different inputs, results, credentials, costs, and failure modes. Dify's model manager exposes them as distinct model-type operations while the surrounding retrieval code decides when each operation is needed.

The plugin runtime makes the provider-specific crossing explicit. Dify discovers model providers through the plugin service and binds a runtime to the tenant and optional caller scope. When it needs a schema, the runtime can request that schema from the plugin daemon and cache the result with tenant, provider, model, caller, and credential-sensitive identity. When it invokes an LLM, the client sends the provider name, model, credentials, model parameters, prompt messages, optional tools, stop settings, and stream choice through a daemon dispatch request. A streaming response remains a stream at the adapter boundary instead of being silently collapsed into one string.

This design gives the graph a stable interface while preserving a real transport boundary. The workflow node does not import a vendor SDK, but the request still depends on a plugin daemon, a plugin package, a provider implementation, valid credentials, and protocol-compatible data. A plugin can expose schemas and invocation methods for embeddings, multimodal embeddings, reranking, speech, moderation, or text-to-speech as well. The same idea extends capability without forcing the core workflow layer to know every vendor-specific detail.

The boundary also carries application context. Dify can attach tenant, application, user, and app-type metadata to model requests. That context supports usage accounting and helps downstream services attribute a call to the correct application. It is different from the prompt itself: a model may see a user question, while the runtime needs identity and policy data that should not be left for the model to infer.

Structured output illustrates the same pattern. The plugin runtime can obtain a model schema, then use a Dify-side helper to request structured output against that schema. The adapter binds provider, model, and credentials to the lower-level runtime without constructing a second, unrelated model-management path. The result is still subject to parsing and validation; an advertised schema does not guarantee that a provider will always return usable structured data.

Dify also supports long-running model operations through a polling path for providers that expose it. The workflow's model wrapper starts a provider-side polling job, checks its state, and settles quota when a result succeeds or releases it when the operation does not. This is a different lifecycle from an ordinary streamed response, even though both are presented to the node through a model protocol. The point is not that every provider supports polling; the point is that the capability boundary has room for provider-specific execution modes without moving that complexity into the visual workflow.

Load balancing is policy under failure, not magic redundancy

When a tenant has multiple usable credentials for a model, Dify can select them through a round-robin load-balancing manager. A shared counter determines the next candidate, while temporary cooldown state removes a credential that has just produced a rate-limit, authorization, connection, or policy-compliance problem. The next request can try another candidate rather than repeatedly sending traffic to a known-bad path. If all candidates are unavailable, the invocation fails instead of pretending that redundancy exists.

This mechanism changes the meaning of a model call. The workflow asks for a provider and model, but the runtime may choose one of several credential configurations. A rate limit is therefore not necessarily a graph failure; it can be a credential-level event that changes the next selection. An authorization failure can be similarly isolated for a time. Yet load balancing cannot repair an invalid prompt, a model-wide outage, incompatible provider behavior, or a side effect that was already performed before a connection failed. It improves one failure boundary while leaving others intact.

Quota policy creates a second kind of constraint. A hosted model may be visible in configuration but unavailable because its current quota is exhausted or because the tenant's active quota does not permit that model. Dify's provider assembly chooses among available quota types and the quota-managed model instance accounts for individual calls. A team therefore needs to distinguish “the provider is configured,” “the model is permitted,” and “this invocation can be charged now.” Treating them as one boolean produces confusing operational diagnoses.

The abstraction also does not standardize model behavior. Two providers may both implement an LLM interface but differ in context limits, tool-calling support, structured-output reliability, token counting, latency, error semantics, or multimodal features. Dify can expose a common route to those capabilities, and a model schema can advertise relevant features, but the application still needs to design prompts, output checks, thresholds, and fallbacks around the actual capability. The interface reduces integration coupling; it does not erase provider variation.

For the support assistant, this means a model failure should be located before it is fixed. If the graph never reaches the LLM node, inspect input mapping and branch control. If the node is created but cannot resolve credentials, inspect tenant configuration and access policy. If the runtime reaches the plugin daemon but the provider rejects the call, inspect provider behavior, schema, and transport. If one credential is rate-limited while another works, load balancing may keep the application running, but the event and usage record should still make the degraded path visible.

Dify's model boundary therefore provides replaceability with conditions. It lets a workflow remain focused on business behavior while model types, providers, credentials, quotas, and plugin transports evolve around it. The resulting application is more adaptable than one that hard-codes a vendor into every node, but it is not independent of the capability plane. The model still determines what can be understood or produced, the provider still determines how the call behaves, and the surrounding runtime still has to account for policy and failure.

Once those distinctions are in place, the full request path becomes easier to reconstruct. A graph chooses when a capability is needed; tenant configuration determines which capability is admissible; a model manager materializes it; the plugin boundary performs the provider-specific work; and events record the outcome. The remaining question is what happens when the model needs knowledge beyond its prompt—and why the quality of that knowledge depends on decisions made before the request ever arrives.

References