When the Image Joins the Prompt: Editing and High-Resolution Control
This chapter examines how AUTOMATIC1111/stable-diffusion-webui changes the generation problem when a request includes an existing image, a spatial mask, or a second resolution pass. The practical question is what information enters the pipeline, what information remains protected, and what gets generated again. That distinction lets a reader choose an editing mode for a real goal without treating denoising, masking, or upscaling as magic switches with fixed visual results.
Text-to-image, image-to-image, and inpainting share the processing lifecycle described earlier, but they do not begin with the same state. Text-to-image starts with a seed-driven source of variation and asks the model to construct an image from conditioning. Image-to-image supplies an existing image that the runtime resizes and encodes into the model’s latent representation before sampling. Inpainting adds a mask that tells the runtime where generated content may replace or mix with the initial content. High-resolution fix adds yet another stage after a first result exists. These are different information flows, not merely different names for a prompt form.
Starting from noise, image, or mask
The simplest choice is text-to-image. The request provides text, a seed, dimensions, a sampler, and other processing values; the runtime uses those inputs to create a new latent sample. There is no initial picture whose composition it must preserve. The seed still does not determine the subject by itself: it identifies a path through variation inside the selected model and recipe. Change the checkpoint, prompt conditioning, sampler, dimensions, or relevant settings and the same seed can lead somewhere else.
Image-to-image adds an initial image before sampling. The runtime first makes the image fit the requested processing dimensions, then converts its pixels into a latent representation. Sampling begins from that encoded state rather than from an entirely unconstrained starting point. The result can retain broad structure from the input while allowing the model to reinterpret it. “Retain” is the right way to describe this, not “copy”: the repository defines the dataflow and the denoising control, but it does not promise a particular visual amount of preservation for every checkpoint.
Denoising strength is the main control for that tension. At one end, the initial latent has a stronger role in the result; at the other, the sampling process has more room to replace or reinterpret what it received. The control is not a percentage of pixels that will be preserved. It changes the conditions under which the sampler moves from the encoded input toward a new sample. A small change can therefore affect details, composition, or semantic reading in ways that are not linear. A value that works for a line drawing may not behave the same way for a textured photograph, because the input and the model present different information to the same pipeline.
This gives img2img a useful role in an editing workflow. If the goal is to keep a subject’s broad arrangement while changing style, the initial image supplies structure and the prompt supplies a new condition. If the goal is to explore substantially different compositions, text-to-image may be the cleaner starting point because it does not carry an earlier arrangement into the latent state. If the goal is to make a bounded repair, image-to-image alone is too broad: it gives the sampler an image but not a precise spatial boundary.
Inpainting adds that boundary. The interface accepts an image and a mask; the processing code converts the incoming mask into a binary form, can invert it, and can blur it separately along the horizontal and vertical directions. The resulting mask is resized to the working dimensions. When the full-resolution masked-area option is used, the runtime can find the masked region, expand it to fit the requested dimensions, process that crop, and remember where it belongs in the original image. The mask therefore controls both what is offered to the model and how the result is placed back into the larger picture.
The latent and visible-image paths have different responsibilities. The latent mask identifies where the generated latent may take effect and where the initial latent should remain. The runtime blends the sampled latent with the initial latent using those complementary regions. After decoding, it can use the saved mask and original image for an overlay or composite. A soft edge is not just a prettier outline: blur changes how abruptly the generated region meets the preserved region, while crop and padding change how much surrounding context the model sees. A mask that is technically inside the desired object can still produce a poor boundary if the model lacks enough neighboring context or if the blend makes the transition too abrupt.
The interface’s several image modes expose variations on this same principle. A sketch supplies a drawn starting image; a color-sketch mode derives a change mask by comparing an edited image with its original; an uploaded inpaint mask supplies the boundary separately; and batch mode repeats the process over multiple inputs. The names help users find the tools, but the reliable way to understand them is to ask what becomes the initial image, what becomes the mask, and when the original is reintroduced. That question remains valid even when an extension changes the UI.
The spatial constraint also explains why an inpaint result can look different outside the painted mark. The model receives a prepared crop or resized condition, not an isolated pixel patch with perfect awareness of the untouched image. The final composite may restore the original outside the mask, but the generated area was still sampled using the context made available at the preparation stage. A narrow mask can preserve more of the source at the output boundary while giving the model less room to understand the surrounding form; a wider contextual region can improve coherence while increasing the area that may be reinterpreted. The repository supports these mechanisms, not a universal best mask shape.
The diagram’s convergence is important. These modes ultimately return to decoding, compositing, post-processing, and persistence, so they can share metadata and output handling. Their differences arise earlier, when the runtime decides what starting information and spatial constraints the sampler will receive. A person choosing among them is choosing an information boundary.
Second passes are decisions, not neutral enlargement
A larger output can be produced in more than one way. A conventional upscaler can reconstruct or enlarge an already decoded image. AUTOMATIC1111’s high-resolution fix can instead take a first-pass result, scale it in latent space or through an image upscaler, and run a second sampling pass. The second pass has its own denoising behavior and can use a different sampler, scheduler, checkpoint, or prompt. It is therefore another generative decision attached to the first one.
The first pass and second pass solve different problems. The first pass establishes a composition at a manageable working resolution. The second pass gives the model a larger representation and an opportunity to add or revise details. If the second-pass denoising is too weak for the intended change, the result may remain close to the first pass while adding limited new structure. If it is stronger, the model has more freedom to reinterpret edges, textures, faces, or small objects. The repository exposes this as a configurable second stage; it does not establish which setting preserves identity best for every image.
The choice of scaling path matters. In a latent upscale path, the runtime interpolates the internal representation and then samples from it. In a pixel-oriented path, it decodes the first result, resizes the visible image with an upscaler, encodes that image again, and uses the resulting latent for the second pass. Those paths place different transformations between the two sampling stages. A reader does not need to memorize their implementation to make a sound judgment: ask whether the intermediate object is an internal latent or a decoded image, and then record the selected path as part of the recipe.
High-resolution fix can also change model identity. The second pass may use the same checkpoint, but the controls allow a different checkpoint or a refiner-style transition. It may use a different sampling method or schedule and can accept separate high-resolution positive and negative prompts. This creates a useful opportunity when the first pass and final detail need different treatment, but it also increases coupling. A change in the final image may come from the new checkpoint, the new sampler, the second prompt, the scaling method, or the denoising strength rather than from “more pixels.”
That is why high-resolution outputs need their own generation information. The runtime adds high-resolution settings such as the upscale or resize target, denoising strength, second-pass steps, upscaler, checkpoint, sampler, scheduler, and optional second-pass prompts to the extra generation parameters. When the output is saved with metadata, those choices help the reader distinguish a first-pass image from a refined image. Without them, a large picture can look like a single operation even though it passed through two model-facing stages.
Restoration and neural upscaling sit at a different boundary. Face restoration operates on the generated image after decoding, and the Extras path can apply image enhancement independently to an existing file. Neural upscalers such as those listed by the project use their own model artifacts and can be selected as post-processing tools. They may improve a local visual defect or add apparent detail, but their output is not evidence about what the base diffusion checkpoint produced. Treating the result as a layered artifact makes it possible to preserve or inspect pre-restoration and pre-high-resolution versions when the relevant options are enabled.
The distinction is practical. Suppose a portrait has a good pose but a weak face. Image-to-image can reinterpret the whole image, inpainting can target the face region, high-resolution fix can regenerate detail across the image, and face restoration can modify the decoded face afterward. These interventions do not carry the same risk of changing the pose, background, or identity. The repository supports all of them as different stages, but it cannot choose the correct one without a goal. “Make it better” is not a sufficient technical specification; “change the face while preserving the pose and background” points toward a spatially bounded intervention, subject to the model and mask behavior.
The same reasoning applies to a small object in a scene. A prompt change affects the conditioning for the whole request. Image-to-image changes the starting latent for the whole image. Inpainting gives a local region a different treatment and uses compositing to preserve the rest. High-resolution fix may revisit every region during its second pass. A neural upscaler works after diffusion and may sharpen edges without understanding the original semantic goal in the same way. Choosing the least disruptive stage is not a guarantee of success, but it is a more defensible starting decision than selecting the most powerful-looking control.
Every added stage has a cost beyond compute time. It can introduce another model file, another compatibility condition, another source of visual change, and another piece of provenance that must be retained. A high-resolution pass can require a different memory profile from the first pass. An inpaint crop can create a boundary that needs inspection. A face-restoration model can solve one artifact while changing a feature the user wanted to keep. A workflow that saves only the final image loses the evidence needed to tell these causes apart.
A disciplined editing request therefore has three parts. First, state the protected information: perhaps the pose, the composition, the unmasked background, or the broad shape of an existing object. Second, state the information allowed to change: style, face detail, a masked region, or resolution-dependent texture. Third, choose the stage that matches that boundary and record the relevant parameters. The choice remains empirical; the repository’s code defines how information flows, not what an individual checkpoint will invent or preserve.
This is the larger lesson of editing in AUTOMATIC1111. The prompt is only one source of control. An initial image supplies a latent starting state, a mask supplies a spatial permission structure, a high-resolution pass supplies another round of sampling, and restoration or upscaling supplies a post-generation transformation. When the image changes, the reader should ask which boundary admitted the change. That question turns a confusing collection of tabs into a traceable set of interventions and prepares the next judgment: how model identity, memory, precision, and hardware constrain the choices that are available in the first place.