Cheat sheets

ControlNet, Inpainting and Image Editing

Diffusers calls for text to image, img2img, inpainting, LoRA and ControlNet. Knobs, defaults and fixes.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Control signals into the U-Net#

featuresadded to decodertext tokensimage tokenslow-rank deltaclientCondition imagedepth, pose, edgesmodelControlNetcloned encoder, trainableprocessZero conv1×1, starts at zeroclientPromptmodelText encoderCLIP-L or T5gpuFrozen SD U-Netcross-attention to textclientReference imagemodelIP-AdapterCLIP image tokensstoreLoRAW + BA in attentionfeaturesadded to decodertext tokensimage tokenslow-rank deltaclientConditionimagedepth,pose,edgesmodelControlNetclonedencoder,trainableprocessZeroconv1×1,starts atzeroclientPromptmodelTextencoderCLIP-L orT5gpuFrozenSD U-Netcross-attentionto textclientReferenceimagemodelIP-AdapterCLIPimagetokensstoreLoRAW + BA inattention

Text to image#

pipe = StableDiffusionPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5",
torch_dtype=torch.float16,halves VRAM
).to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
image = pipe(
prompt="<prompt>",
guidance_scale=7.5,CFG scale w, SD default
num_inference_steps=25,matches 50 steps of DDIM
generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("dog.png")

Img2img, inpaint and LoRA#

out = img2img(prompt="<prompt>", image=init_image,
strength=0.6, guidance_scale=7.5).images[0]0.0 unchanged, 1.0 new
mask = Image.open("dog_mask.png").convert("L").resize((512, 512))
out = inpaint(prompt="<prompt>", image=image,
mask_image=mask, guidance_scale=7.5).images[0]white pixels regenerate
pipe.load_lora_weights("<repo>", weight_name="<file>")
pipe.fuse_lora(lora_scale=0.8)bakes in, no swap
pipe.unfuse_lora()before another adapter
pipe.set_adapters([...], adapter_weights=[...])per-request LoRA mix

Tasks in code#

TaskCode
Install the stackpip install diffusers transformers accelerate
Image to imageStableDiffusionImg2ImgPipeline
Inpaint with SDStableDiffusionInpaintPipeline
Inpaint with FluxFluxFillPipeline
Stochastic samplerEulerAncestralDiscreteScheduler
Encode to latentspipe.image_processor.preprocess(img)
Reuse old latentspass latents=... and skip one VAE encode
Scale VAE latentslatent_dist.sample() * 0.18215

Mask-aware reverse loop#

1Startdraw x from N(0, I) for every dimension.2Reinjectnoise the clean source to step t. Write it intoeach unmasked dimension.3Predictthe denoiser gives eps_hat for x at step t.4Reversecompute the DDPM mean. Add noise when t > 0.5Repeatgo from t = T - 1 down to 0.6Pincopy the clean values into the unmaskeddimensions at the end.7Outpaintinvert the mask. The training objective staysthe same.

Conditioning tools side by side#

FactorControlNetLoRAIP-AdapterDreamBooth
Controlsspatial structurestyle, subject, conceptlook of a reference imageone subject
Size70-360MB20-200MB20MB2-5GB
Mechanismcloned encoder, zero-conv skipslow-rank delta on attentionCLIP image tokens in cross-attentionfull fine-tune
Use whenexact layout or compositionpersonalization or styleno text describes the lookstrong identity, high compute

Formulas and numbers#

CFG
\epsilon = \epsilon_{\text{uncond}} + w \cdot (\epsilon_{\text{cond}} - \epsilon_{\text{uncond}})
LoRA
W' = W + \alpha \cdot B A with r \ll d. New parameters: 2 \cdot d \cdot r instead of d^2.
LoRA size
d=640, r=16 gives 20k parameters instead of 410k.
ControlNet mix
features += weight_a * control_a(depth) + weight_b * control_b(pose)
Latent diffusion
3x512x512 to 4x64x64 cuts compute by 48x.
Inpaint U-Net
9 input channels. Noisy latent 4, encoded image 4, mask 1.
SDEdit t/T
0.3 nearly the source. 0.6 keeps coarse structure. 0.9 near noise.
Zero conv
ControlNet starts as identity. It does no harm before training.

Editing methods side by side#

FactorInpaint modelSDEdit or img2imgInstructPix2PixRePaint
Inputimage, mask, promptimage, noise level, promptimage and instructionimage and mask
Training9-channel U-Netnonefine-tune on triplesnone, unconditional model
Keepspixels outside the maskcoarse structurestructure of the inputpixels outside the mask
Main knobmask and CFGt/T or strengthimage scale and text scaleresample steps
Use forremove object, replace skystyle of one region"make it sunset"no inpaint model at hand

Defaults#

SettingDefaultNote
guidance_scale7.5SD default CFG scale
num_inference_steps25With DPM-Solver++
img2img strength0.60.5-0.7 for style transfer
Prompt drop in CFG training10%Gives the unconditional branch
VAE scale factor0.18215SDXL uses 0.13025
LoRA rank r4-1664-128 for heavy fine-tunes
Sum of ControlNet weights1.0Safe default
Mask dilation in pixels8-16Hides seams

Troubleshooting#

SymptomCauseFix
Over-stylized or broken outputLoRA α at 2 or 3Keep α at 1.5 or below
Pose plus depth overshootsTwo ControlNets at weight 1.0Make the weights sum to 1.0
LoRA has no effectSDXL LoRA on SD 1.5Match the base model
Cannot load another LoRAfuse_lora baked it inCall pipe.unfuse_lora() first
Visible seams at the maskNaive reinjectionDilate the mask or use a 9-channel model
Saturated patchHigh CFG on a small maskReduce CFG for small edits
Subject loses identitySDEdit t/T from 0.5 to 0.6Sweep and checkpoint
Fill ignores the scenePrompt names only the new contentDescribe the whole image

Use X when#

  • ControlNet-Openpose + SDXL + text a specific pose with a prompt.
  • ControlNet-Scribble or ControlNet-Canny an exact layout.
  • LoRA on ~30 curated images at rank 32 a brand art style.
  • SD-Inpaint or Flux-Fill remove an object under a small mask.
  • SAM mask, then SD-Inpaint background replacement.
  • InstructPix2Pix or Flux-Kontext an instruction edit with no mask.
  • LCM-LoRA latency under 1s.

Questions#

Why init the ControlNet gate at zero?
At step 0 the output is identical to the base.
Merge LoRAs or hot-swap them?
Hot-swap. A merge is 3-5% faster but freezes α and the base.
Why does strength 1.0 ignore the input?
It adds full noise. The result is a full regeneration.
Why use a 9-channel model, not reinjection?
The model sees the context around the mask.
Can I swap the scheduler after training?
Yes. Scheduler state is decoupled from U-Net weights.
What does an extra ControlNet cost?
Budget about 1.5× step cost for each active ControlNet.