Beginner guide to work with multiple Loras.

1. Abstract

Generating images with multiple Loras can be confusing for beginners who just started stable diffusion,
2 images were generated with different methods in this article, aiming to showcase my workflow as a
reference to achieve this goal for beginners. Basically inpainting images with or without ControlNet.
These 2 methods are not the only ways, but I found these 2 methods the most reliable and stable.

2. Introduction
There are various types of Loras out there, lighting, styles, characters and poses etc.; it can be difficult
to combine them all in the way you wanted. On the other hand stable diffusion is notoriously known for
bad hands, fused body and weird anatomy, the methods below helps reduce the random variables from
generation.

3. Methodology
There are 2 methods to achieve the goal, 1 with ControlNet and 1 without it.
3.1. Without ControlNet
If you have not learnt what ControlNet is or you had trouble installing it, this method is for you. All
you need is stable diffusion and the Loras you wish to work with.
3.1.1.Take one of my works as an example:

This is an image generated based on a major Lora, [Hiding From Terminator Meme | Concept
LoRA] published by @FallenIncursio (a very creative creator). This Lora provides the major
composition of the image, I used 3 other Loras in this image, if i were to mix them all in txt2img,
they would become total chaos (Fig.1). It is possible to use regional prompt, but it usually doesn't
work for me; I inpaint them separately instead. It is easier to work with the general composition
first, such as characters’ placement, camera angles and postures etc. Because these parameters
require drastic changes; if you try to fix it in the later stage you might overwrite some parts of the
image that you desired.
 Fig.1
(upper 2) Total chaos generated with 4 Loras at full weight.

Prompt: <lora:GirlHidingFromTerminator:1> HidingFromTerminator, sitting, under

table, crying,, <lora:mikasaAckermanLora_offset:1> mikasa ackerman,
<lora:colossalTitanAttackOn_v01:1> colossal titan, <lora:GlowingRedEyeMeme:1>
Glowing Eyes Meme, (Glowing Eyes Meme:1.2), glowing eyes, glowing, epic

(bottom 2) Undesired composition generated with 3 Loras at full weight.

Prompt: <lora:GirlHidingFromTerminator:1> HidingFromTerminator, sitting, under

table, crying,, <lora:mikasaAckermanLora_offset:1> mikasa ackerman,
<lora:colossalTitanAttackOn_v01:1> colossal titan,

3.1.2.In order to change the characters in this Lora, I generated a batch of 4 images with this Lora
alone(*Fig.2 at the end of part 3.1 for further information) and I picked the one closest to my
desired composition. It doesn't matter if the characters generated do not look like the one you
wanted, we will inpaint them separately later. The most important part is to get the right
composition.
 Prompt: <lora:GirlHidingFromTerminator:0.85> HidingFromTerminator, sitting, under table, crying,
skeleton,
3.1.3.Once we got it, we send this image to inpaint, the girl hiding under table is the first to be adjusted.
(For demonstration purpose and consistency testing, I had inpainted it differently.) Mark the
character you wish to change, adjust the width, height and denosing strength.

Prompt: sitting, under table, crying, <lora:mikasaAckermanLora_offset:0.9> mikasa ackerman, hands

holding knees, brown jacket, white pants, red scarf, black hair

3.1.4.Afterwards, we need to change the terminator(the skeleton behind). I had tuned down the weight
for kermit the green frog; full weight of this kermit Lora will generate a normal doll size kermit
which is not what we want.
 <lora:Kermit:0.3> Kermit, big and tall green doll, indoor office, standing behind table, window
<lora:muscularGirlsAndBoys_v10:1.2> ((muscular))

3.1.5.There we have it, we have mixed 4 Loras together without controlNet, from complete chaos to a
controlled image.
 Fig.2 Example of mixing 2 characters in txt2img stage with light weight (0.2-0.5) first before
sending it to inpaint. It is not necessary, but it matches the art style of the background if none of the
Loras interfere with it.

Think Mark Think Meme (Lycoris) published by PANyZHAL, another great creator.

3.2. With ControlNet

If you have worked with ControlNet already, you can save some time or work with more complex
images with Depth or Canny. The idea is to generate a map for stable diffusion to follow, it is a lot
easier to mix Loras with controlNet, especially for multiple Loras that will interfere with a
character's posture or involve more than 1 character. (For those who wish to learn more, please
visit GitHub - lllyasviel/ControlNet: Let us control diffusion models! for more information.)
3.2.1.We will use another Lora as an example in this part, [Slapping MEME 啪啪啪lora] published by
@romiolast, yet another amazing creator. Let's say we wanted to change the characters of this
Lora. The idea is similar, this Lora provides the major composition of the image so we are going to
work with this Lora first. The reason we are using controlNet for this lora is because the slapping
hand can easily become the hand of the person being slapped or simply disappeared. Since the
background is not specified, it’s better to generate the image with characters in low weight first.
 <lora:耳光slapping:1> slapping,smith, 2 females <lora:NamiV2:0.2> NamiV2,
<lora:onePieceWanoSagaStyle_v2Offset:0.2> wanostyle, luffy

Once again, It doesn't matter if the characters generated do not look like the one you wanted, we
will inpaint them separately later. The most important part is to get the right composition; after
generating about 4-6 images, I got the one with correct anatomy and some desired features.

3.2.2.After that we sent this image to inpaint, we place the same image into controlNet, enable depth in
controlNet, press allow preview and the red button between the Preprocessor and Model to see
which depths work better(I use depth_midas this time because the slapping hand is clearly
distinguished from the person being slapped). This preprocessor will be used as a guide for stable
diffusion to inpaint the character.

3.2.3.Inpaint the slapping person, Tune down the Control Weight in controlNet to about 0.5-0.7 because
it’s quite different to the person we wanted; also tune down the slapping Lora to reduce the
interference to the slapping person’s appearance.
 <lora:耳光slapping:0.3> slapping,smith, 2 females <lora:NamiV2:0.85> NamiV2, long orange hair,
white and green strips bikini, blue jeans

(I’m sorry I had deleted some of the img2img records to free up disk spaces and I can’t be
bothered to generate again from scratch, that’s why the person getting slapped changed too even
though it’s not masked.)

3.2.4.Do the same for the other character.

<lora:耳光slapping:0.3> slapping,smith, 2 females, <lora:onePieceWanoSagaStyle_v2Offset:0.6>

wanostyle, luffy, female, open long sleeve red shirt, big breast, cleavage, navel

3.2.5.Add some facial expression to the person being slapped

 <lora:onePieceWanoSagaStyle_v2Offset:0.6> wanostyle, luffy, female, open long sleeve red shirt,
big breast, cleavage, navel ((from side)) <lora:scaredExpression_v157:1> scared expression,
clenched teeth, open mouth

3.2.6.There we have it, we have mixed 4 Loras together with controlNet(depth).

4. Conclusion
It can be tricky to generate images with multiple characters and complex group poses, but with inpaint
and controlNet, the random variables can be limited and you can achieve the image you have in mind
ultimately. Do note that these methods require some trial and error, adjusting the weight of each loras
accordingly is the key to success.

For the image demonstrated in (3.2), I had enlarged the image with tiled diffusion before I sent it to
inpaint, because the canvas was too small for stable diffusion to put details into the image. For more
information, please visit Tutorial for multidiffusion upscaler for automatic1111 published by
Potatovision, a very helpful creator who helped me alot.
 It is difficult to write down all the details down in text, such as denosing strength, canvas size and
inpaint size; I did not include these parameters in this article because I think these are a bit out of the
scope, there should be tutorials about this out there in Civitai.

Finally, this is just the way I generate images, please feel free to discuss and ask questions regarding
the methodology. I hope this guide helped.

5. Reference

Lora list (not mentioned):

Mikasa Ackerman LoRA Lykon

Colossal Titan - Attack on Titan psoft
Glowing Red Eyes Meme | 1 MB Concept LoRA FallenIncursio
Kermit the Frog djo226
Muscular girls (and boys, probably) lin_obscure
Nami (ナミ) One Piece Character LoRA (Post-timeskip) 12user34kn276
One Piece (Wano Saga) Style LoRA Lykon
scared expression WindFlare