Download: headshotv2.safetensors

Use this lora with a stable diffusion XL checkpoint

This is a proof of concept for training an SDXL lora on SD1.5 knowledge. Trained on ‘headshot of person’ where person is a wildcard.

This is also available on https://ntcai.xyz/sdxl/headshot

Quick note

I just setup a patreon at patreon.com/NTCAI if you’d like to support my work or vote on what to work on next. There’s also an nft drop planned for supporters but it won’t have financial value, just having fun.

This process is

Use img2img on SDXL output using SD1.5. I wrote a tutorial on this here https://ntcai.xyz/articles/applying_stable_diffusion_15_models_loras_controlnet_to_sdxl_with_comfyui_and_img2img/ Create a large dataset with this technique - this is trained on 800 images Once you have this

Choose a subject matter that sd1.5 knows well and perhaps reject any distorted images. I did not do that because this is just an example.

Get your files in the correct form. This tutorial helped me

Train SDXL using https://github.com/kohya-ss/sd-scripts on the generated images. My command is below

CUDA_VISIBLE_DEVICES=0 accelerate launch --num_cpu_threads_per_process=2 "sdxl_train_network.py" --enable_bucket --pretrained_model_name_or_path="/ml2/trained/ComfyUI/models/checkpoints/stable-diffusion-xl-base-1.0/sd_xl_base_1.0.safetensors" --train_data_dir="/ml2/trained/sd-scripts/data/aniverse" --resolution="1024,1024" --output_dir="/ml2/trained/ComfyUI/models/loras/Lora/sdxl" --logging_dir="./logs" --save_model_as=safetensors --output_name="aniverse2" --network_alpha="1" --network_module=networks.lora --text_encoder_lr=0.0004 --unet_lr=0.0004 --network_dim=256 --lr_scheduler="constant" --train_batch_size="2" --max_train_steps="8000" --mixed_precision="bf16" --save_every_n_epochs="1" --save_precision="bf16" --caption_extension=".txt" --optimizer_type="Adafactor" --optimizer_args scale_parameter=False relative_step=False warmup_init=False --max_data_loader_n_workers="0" --bucket_reso_steps=64 --gradient_checkpointing --xformers --bucket_no_upscale

My hope is that this can help creators migrate their work. Happy training!

Resources used

Other NTC creations