Reproduce “Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shih-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard B W Yang, Giyeong Oh, Yanmin Gong
LyCORIS provides a library of parameter-efficient fine-tuning methods and a framework for comparing their effects when adapting Stable Diffusion to concepts. It compares LoRA, LoHa, LoKr, and native fine-tuning across prompts, metrics, and concept categories. LoHa ranks second in fidelity but fourth in controllability and diversity, while LoKr ranks fourth, first, and second; these tentative trends are not universal winners.
Key insights
- Fine-tuning evaluation is complex due to hyperparameter sensitivity, concept-specific performance variations, and inherent trade-offs between evaluation criteria like fidelity and diversity.
- Increasing training epochs and learning rates generally improves concept fidelity but often reduces text-image alignment, image diversity, and base model preservation, with overtraining observed for smaller datasets.
- No single fine-tuning algorithm (LoRA, LoHa, LoKr) is universally optimal; their effectiveness depends on the specific customization task, desired balance of metrics, and the complexity of the concepts being learned.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.