Divide and control: generation of multiple component comic illustrations with diffusion models based on regression

Zixuan Wang
,
Peng Du
,
Zhenghui Xu
,
Qihan Hu
,
Hao Zeng
,
Youbing Zhao

Communication University of Zhejiang
,
Uber Technologies, Inc.

Research Output: Chapter in Book/Report/Conference proceeding Conference contribution Peer-review

Abstract

Diffusion-based text-to-image generation has achieved huge success in creative image generation and editing applications. However, when applied to comic illustrations, it still struggles to deliver predictable high-quality productions with multiple characters due to the interference of the text prompts. In this paper, we propose a practicable method to use ControlNet and stable diffusion to generate controllable outputs of multiple components. The method first generates images for individual components separately and then degenerates those images to a regressed form, such as line drawings or Canny edges. Those regressed forms of individual components are then merged and fed into ControlNet to generate the final image. Experiments show that this method is highly controllable and can produce high-quality comic illustrations with multiple components.

Publication Information

Output type

Research Output: Chapter in Book/Report/Conference proceeding Conference contribution Peer-review

Original language

English

Pages from-to (Number of pages)

Pages 59-69 (11 pages)

Publication milestones

Published - 02/11/2023

Publication status

Published - 02/11/2023

Edition

1946

Publisher

Springer, Japan, India, Australia, Germany, United States, United Arab Emirates, Austria, Switzerland, Italy, China, United Kingdom, Netherlands, Brazil, France, Singapore