Human reposing involves changing the orientation of a
source image to a desired target pose. To get accurate results, we
learn to preserve the region visible (green) in the source image and
transfer the appropriate style to the invisible region (red)
Abstract
The task of human reposing involves generating a realistic image of a person standing in an arbitrary conceivable pose. There are multiple difficulties in generating perceptually accurate images, and existing methods suffer from limitations in preserving texture, maintaining pattern coherence, respecting cloth boundaries, handling occlusions, manipulating skin generation, etc. These difficulties are further exacerbated by the fact that the possible space of pose orientation for humans is large and variable, the nature of clothing items is highly non-rigid, and the diversity in body shape differs largely among the population. To alleviate these difficulties and synthesize perceptually accurate images, we propose VGFlow. Our model uses a visibilityguided flow module to disentangle the flow into visible and invisible parts of the target for simultaneous texture preservation and style manipulation. Furthermore, to tackle distinct body shapes and avoid network artifacts, we also incorporate a self-supervised patch-wise ”realness” loss to improve the output. VGFlow achieves state-of-the-art results as observed qualitatively and quantitatively on different image quality metrics (SSIM, LPIPS, FID).
Network architecture
VGFlow FlowVis module takes in concatenated Is, Ks & Kt as input. We predict two flow pyramids at multiple scales for different
visibility regions using a Unet architecture. Subsequently, these flow pyramids are combined using Gated Aggregation and upsampled
via Convex upsampling. Losses L1, Lvgg, are imposed on the visible areas and Lvgg , Lsty are imposed on the invisible areas.
Results
We underscore improvements along different qualitative aspects over previous works. We emphasize enhancement
in preserving pattern & segmentation (a), handling complex poses (b), design consistency (c), conserving text readability (d), reducing
bleeding, skin generation & identity reproduction (e), maintaining geometric integrity (f) and pattern coherence (g). (Best viewed in zoom)
Resources
- Paper: CVPR
- Supplementary: CVPR_supplementary
- Video: YouTube
- Results for comparison: Zip file
Citation
@inproceedings{jain2023vgflow,
title={VGFlow: Visibility guided Flow Network for Human Reposing},
author={Jain, Rishabh and Singh, Krishna Kumar and Hemani, Mayur and Lu, Jingwan and Sarkar, Mausoom and Ceylan, Duygu and Krishnamurthy, Balaji},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={21088--21097},
year={2023}}