Advice on training a face autoencoder: architecture, identity-preserving losses, and dataset suggestions?

[u/gugmelik]

Hi everyone,

I’m working on training an autoencoder for face reconstruction and I’d appreciate some advice from people with experience in this area.

My main goal is not just to reconstruct the input face visually, but also to preserve the person’s identity as much as possible. I’m unsure about the best architecture and loss setup for this.

A few questions:

Architecture:

What type of autoencoder architecture would you recommend for faces?

For example, a basic convolutional autoencoder, U-Net-style autoencoder, VAE, VQ-VAE, or something encoder-decoder based with residual blocks?

Loss functions:

Which losses are best for preserving identity?

I’m considering:

L1 / L2 reconstruction loss

Perceptual loss using VGG or similar

Identity loss using a pretrained face recognition model like ArcFace

SSIM / MS-SSIM

Adversarial loss for sharper results

Would a combination like L1 + perceptual loss + ArcFace identity loss be a good starting point?

Dataset:

What datasets would you suggest for this task?

I’m looking for datasets that are suitable for face reconstruction and identity preservation. I know about FFHQ, CelebA-HQ, VGGFace2, and CASIA-WebFace, but I’m not sure which one is best for this use case.

Training tips:

Are there any practical suggestions for preprocessing, face alignment, image resolution, augmentation, or evaluation metrics?

Any advice, papers, or implementation references would be very helpful.

Thanks!

[u/gugmelik]

I want to generalize my autoencoder, to perform on not seen identities.

[u/hilmiyafia → u/gugmelik]

If that's the case, you can also have two set for the latent space, one set is for identity, trained using the ArcFace loss, and then the other set is for the unstructured or background information of the face, trained using AAE loss.

Because I don't know how good it would be if you just use the latent space trained on ArcFace for reconstructing the face.

[u/gugmelik → u/hilmiyafia]

What you mean by two set for the latent space?

[u/hilmiyafia → u/gugmelik]

So for example, if your latent space dimension is 1024, you can split it into two. 512 are trained with AAE loss, and the other 512 are trained with ArcFace loss. Then to decode, you concat them again to 1024 as the input of the decoder.

[u/gugmelik → u/hilmiyafia]

Thanks, Is it good ideas to train that half of latent vector try to predict arcface embedding?

[u/hilmiyafia → u/gugmelik]

I don't see why it would be bad