EIFER
Electromyography-Informed Facial Expression Reconstruction For Physiological-Based Synthesis and Analysis

Tim Büchner1, Christoph Anders2, Orlando Guntinas-Lichius2, Joachim Denzler1
1Computer Vision Group 2Jena University Hospital
Friedrich Schiller University Jena, Germany
🎉
CVPR 2025 🎉
arXiv Preprint Supplementary Material Get Updates
Code (Soon) Models (Soon) Data Sample (Soon) Data Set (Soon)

Bridging the gap between mimics and muscles: Our method EIFER utilizes neural unpaired image-to-image translation to decouple facial geometry and appearance for muscle-activity-based expression synthesis and electrode-free facial electromyography.

Abstract

The relationship between muscle activity and resulting facial expressions is crucial for various fields, including psychology, medicine, and entertainment. The synchronous recording of facial mimicry and muscular activity via surface electromyography (sEMG) provides a unique window into these complex dynamics. Unfortunately, existing methods for facial analysis cannot handle electrode occlusion, rendering them ineffective. Even with occlusion-free reference images of the same person, variations in expression intensity and execution are unmatchable. Our electromyography-informed facial expression reconstruction (EIFER) approach is a novel method to restore faces under sEMG occlusion faithfully in an adversarial manner. We decouple facial geometry and visual appearance (e.g., skin texture, lighting, electrodes) by combining a 3D Morphable Model (3DMM) with neural unpaired image-to-image translation via reference recordings. Then, EIFER learns a bidirectional mapping between 3DMM expression parameters and muscle activity, establishing correspondence between the two domains. We validate the effectiveness of our approach through experiments on a dataset of synchronized sEMG recordings and facial mimicry, demonstrating faithful geometry and appearance reconstruction. Further, we synthesize expressions based on muscle activity and how observed expressions can predict dynamic muscle activity. Consequently, EIFER introduces a new paradigm for facial electromyography, which could be extended to other forms of multi-modal face recordings.

Main Insights

Method

EIFER Method

EIFER (Electromyography-Informed Facial Expression Reconstruction) introduces a novel approach for facial expression analysis that addresses the challenges posed by surface electromyography (sEMG) electrode occlusion. Our method is grounded in the principle of decoupling facial geometry and appearance, achieved through a neural unpaired image-to-image translation framework, see the above figure. EIFER leverages 3D Morphable Models (3DMMs), specifically FLAME , to provide a parametric representation of facial shape and expression. We see the 3DMM expression space and muscle activity as the missing link between computer animation and physiologically-grounded facial expression analysis.

For monocular 3D face reconstruction, EIFER employs neural differential rendering and utilizes pre-trained SMIRK encoder networks. These encoders, consisting of sub-encoders based on MobileNetV3 backbones, estimate 3DMM parameters (shape, expression, pose) from the sEMG electrode occluded input facial images. The core of EIFER's occlusion handling lies in a CycleGAN-like adversarial architecture for unpaired image-to-image translation. This architecture comprises generator and discriminator networks trained adversarially using unpaired sets of sEMG-occluded and occlusion-free (reference) facial images. Crucially, EIFER learns a bidirectional mapping between the 3DMM expression parameter space and measured sEMG muscle activity through multi-layer perceptron (MLPs). This bidirectional mapping enables both the synthesis of facial expressions from muscle activity and the prediction of muscle activity from observed facial expressions, effectively achieving electrode-free facial electromyography.

Key Advantages of EIFER:

  • Reconstructs Facial Expressions Under Occlusion: EIFER uniquely addresses the challenge of sensor occlusion, accurately reconstructing facial expressions even when electrodes or other visual obstructions are present.
  • Combines Video and Muscle Activity Data: By leveraging both video and muscle activity (sEMG) data, EIFER provides a more comprehensive and physiologically grounded fully data-driven understanding of facial expressions.
  • Enables Electrode-Free Facial Electromyography: EIFER can predict muscle activity from facial expressions alone, paving the way for non-invasive, electrode-free facial electromyography in the future.
  • Provides Accurate 3D Facial Geometry and Appearance: EIFER not only reconstructs a visually realistic face but also captures the underlying 3D geometry, providing richer data for analysis and synthesis.
  • Utilizes Advanced AI Techniques: Built upon state-of-the-art 3D Morphable Models and unpaired image-to-image translation, EIFER represents a significant advancement in facial analysis technology.
  • Opens Doors for Multi-Modal Facial Analysis: EIFER facilitates the integration of various data streams for a more holistic understanding of facial expressions, with potential applications in medicine, psychology, human-computer interaction, and animation.

Related Links

EIFER combines many ideas from recent works in computer vision, graphics, and machine learning. Here are some related works that you might find interesting:

  • FLAME introduced a 3D Morphable Model that captures facial shape and expression variations.
  • SMIRK replaced the traditional appearance model with a neural renderer and decoupled the facial encoding into three sub-encoders.
  • CycleGAN and MC-CycleGAN are unpaired image-to-image translation methods that have inspired our adversarial training.
  • DECA, EMOCAv2, SMIRK, Deep3DFace, and FOCUS are all state-of-the-art methods for monocular 3D face reconstruction and analysis. We provide Exp2EMG and EMG2Exp models for these methods.

Acknowledgments

This publication is a joint effort of the Computer Vision Group at Friedrich Schiller University Jenaand the Department of Otorhinolaryngology at Jena University Hospital. The project Bridging the gap: Mimics and Muscles was funded by the German Science Foundation under the grant number DFG DE-735/15-1 and DFG GU-463/12-1. We would like to thank all participants who volunteered for the data collection and the reviewers for their valuable feedback.

We thank Niklas Penzel, Gideon Stein, Sai Vemuri, Sven Sickert, and Maha Shadaydeh for their manuscript feedback and advice throughout the project. Additionally, we would like to thank Nadiya Müller, Vanessa Tretzsch, Martin Heinrich, Anna-Maria Kuttenreich, Christian Dobel, Gerd Fabian Volk, Roland Graßme, Paul Funk, and Richard Schneider for their support. Furthermore, we thank Bernhard Egger highlighting the potential of our data for facial expression synthesis.

BibTeX (arXiv Preprint)

If you find our work useful, utilize with our models, start your own research with our data set, or use our parts of our code, please cite our work: 
@article{buchner2025electromyography,
      title={Electromyography-Informed Facial Expression Reconstruction for Physiological-Based Synthesis and Analysis},
      author={B{\"u}chner, Tim and Anders, Christoph and Guntinas-Lichius, Orlando and Denzler, Joachim},
      journal={arXiv preprint arXiv:2503.09556},
      year={2025}
}