Facial Expression Recognition for Clinical Trial Self-recordings

Abstract

Facial expressions have long been linked to human emotions, physiology, and behavior. Novel work suggests that clinical trial subject facial expression relates to self-reported quality of life. This implies automated facial expression recognition tools have potential to increase understanding of emotional and social functioning in response to treatment. Limited availability of labeled facial image datasets has however historically posed challenges for building robust facial expression recognition tools that generalize to diverse populations. This thesis project presents a network predicting gold-standard facial expression labels called action units. To overcome data scarcity challenges a multi-stage pretraining approach is introduced. The model achieves ROC AUC of 0.88 over six action units for cross validation with held-out subjects. Predictions for action units that are rare in the training data generalize to unseen subjects which highlights the benefit of the pretraining approach. The results further indicate a possible advantage of including a subject-level baseline.