PEDFE-EmoAnalysis

Analysis of elicited (genuine) and acted (posed) emotional expressions in the Padova Emotional Dataset of Facial Expressions (PEDFE). This project runs the Py-Feat automated facial-expression toolbox over the PEDFE video clips to predict each clip's emotion, then statistically compares those automated predictions against PEDFE's human-rater ground truth (emotion label, hit rate, and genuineness/intensity ratings) to see how accurately an automated system classifies genuine versus posed expressions and what factors (intensity, genuineness, human hit rate) relate to classifier confidence.

Contributors

• Zikun Fu
• Tony Wang

Data

• Source dataset: Miolla, Cardaioli, and Scarpazza, "Padova Emotional Dataset of Facial Expressions (PEDFE): A unique dataset of genuine and posed emotional facial expressions," Behavior Research Methods 55.5 (2023): 2559-2574.
• data/PEDFE_set_clips/: 1,456 face-focused .avi clips from PEDFE (Pyfeat.ipynb describes this as the "modified clips (focusing solely on the face)" version of the dataset).
• data/Supplemental_Material_T1.csv: 1,458 labeled clips (751 posed, 707 genuine), with columns Subject, Gender, PEDFE_code, Type (Genuine/Posed), Emotion (one of disgust, happiness, fear, anger, surprise, sadness), Hit rate Emotion (%), Hit rate Typo (%), Mean_Genuineness, SD_Genuineness, Mean_Intensity, SD_Intensity, and clip Duration(s).
• data/combined_results.csv: per-clip mean emotion scores (mean_anger, mean_disgust, mean_fear, mean_happiness, mean_sadness, mean_surprise, mean_neutral) produced by the Py-Feat pipeline in Pyfeat.ipynb, for 1,456 of the clips (2 fewer than the 707 labeled genuine clips make it into the merged analysis, consistent with the Pyfeat.ipynb code path that skips and logs any video where no face is detected).
• The clips and labels needed to reproduce the notebooks are included directly in this repository under data/.

Method

Feature extraction and emotion prediction (Pyfeat.ipynb)

• Uses the Py-Feat Detector configured as: face_model="img2pose", landmark_model="mobilefacenet", au_model="xgb", emotion_model="resmasknet", facepose_model="img2pose", device="cuda".
• For each .avi clip in data/PEDFE_set_clips, calls detector.detect_video(video_path, skip_frames=4, aggregate=True), then averages the per-frame emotion scores per clip (extract_mean()) and appends the PEDFE_code plus the 7 mean emotion scores to data/combined_results.csv.
• Clips where no face is detected are logged and skipped rather than written to the output CSV.

Statistical analysis (Analysis.ipynb)

• Loads data/combined_results.csv and data/Supplemental_Material_T1.csv, merges them on PEDFE_code.
• Derives predicted_emotion as the emotion column with the highest mean score for each clip (idxmax over the 7 mean-emotion columns), pred as a binary flag for whether predicted_emotion matches the labeled Emotion, and pred (%) as the maximum mean emotion score (i.e., the model's confidence).
• Splits the merged data into Genuine and Posed subsets (by the Type column) and, for each subset, computes:
  • A Pearson correlation matrix among Mean_Intensity, Mean_Genuineness, Hit rate Emotion (%), and pred (%).
  • A scikit-learn classification_report (precision/recall/F1) comparing labeled Emotion to predicted_emotion across the six emotion classes present in the labels (disgust, happiness, fear, anger, surprise, sadness).
  • A row-normalized confusion matrix (confusion_matrix(..., normalize='true')).
• Fits a Type II ANOVA (statsmodels.formula.api.ols + sm.stats.anova_lm(..., typ=2)) on the full merged dataset with pred (%) as the dependent variable and Mean_Intensity, Mean_Genuineness, and Hit rate Emotion (%) as independent variables.

Findings

Findings below are quoted directly from the saved outputs in Analysis.ipynb.

Classification performance (scikit-learn classification_report, labeled Emotion vs. Py-Feat predicted_emotion):

Subset	support	micro avg (P/R/F1)	macro avg (P/R/F1)	weighted avg (P/R/F1)
Posed	751	0.58 / 0.52 / 0.55	0.57 / 0.47 / 0.46	0.59 / 0.52 / 0.50
Genuine	705	0.55 / 0.49 / 0.52	0.52 / 0.37 / 0.36	0.57 / 0.49 / 0.46

• In both subsets, happiness has by far the highest recall (posed: 0.98; genuine: 0.93), while fear and anger have the lowest recall (posed: 0.11 and 0.09; genuine: 0.03 and 0.14).

Correlation with classifier confidence (pred (%)) (Pearson r, from the printed correlation matrices):

Variable	Posed	Genuine
Mean_Intensity	-0.052041	0.155220
Mean_Genuineness	0.023482	0.176916
Hit rate Emotion (%)	0.183449	0.261928

Type II ANOVA (dependent variable pred (%), full merged dataset, n = 1,456):

Term	sum_sq	df	F	PR(>F)
Mean_Intensity	0.139040	1	3.758497	0.0527
Mean_Genuineness	0.511769	1	13.833991	0.000207
Hit rate Emotion (%)	2.784032	1	75.257086	1.09e-17
Residual	53.714744	1452	-	-

At alpha = 0.05, Mean_Genuineness and Hit rate Emotion (%) are statistically significant predictors of classifier confidence in this model; Mean_Intensity is not (p = 0.0527).

How to run

No requirements.txt is included in this repository; the notebooks import the following packages:

pip install pandas seaborn matplotlib statsmodels scikit-learn py-feat

Then, from the repository root:

1. Open Pyfeat.ipynb and run all cells to regenerate data/combined_results.csv from the clips in data/PEDFE_set_clips (this step uses device="cuda" in the Detector, so it expects a CUDA-capable GPU as configured; edit that argument to run on CPU).
2. Open Analysis.ipynb and run all cells to reproduce the correlation matrices, classification reports, confusion matrices, and ANOVA table above.

Repository structure

• Pyfeat.ipynb: feature extraction and emotion prediction with Py-Feat.
• Analysis.ipynb: merges predictions with labels and runs the correlation, classification, and ANOVA analysis.
• data/PEDFE_set_clips/: PEDFE video clips (see Data section).
• data/Supplemental_Material_T1.csv: PEDFE clip labels and human-rater ratings.
• data/combined_results.csv: Py-Feat per-clip mean emotion scores.

Citations

1. Miolla, Alessio, Matteo Cardaioli, and Cristina Scarpazza. "Padova Emotional Dataset of Facial Expressions (PEDFE): A unique dataset of genuine and posed emotional facial expressions." Behavior Research Methods 55.5 (2023): 2559-2574.
2. Jolly, E., Cheong, J. H., Xie, T., & Chang, L. J. (2022). Py-Feat. https://py-feat.org/pages/models.html