Multimodal computational analysis of longitudinal stress profiles in healthcare workers

Healthcare workers (HCWs) face elevated risk for burnout and stress-related disorders, yet conventional self-report assessments are limited by professional stigma and underreporting. We evaluated whether multimodal computational analysis of brief, remotely collected stress narratives could distinguish longitudinal stress profiles. In a prospective six-week cohort study, N = 750 HCWs were invited to provide weekly naturalistic narratives describing recent stressful experiences and repeated measures of anxiety, depression, burnout, and subjective distress.

Among 553 participants with sufficient repeated self-report data, multivariate longitudinal clustering identified two distinct profiles: Resilient (n = 295) and Vulnerable (n = 258). Linguistic, acoustic, and facial expression embeddings were extracted from the recordings and integrated using a hierarchical multimodal transformer. In a held-out test set, the model achieved an AUROC of 0.75, outperforming unimodal (linguistic embeddings; AUROC = 0.63) and bimodal (linguistic + acoustic; AUROC = 0.70) configurations.

These findings provide proof of concept that multimodal embeddings extracted from brief stress narratives are associated with occupational stress profiles, potentially complementing traditional assessment methods and informing targeted prevention strategies. We are grateful to the participants for their time, effort, and participation in the study. The authors would like to thank Sarah Riley for her assistance with study enrollment and data collection.

This study was funded by the Defense Advanced Research Projects Agency (DARPA), Research and Technology Development, under contract number HR001122S0003. K.S. received support from the National Institute of Mental Health (R01MH129856) and the National Heart, Lung, and Blood Institute (R01HL156134). M.M.'s research was supported by the National Institute of Mental Health (NIMH) through grant K23MH134068.

The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. The funders played no role in study design, data collection, analysis and interpretation of data, or the writing of this manuscript. These authors contributed equally: Charles R.

Marmar, Katharina Schultebraucks. Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA Sapir Gershov, Dayeon Cho, Kacey Ferguson, Matteo Malgaroli, Thea Gallagher, Charles R. Marmar & Katharina Schultebraucks Dimitra Vergyri, Alan Taitz & Chase Adams Center for Data Science, New York University, New York, NY, USA Neuroscience Institute, New York University, New York, NY, USA Department of Population Health, NYU Grossman School of Medicine, New York, NY, USA Correspondence to Katharina Schultebraucks.