The visual environment is a fundamental yet unquantified determinant of mental health. While the concept of the environmental exposome is well-established, current methods rely on coarse geospatial proxies or biased self-reports, failing to capture the first-person visual context of daily life. Here we show that coupling ecological momentary assessment with vision-language models can quantify the semantic richness of human visual experience.
Across 2,674 participant-generated photographs, model-derived estimates of “greenness” robustly predicted momentary affect and chronic stress, consistent with established benchmarks. We then developed a semi-autonomous large language model-based pipeline that mined over seven million scientific publications to extract nearly 1,000 environmental features empirically linked to mental health. When applied to real-world imagery, up to 33% of model-extracted context ratings significantly correlated with affect and stress.
Mental illness creates an immense global burden on public health and economies1. Decades of research suggest that everyday surroundings exert a profound influence on mental health2, with greenness emerging as one of the most consistently replicated environmental predictors3,4,5,6. Collectively, these surroundings can be summarised as the 'environmental exposome'7, with the visual perception of the environment termed the visual exposome.
Still, our ability to understand and quantify how the visual exposome shapes mental health remains surprisingly limited: we can measure where people are and how they subjectively perceive their surroundings, but we remain effectively blind to what people actually see in their daily lives. GPS-based metrics and satellite-derived indices provide useful macro-level proxies of environmental features4,8,9,10, but they cannot cover the semantic richness, spatial nuance, or social signals embedded in real-world visual scenes. Most crucially, they overlook the immediate, first-person environments individuals actually encounter, whether a space is crowded or quiet, whether animals or other people are present, whether natural light, greenery, or stress-inducing cues appear in view11,12.
Self-report measures reflect only subjective impressions and are prone to response biases. As a result, the visual exposome, arguably the most proximal form of contextual exposure, has remained empirically inaccessible at scale. An optimal way to quantify the visual exposome would be to use photographs taken by participants themselves to capture health-relevant contextual features of their environments.
This method provides a large number of data points that reflect the participant’s surroundings from a first-person perspective, allowing researchers to see the world through the participants’ eyes. To date, however, this approach has been used almost exclusively in qualitative research, for example, to explore subjective experiences, health contexts, or to illustrate environmental features13,14,15, but not in quantitative frameworks examining associations between visually captured aspects of daily environments and psychological and mental health-relevant outcomes. This limited application is most likely due to the reliance on manual expert evaluation of visual content according to predefined categories, an approach that is time-consuming, subjective and inherently not scalable, restricting its use in intensive longitudinal studies with large sample sizes.
Extract — continue reading at the source.
