Hallucinations are the most common psychotic experiences. They are prevalent in the context of a wide variety of mental disorders but also occur in approximately 10% of the general population. The predominant clinical view of hallucinations is that they are a hallmark symptom of major psychiatric disorders with significant negative outcomes that will require intensive and chronic care. However, data from epidemiologic and phenomenological studies paint a more complex picture. Some studies suggest that hallucinations are significant risk factors for serious negative outcomes such as emergency service use, hospitalization, and suicide. Other studies show that hallucinations are commonly reported by people who seldom, if ever, require mental health care. Identifying when individuals experiencing hallucinations have impending negative outcomes is a prerequisite to timely and appropriate treatment. Unfortunately, our diagnostic and risk evaluation capabilities are hampered by the limitations of our assessment strategies, which typically involve resource intensive clinical interviews, administered by trained clinicians, in clinic settings.

Ubiquitous smartphone technologies offer us novel opportunities to administer behavioral measures that can capture the experience and impact of hallucinations at a scope, scale, and ecological validity that far exceed clinic-based assessment capabilities. Applying powerful computational methods to the rich data collected using mobile behavioral tasks has the potential to yield tools for identification of those at heightened risk for major clinical and functional impairments. If successful, data-driven tools would advance our ability to allocate the right clinical resources, at the right time, to the right people.

Our multidisciplinary group is uniquely well-situated to accomplish these objectives. We have conducted several pioneering studies demonstrating our ability to: A. Efficiently recruit large, diverse, and representative samples of people experiencing hallucinations, using online methodologies (>1,000 total); B. Engage participants from over 40 U.S. States in remote digital data collection, administering brief and scalable smartphone-delivered behavioral tasks including ecological momentary assessment, audio diaries, and verbal memory tasks; C. Retrieve, process, securely store, integrate, and synthesize high-density repeated measures audio recorded speech data from participant smartphones; D. Apply Natural Language Processing methods, including state-of- the-art neural language models, to flag clinically relevant indicators (e.g., thought disorganization, suicide risk) in participant data in a manner comparable to expert clinical raters. In response to RFA-MH-23-105, we propose to adopt best practices from these studies and to integrate existing and new computational modelling techniques to promote scalable smartphone-administered measurement to inform development and optimization of key clinical signatures. We will recruit 2,000 individuals experiencing hallucinations using our demonstrated remote recruitment strategies. Participants will install our smartphone behavioral measurement (SBM) package that will prompt them to complete brief self-report measures, audio diaries describing their hallucinatory experiences, and validated audio-administered verbal memory task in their own environments; initially over a 30-day period (baseline) and then for shorter week-long data collection “bursts”, in three-month intervals. Clinical outcomes data will be collected longitudinally at 3,6,9, and 12 months. Our objective is to leverage novel data sets and cutting-edge computational methods to generate clinical signatures that can be used to inform scalable and timely risk detection and clinical decision making. The study will produce the largest corpus of audio recorded speech data from a cohort of people experiencing hallucinations to date. Specifically, we aim to:

Aim 1: Derive data-driven clinical signatures from mobile behavioral tasks to predict individual differences in severe negative outcomes among people experiencing hallucinations. Using multiple computational approaches, we will evaluate which self-report items, speech derived features, and verbal recall patterns differentiate individuals who do or do not develop severe negative outcomes (i.e., emergency service use, hospitalization, suicide attempt) prospectively, over a one-year period.

Aim 2: Identify and mitigate bias in modelling across groups defined by race, sex, and age. All speech recognition and feature extraction methods will be evaluated for differences across participant subgroups. We will fine-tune speech and language models on publicly available subpopulation-derived data, extend lexical dictionaries as needed, and de-bias downstream machine learning models to address observed discrepancies.

Aim 3: Examine whether adding smartphone-captured behavioral data to information that is typically available in the clinical record improves model clinical utility. We will use likelihood ratio tests and model fit indices to compare hierarchically nested models of the additional parameters above and beyond the type of individual-level data that is typically available in electronic health records.

Aim 4: Produce machine learning-ready data structures that adhere to FAIR (Findable, Accessible, Interoperable, Reusable) principles. Working closely with the Data Coordinating Center, we will apply a tailored de-identification system to transcribed text. De-identified transcripts and extracted features will be harmonized with baseline data, enriched with granular metadata, indexed in searchable resources, and made available to the broader scientific community in AI-ready formats to advance machine learning efforts in this area.