Statewide assessments of community priorities conducted by University of Kentucky (UK) Extension identified substance use and mental health as immediate priority issues in an overwhelming majority of Kentucky counties in 2019 and again in 2023 (UK Cooperative Extension Service, 2019; 2023). The findings of the needs assessments are not surprising given that Kentucky residents disproportionately experience mental health disorders, substance use disorder (SUD), and related harms (Kaiser Family Foundation, KFF, 2023). Kentucky reports consistently high rates of poor mental health outcomes when compared to the national averages and those indicators have been exacerbated by the COVID-19 pandemic (Salt et al., 2023).  

University of Kentucky Extension addresses the pressing and overlapping issues of substance use and mental health through direct and indirect education. These efforts range from programs addressing mental health and wellbeing in rural communities to those building community and individual recovery capital (e.g., financial education for adults in recovery, prevention programs in middle schools, mental health awareness for farmers and farm families).  Given the breadth and scope of SUMH programming, UK Extension was challenged to evaluate the collective impact of its substance use and mental health programs and outreach efforts. Specifically, each of the substance use or mental health programs focused on individual program evaluation and measured different outcomes, which hindered the aggregation of evaluation data across programs, resulting in limited statewide assessment of broader impact.

Recognizing the need to streamline evaluation processes to fully capture the impact of these efforts, UK Extension commissioned a Substance Use and Mental Health Collective Impact Initiative (hereafter, SUMH) in 2022. The SUMH used the collective impact model (Kania & Kramer, 2011) as a conceptual and methodological framework for planning and engaging Extension specialists and faculty from various program areas to drive the necessary systems change (Laue et al., 2024) needed to design and implement a collective impact evaluation of UK Extension’s outreach and programmatic efforts addressing either substance use or mental health challenges in Kentucky.   

Development of the priority indicators was data-driven, informed by both a review of existing indicators reported in the KERS reporting system and the findings of the program description survey. The SUMH created 15 priority indicators of success (i.e., eight SUR indicators and seven MHW indicators) comprising three categories: (1) process indicators focused on program partnerships; (2) short-term indicators focused on program participants’ intention to change and gains in knowledge, opinions, skills, and aspirations; and (3) behavior change indicators focused on participants’ behaviors, practices, and actions resulting from program participation.

Four in-service workshops were conducted to familiarize Extension agents and other personnel with the new structure of aggregated success indicators, as well as the newly developed evaluation and reporting tools.  A total of 91 Extension personnel have been trained to use the indicators and evaluation tools, of which 81 completed the post-participation evaluation survey. About 94% of survey respondents reported that the training increased their awareness of UK Extension SUR and MHW programs/outreach; 93% reported that they feel confident that they can use the indicators and tools to evaluate their programs; 99% reported that they understand how to report the data from the evaluation tools in KERS; 94% reported that the tools are easy to use; and 94% reported that the indicators and evaluation tools will make evaluation of their programs easier.  

The establishment and functioning of the UK Extension SUMH initiative will inform other organizations and Extension Services looking to measure collective impact for emerging and/or critical issues. The methodologies and program planning framework described in our manuscript (Adedokun et al., 2025) could be adapted and/or applied to other complex interdisciplinary issues routinely addressed through complex, multi-layered interventions and programs. Through this extensive process, seven larger lessons were learned that serve as important considerations for others contemplating the work.