Proctor and colleagues' 2011 paper proposed a taxonomy of eight implementation outcomes and challenged the field to address a research agenda focused on conceptualization, measurement, and theory ...building. Ten years later, this paper maps the field's progress in implementation outcomes research. This scoping review describes how each implementation outcome has been studied, research designs and methods used, and the contexts and settings represented in the current literature. We also describe the role of implementation outcomes in relation to implementation strategies and other outcomes.
Arksey and O'Malley's framework for conducting scoping reviews guided our methods. Using forward citation tracing, we identified all literature citing the 2011 paper. We conducted our search in the Web of Science (WOS) database and added citation alerts sent to the first author from the publisher for a 6-month period coinciding with the WOS citation search. This produced 1346 titles and abstracts. Initial abstract screening yielded 480 manuscripts, and full-text review yielded 400 manuscripts that met inclusion criteria (empirical assessment of at least one implementation outcome).
Slightly more than half (52.1%) of included manuscripts examined acceptability. Fidelity (39.3%), feasibility (38.6%), adoption (26.5%), and appropriateness (21.8%) were also commonly examined. Penetration (16.0%), sustainability (15.8%), and cost (7.8%) were less frequently examined. Thirty-two manuscripts examined implementation outcomes not included in the original taxonomy. Most studies took place in healthcare (45.8%) or behavioral health (22.5%) organizations. Two-thirds used observational designs. We found little evidence of progress in testing the relationships between implementation strategies and implementation outcomes, leaving us ill-prepared to know how to achieve implementation success. Moreover, few studies tested the impact of implementation outcomes on other important outcome types, such as service systems and improved individual or population health.
Our review presents a comprehensive snapshot of the research questions being addressed by existing implementation outcomes literature and reveals the need for rigorous, analytic research and tests of strategies for attaining implementation outcomes in the next 10 years of outcomes research.
Nonprofit human service organizations operating within the same regional network are often faced with dual pressure to compete as well as coordinate administrative operations (by sharing funding, ...staff, or space) to enhance efficiency. Emerging evidence has demonstrated that competing organizations coordinate, despite the risks. Trust, or perceived trustworthiness between two organizations may mitigate the negative influence of competition on coordination, however there have been few explicit tests of this hypothesis among nonprofit organizations. Drawing on quantitative data collected from a network of 36 nonprofit children’s behavioral health organizations, this article empirically tests how competition and perceived trustworthiness interact to influence administrative coordination. Results support the hypothesis that trustworthiness moderates the influence of competition on administrative coordination. Findings suggest that as competing nonprofit leaders build trust, the more their agencies coordinate their administrative functions. This study highlights the importance of leaders’ perceptions for organizational strategy.
Purpose Worldwide mandates for social distancing and home-quarantine have contributed to loneliness and social isolation. We conducted a systematic scoping review to identify network-building ...interventions that address loneliness and isolation, describe their components and impact on network structure, and consider their application in the wake of COVID19. Methods We performed forward and backward citation tracking of three seminal publications on network interventions and Bibliographic search of Web of Science and SCOPUS. We developed data charting tables and extracted and synthesized the characteristics of included studies, using an iteratively updating form. Findings From 3390 retrieved titles and abstracts, we included 8 studies. These interventions focused on building networks at either individual- or group-levels. Key elements that were incorporated in the interventions at varying degrees included (a) creating opportunities to build networks; (b) improving social skills; (c) assessing network diagnostics (i.e. using network data or information to inform network strategies); (d) promoting engagement with influential actors; and (e) a process for goal-setting and feedback. The effect of interventions on network structures, or the moderating effect of structure on the intervention effectiveness was rarely assessed. Conclusions As many natural face-to-face opportunities for social connection are limited due to COVID19, groups already at risk for social isolation and loneliness are disproportionately impacted. Network-building interventions include multiple components that address both the structure of individuals' networks, and their skills and motivation for activating them. These intervention elements could be adapted for delivery via online platforms, and implemented by trained facilitators or novice volunteers, although more rigorous testing is needed.
Social networks transmit knowledge, influence, and resources. These relationships among patients, professionals, and organizations can shape how innovations are disseminated, adopted, implemented, ...and sustained. Network alteration interventions-interventions that change or rewire social networks-have the potential to be used as implementation strategies. Yet, the types, mechanisms, and effectiveness of these interventions for implementation are unclear. This scoping review and iterative synthesis identified and described network alteration strategies that could be tested for implementation.
We used forward and backward citation tracking of influential articles on network interventions, bibliometric searches, and hand searches of peer-reviewed social network journals. At least two team members screened article titles/abstracts to identify studies that met inclusion criteria: empirical studies of an intervention, the intervention was designed to alter some element of a social network, and changes in social network metrics were measured at two or more time points. During full-text reviews, information about the network interventions, actors, ties, and main findings was extracted. Reporting was informed by the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA). To develop our typology, we synthesized these results using an iterative team-based and consensus-building process.
Fifty-three articles met the inclusion criteria. The interventions described were conducted in healthcare systems or behavioral health systems (34%), communities (26.4%), and schools (22.6%). The majority included records describing interventions designed to alter social support, information-sharing, or friendship networks (65%) among individual actors (84.9%), or to increase ties. Eight strategies emerged. Three strategies targeted the general context: (1) change the environment, (2) create groups, and (3) change the composition. Four strategies targeted individual actors: change (4) motivations, (5) skills for networking, (6) knowledge of one's social network, and (7) prominence/roles. One strategy (8) targeted specific ties within the network (targeting a particular pair-wise relationship or changing the nature of an existing tie).
The network alteration strategies in this typology provide further operational specificity for how implementation strategies target relationships. Advancing these strategies will require greater theoretical specification, the development of strategies that target professionals and organizations, and studies that examine the impact on implementation outcomes.
Implementation science has focused mainly on the initial uptake and use of evidence-based practices (EBPs), with less attention to sustainment-i.e., continuous use of these practices, as intended, ...over time in ongoing operations, often involving adaptation to dynamic contexts. Declining EBP use following implementation is well-documented yet poorly understood. Using theories, models, and frameworks (TMFs) to conceptualize sustainment could advance understanding. We consolidated knowledge from published reviews of sustainment studies to identify TMFs with the potential to conceptualize sustainment, evaluate past uses of TMFs in sustainment studies, and assess the TMFs' potential contribution to developing sustainment strategies.
We drew upon reviews of sustainment studies published within the past 10 years, evaluated the frequency with which included articles used a TMF for conceptualizing sustainment, and evaluated the relevance of TMFs to sustainment research using the Theory, Model, and Framework Comparison and Selection Tool (T-CaST). Specifically, we examined whether the TMFs were familiar to researchers, hypothesized relationships among constructs, provided a face-valid explanation of relationships, and included sustainment as an outcome.
Nine sustainment reviews referenced 648 studies; these studies cited 76 unique TMFs. Only 28 TMFs were used in more than one study. Of the 19 TMFs that met the criteria for T-CaST analysis, six TMFs explicitly included sustainment as the outcome of interest, 12 offered face-valid explanations of proposed conceptual relationships, and six identified mechanisms underlying relationships between included constructs and sustainment. Only 11 TMFs performed adequately with respect to all these criteria.
We identified 76 TMFs that have been used in sustainment studies. Of these, most were only used once, contributing to a fractured understanding of sustainment. Improved reporting and use of TMFs may improve understanding of this critical topic. Of the more consistently used TMFs, few proposed face-valid relationships between included constructs and sustainment, limiting their ability to advance our understanding and identify potential sustainment strategies. Future research is needed to explore the TMFs that we identified as potentially relevant, as well as TMFs not identified in our study that nonetheless have the potential to advance our understanding of sustainment and identification of strategies for sustaining EBP use.
Public health programs can only deliver benefits if they are able to sustain activities over time. There is a broad literature on program sustainability in public health, but it is fragmented and ...there is a lack of consensus on core constructs. The purpose of this paper is to present a new conceptual framework for program sustainability in public health.
This developmental study uses a comprehensive literature review, input from an expert panel, and the results of concept-mapping to identify the core domains of a conceptual framework for public health program capacity for sustainability. The concept-mapping process included three types of participants (scientists, funders, and practitioners) from several public health areas (e.g., tobacco control, heart disease and stroke, physical activity and nutrition, and injury prevention).
The literature review identified 85 relevant studies focusing on program sustainability in public health. Most of the papers described empirical studies of prevention-oriented programs aimed at the community level. The concept-mapping process identified nine core domains that affect a program's capacity for sustainability: Political Support, Funding Stability, Partnerships, Organizational Capacity, Program Evaluation, Program Adaptation, Communications, Public Health Impacts, and Strategic Planning. Concept-mapping participants further identified 93 items across these domains that have strong face validity-89% of the individual items composing the framework had specific support in the sustainability literature.
The sustainability framework presented here suggests that a number of selected factors may be related to a program's ability to sustain its activities and benefits over time. These factors have been discussed in the literature, but this framework synthesizes and combines the factors and suggests how they may be interrelated with one another. The framework presents domains for public health decision makers to consider when developing and implementing prevention and intervention programs. The sustainability framework will be useful for public health decision makers, program managers, program evaluators, and dissemination and implementation researchers.
Abstract
Background and Objectives
Prolonged working life is not necessarily associated with good mental health. Despite the importance of healthy working life in later years, little research has ...been conducted on predictors of mental health in the workplace among older workers. This study aimed to investigate how personal (self-efficacy) and team (leader equity) factors are associated with older workers’ mental health through work engagement.
Research Design and Methods
We analyzed responses of 508 U.S. workers aged 50 years and older from the Age and Generations Study data using structural equation modeling.
Results
Results showed that perceived self-efficacy was a strong predictor of mental health. Also, work engagement was a powerful mechanism for promoting older adults’ mental health; engagement partially mediated the relationship between perceived self-efficacy and mental health, and fully mediated the relationship between team leader equity and mental health.
Discussion and Implications
The findings highlight how important it is for employers to invest in human capital development, suggesting human resource programs should focus on strategies that target older adults’ engagement through tailored self-efficacy programs and inclusive leadership training programs. Such attempts would contribute to enhancing the mental health of older workers.
It has been noted that implementation climate is positively associated with implementation effectiveness. However, issues surrounding the measurement of implementation climate, or the extent to which ...organizational members perceive that innovation use is expected, supported and rewarded by their organization remain. Specifically, it is unclear whether implementation climate can be measured as a global construct, whether individual or group-referenced items should be used, and whether implementation climate can be assessed at the group or organizational level.
This research includes two cross-sectional studies with data collected via surveys at the individual level. The first study assessed the implementation climate perceptions of physicians participating in the National Cancer Institute's (NCI) Community Clinical Oncology Program (CCOP), and the second study assessed the perceptions of children's behavioral health clinicians implementing a treatment innovation. To address if implementation climate is a global construct, we used confirmatory factor analysis. To address how implementation climate should be measured and at what level, we followed a five-step framework outlined by van Mierlo and colleagues. This framework includes exploratory factor analysis and correlations to assess differences between individual and group-referenced items and intraclass correlations, interrater agreements, and exploratory factor analysis to determine if implementation climate can be assessed at the organizational level.
The confirmatory factor analysis demonstrated that implementation climate is a global construct consisting of items related to expectations, support and rewards. There are mixed results, however, as to whether implementation climate should be measured using individual or group-referenced items. In our first study, where physicians were geographically dispersed and practice independently, there were no differences based on the type of items used, and implementation climate was an individual level construct. However, in the second study, in which clinicians practice in a central location and interact more frequently, group-referenced items may be appropriate. In addition, implementation climate could be considered an organizational level construct.
The results are context-specific. Researchers should carefully consider the study setting when measuring implementation climate. In addition, more opportunities are needed to validate this measure and understand how well it predicts and explains implementation effectiveness.
Abstract
Background
Traumatic brain injury (TBI) is a complex condition common among individuals treated in behavioral healthcare, but TBI screening has not been adopted in these settings which can ...affect optimal clinical decision-making. Integrating evidence-based practices that address complex health comorbidities into behavioral healthcare settings remains understudied in implementation science, limited by few studies using theory-driven hypotheses to disentangle relationships between proximal and medial indicators on distal implementation outcomes. Grounded in the Theory of Planned Behavior, we examined providers’ attitudes, perceived behavioral control (PBC), subjective norms, and intentions to adopt The Ohio State University TBI Identification Method (OSU TBI-ID) in behavioral healthcare settings.
Methods
We used an explanatory sequential mixed-methods design. In Phase I, 215 providers from 25 organizations in the USA completed training introducing the OSU TBI-ID, followed by a survey assessing attitudes, PBC, norms, and intentions to screen for TBI. After 1 month, providers completed another survey assessing the number of TBI screens conducted. Data were analyzed using structural equation modeling (SEM) with logistic regressions. In Phase II, 20 providers were purposively selected for semi-structured interviews to expand on SEM results. Qualitative data were analyzed using thematic analysis, integrated with quantitative results, and combined into joint displays.
Results
Only 25% (55/215) of providers adopted TBI screening, which was driven by motivations to trial the intervention. Providers who reported more favorable attitudes (
OR
: 0.67,
p
< .001) and greater subjective norms (
OR
: 0.12,
p
< .001) toward TBI screening demonstrated increased odds of intention to screen, which resulted in greater TBI screening adoption (
OR
: 0.30;
p
< .01). PBC did not affect intentions or adoption. Providers explained that although TBI screening can improve diagnostic and clinical decision-making, they discussed that additional training, leadership engagement, and state-level mandates are needed to increase the widespread, systematic uptake of TBI screening.
Conclusions
This study advances implementation science by using theory-driven hypothesis testing to disentangle proximal and medial indicators at the provider level on TBI screening adoption. Our mixed-methods approach added in-depth contextualization and illuminated additional multilevel determinants affecting intervention adoption, which guides a more precise selection of implementation strategies.
Implementation outcomes research spans an exciting mix of fields, disciplines, and geographical space. Although the number of studies that cite the 2011 taxonomy has expanded considerably, the ...problem of harmony in describing outcomes persists. This paper revisits that problem by focusing on the clarity of reporting outcomes in studies that examine them. Published recommendations for improved reporting and specification have proven to be an important step in enhancing the rigor of implementation research. We articulate reporting problems in the current implementation outcomes literature and describe six practical recommendations that address them.
Our first recommendation is to clearly state each implementation outcome and provide a definition that the study will consistently use. This includes providing an explanation if using the taxonomy in a new way or merging terms. Our second recommendation is to specify how each implementation outcome will be analyzed relative to other constructs. Our third recommendation is to specify "the thing" that each implementation outcome will be measured in relation to. This is especially important if you are concurrently studying interventions and strategies, or if you are studying interventions and strategies that have multiple components. Our fourth recommendation is to report who will provide data and the level at which data will be collected for each implementation outcome, and to report what kind of data will be collected and used to assess each implementation outcome. Our fifth recommendation is to state the number of time points and frequency at which each outcome will be measured. Our sixth recommendation is to state the unit of observation and the level of analysis for each implementation outcome.
This paper advances implementation outcomes research in two ways. First, we illustrate elements of the 2011 research agenda with concrete examples drawn from a wide swath of current literature. Second, we provide six pragmatic recommendations for improved reporting. These recommendations are accompanied by an audit worksheet and a list of exemplar articles that researchers can use when designing, conducting, and assessing implementation outcomes studies.
PDF
