Objective
To evaluate the effects of specification choices on the accuracy of estimates in difference‐in‐differences (DID) models.
Data Sources
Process‐of‐care quality data from Hospital Compare ...between 2003 and 2009.
Study Design
We performed a Monte Carlo simulation experiment to estimate the effect of an imaginary policy on quality. The experiment was performed for three different scenarios in which the probability of treatment was (1) unrelated to pre‐intervention performance; (2) positively correlated with pre‐intervention levels of performance; and (3) positively correlated with pre‐intervention trends in performance. We estimated alternative DID models that varied with respect to the choice of data intervals, the comparison group, and the method of obtaining inference. We assessed estimator bias as the mean absolute deviation between estimated program effects and their true value. We evaluated the accuracy of inferences through statistical power and rates of false rejection of the null hypothesis.
Principal Findings
Performance of alternative specifications varied dramatically when the probability of treatment was correlated with pre‐intervention levels or trends. In these cases, propensity score matching resulted in much more accurate point estimates. The use of permutation tests resulted in lower false rejection rates for the highly biased estimators, but the use of clustered standard errors resulted in slightly lower false rejection rates for the matching estimators.
Conclusions
When treatment and comparison groups differed on pre‐intervention levels or trends, our results supported specifications for DID models that include matching for more accurate point estimates and models using clustered standard errors or permutation tests for better inference. Based on our findings, we propose a checklist for DID analysis.
Objective
To evaluate the impact of hospital value‐based purchasing (HVBP) on clinical quality and patient experience during its initial implementation period (July 2011–March 2012).
Data Sources
...Hospital‐level clinical quality and patient experience data from Hospital Compare from up to 5 years before and three quarters after HVBP was initiated.
Study Design
Acute care hospitals were exposed to HVBP by mandate while critical access hospitals and hospitals located in Maryland were not exposed. We performed a difference‐in‐differences analysis, comparing performance on 12 incentivized clinical process and 8 incentivized patient experience measures between hospitals exposed to the program and a matched comparison group of nonexposed hospitals. We also evaluated whether hospitals that were ultimately exposed to HVBP may have anticipated the program by improving quality in advance of its introduction.
Principal Findings
Difference‐in‐differences estimates indicated that hospitals that were exposed to HVBP did not show greater improvement for either the clinical process or patient experience measures during the program's first implementation period. Estimates from our preferred specification showed that HVBP was associated with a 0.51 percentage point reduction in composite quality for the clinical process measures (p > .10, 95 percent CI: −1.37, 0.34) and a 0.30 percentage point reduction in composite quality for the patient experience measures (p > .10, 95 percent CI: −0.79, 0.19). We found some evidence that hospitals improved performance on clinical process measures prior to the start of HVBP, but no evidence of this phenomenon for the patient experience measures.
Conclusions
The timing of the financial incentives in HVBP was not associated with improved quality of care. It is unclear whether improvement for the clinical process measures prior to the start of HVBP was driven by the expectation of the program or was the result of other factors.
Difference-in-differences (DID) analysis is used widely to estimate the causal effects of health policies and interventions. A critical assumption in DID is “parallel trends”: that pre-intervention ...trends in outcomes are the same between treated and comparison groups. To date, little guidance has been available to researchers who wish to use DID when the parallel trends assumption is violated. Using a Monte Carlo simulation experiment, we tested the performance of several estimators (standard DID; DID with propensity score matching; single-group interrupted time-series analysis; and multi-group interrupted time-series analysis) when the parallel trends assumption is violated. Using nationwide data from US hospitals (n = 3737) for seven data periods (four pre-interventions and three post-interventions), we used alternative estimators to evaluate the effect of a placebo intervention on common outcomes in health policy (clinical process quality and 30-day risk-standardized mortality for acute myocardial infarction, heart failure, and pneumonia). Estimator performance was assessed using mean-squared error and estimator coverage. We found that mean-squared error values were considerably lower for the DID estimator with matching than for the standard DID or interrupted time-series analysis models. The DID estimator with matching also had superior performance for estimator coverage. Our findings were robust across all outcomes evaluated.
•DEA is used to develop a composite measure of health care quality.•An empirical study is carried in US Department of Veterans Affairs nursing homes.•DEA identifies fewer high performers but more ...highly rewards the high performers.•Advantages of DEA for developing composite measure make it worth pursuing further.•Monte Carlo resampling with replacement is applied to reflect DEA data uncertainty.
Composite measures calculated from individual performance indicators increasingly are used to profile and reward health care providers. We illustrate an innovative way of using Data Envelopment Analysis (DEA) to create a composite measure of quality for profiling facilities, informing consumers, and pay-for-performance programs. We compare DEA results to several widely used alternative approaches for creating composite measures: opportunity-based-weights (OBW, a form of equal weighting) and a Bayesian latent variable model (BLVM, where weights are driven by variances of the individual measures). Based on point estimates of the composite measures, to a large extent the same facilities appear in the top decile. However, when high performers are identified because the lower limits of their interval estimates are greater than the population average (or, in the case of the BLVM, the upper limits are less), there are substantial differences in the number of facilities identified: OBWs, the BLVM and DEA identify 25, 17 and 5 high-performers, respectively. With DEA, where every facility is given the flexibility to set its own weights, it becomes much harder to distinguish the high performers. In a pay-for-performance program, the different approaches result in very different reward structures: DEA rewards a small group of facilities a larger percentage of the payment pool than the other approaches. Finally, as part of the DEA analyses, we illustrate an approach that uses Monte Carlo resampling with replacement to calculate interval estimates by incorporating uncertainty in the data generating process for facility input and output data. This approach, which can be used when data generating processes are hierarchical, has the potential for wider use than in our particular application.
Background
Despite robust evidence of fathers’ impact on children and mothers, engaging with fathers is one of the least well‐explored and articulated aspects of parenting interventions. It is ...therefore critical to evaluate implicit and explicit biases manifested in current approaches to research, intervention, and policy.
Methods
We conducted a systematic database and a thematic hand search of the global literature on parenting interventions. Studies were selected from Medline, Psychinfo, SSCI, and Cochrane databases, and from gray literature on parenting programs, using multiple search terms for parent, father, intervention, and evaluation. We tabulated single programs and undertook systematic quality coding to review the evidence base in terms of the scope and nature of data reporting.
Results
After screening 786 nonduplicate records, we identified 199 publications that presented evidence on father participation and impact in parenting interventions. With some notable exceptions, few interventions disaggregate ‘father’ or ‘couple’ effects in their evaluation, being mostly driven by a focus on the mother–child dyad. We identified seven key barriers to engaging fathers in parenting programs, pertaining to cultural, institutional, professional, operational, content, resource, and policy considerations in their design and delivery.
Conclusions
Barriers to engaging men as parents work against father inclusion as well as father retention, and undervalue coparenting as contrasted with mothering. Robust evaluations of father participation and father impact on child or family outcomes are stymied by the ways in which parenting interventions are currently designed, delivered, and evaluated. Three key priorities are to engage fathers and coparenting couples successfully, to disaggregate process and impact data by fathers, mothers, and coparents, and to pay greater attention to issues of reach, sustainability, cost, equity, and scale‐up. Clarity of purpose with respect to gender‐differentiated and coparenting issues in the design, delivery, and evaluation of parenting programs will constitute a game change in this field.
Read the Commentary on this article at doi: 10.1111/jcpp.12321
Objective
To compare methods of analyzing endogenous treatment effect models for nonlinear outcomes and illustrate the impact of model specification on estimates of treatment effects such as health ...care costs.
Data Sources
Secondary data on cost and utilization for inpatients hospitalized in five Veterans Affairs acute care facilities in 2005–2006.
Study Design
We compare results from analyses with full information maximum simulated likelihood (FIMSL); control function (CF) approaches employing different types and functional forms for the residuals, including the special case of two‐stage residual inclusion; and two‐stage least squares (2SLS). As an example, we examine the effect of an inpatient palliative care (PC) consultation on direct costs of care per day.
Data Collection/Extraction Methods
We analyzed data for 3,389 inpatients with one or more life‐limiting diseases.
Principal Findings
The distribution of average treatment effects on the treated and local average treatment effects of a PC consultation depended on model specification. CF and FIMSL estimates were more similar to each other than to 2SLS estimates. CF estimates were sensitive to choice and functional form of residual.
Conclusions
When modeling cost or other nonlinear data with endogeneity, one should be aware of the impact of model specification and treatment effect choice on results.
Summary Background Dalcetrapib modulates cholesteryl ester transfer protein (CETP) activity to raise high-density lipoprotein cholesterol (HDL-C). After the failure of torcetrapib it was unknown if ...HDL produced by interaction with CETP had pro-atherogenic or pro-inflammatory properties. dal-PLAQUE is the first multicentre study using novel non-invasive multimodality imaging to assess structural and inflammatory indices of atherosclerosis as primary endpoints. Methods In this phase 2b, double-blind, multicentre trial, patients (aged 18–75 years) with, or with high risk of, coronary heart disease were randomly assigned (1:1) to dalcetrapib 600 mg/day or placebo for 24 months. Randomisation was done with a computer-generated randomisation code and was stratified by centre. Patients and investigators were masked to treatment. Coprimary endpoints were MRI-assessed indices (total vessel area, wall area, wall thickness, and normalised wall index average carotid) after 24 months and18 F-fluorodeoxyglucose (18 F-FDG) PET/CT assessment of arterial inflammation within an index vessel (right carotid, left carotid, or ascending thoracic aorta) after 6 months, with no-harm boundaries established before unblinding of the trial. Analysis was by intention to treat. This trial is registered at ClinicalTrials.gov , NCT00655473. Findings 189 patients were screened and 130 randomly assigned to placebo (66 patients) or dalcetrapib (64 patients). For the coprimary MRI and PET/CT endpoints, CIs were below the no-harm boundary or the adverse change was numerically lower in the dalcetrapib group than in the placebo group. MRI-derived change in total vessel area was reduced in patients given dalcetrapib compared with those given placebo after 24 months; absolute change from baseline relative to placebo was −4·01 mm2 (90% CI −7·23 to −0·80; nominal p=0·04). The PET/CT measure of index vessel most-diseased-segment target-to-background ratio (TBR) was not different between groups, but carotid artery analysis showed a 7% reduction in most-diseased-segment TBR in the dalcetrapib group compared with the placebo group (–7·3 90% CI −13·5 to −0·8; nominal p=0·07). Dalcetrapib did not increase office blood pressure and the frequency of adverse events was similar between groups. Interpretation Dalcetrapib showed no evidence of a pathological effect related to the arterial wall over 24 months. Moreover, this trial suggests possible beneficial vascular effects of dalcetrapib, including the reduction in total vessel enlargement over 24 months, but long-term safety and clinical outcomes efficacy of dalcetrapib need to be analysed. Funding F Hoffmann-La Roche Ltd.
Objective. To examine differences in use of Veterans Health Administration (VA) and Medicare outpatient services by VA primary care patients.
Data Sources/Study Setting. VA administrative and ...Medicare claims data from 2001 to 2004.
Study Design. Retrospective cohort study of outpatient service use by 8,964 community‐based and 6,556 hospital‐based VA primary care patients.
Principal Findings. A significant proportion of VA patients used Medicare‐reimbursed primary care (>30 percent) and specialty care (>60 percent), but not mental health care (3–4 percent). Community‐based patients had 17 percent fewer VA primary care visits (p<.001), 9 percent more Medicare‐reimbursed visits (p<.001), and 6 percent fewer total visits (p<.05) than hospital‐based patients. Community‐based patients had 22 percent fewer VA specialty care visits (p<.0001) and 21 percent more Medicare‐reimbursed specialty care visits (p<.0001) than hospital‐based patients, but no difference in total visits (p=.80).
Conclusions. Medicare‐eligible VA primary care patients followed over 4 consecutive years used significant primary care and specialty care outside of VA. Community‐based patients offset decreased VA use with increased service use paid by Medicare, suggesting that increasing access to VA primary care via community clinics may fragment veteran care in unintended ways. Coordination of care between VA and non‐VA providers and health care systems is essential to improve the quality and continuity of care.
Many e-health technologies are available to promote virtual patient–provider communication outside the context of face-to-face clinical encounters. Current digital communication modalities include ...cell phones, smartphones, interactive voice response, text messages, e-mails, clinic-based interactive video, home-based web-cams, mobile smartphone two-way cameras, personal monitoring devices, kiosks, dashboards, personal health records, web-based portals, social networking sites, secure chat rooms, and on-line forums. Improvements in digital access could drastically diminish the geographical, temporal, and cultural access problems faced by many patients. Conversely, a growing digital divide could create greater access disparities for some populations. As the paradigm of healthcare delivery evolves towards greater reliance on non-encounter-based digital communications between patients and their care teams, it is critical that our theoretical conceptualization of access undergoes a concurrent paradigm shift to make it more relevant for the digital age. The traditional conceptualizations and indicators of access are not well adapted to measure access to health services that are delivered digitally outside the context of face-to-face encounters with providers. This paper provides an overview of digital “encounterless” utilization, discusses the weaknesses of traditional conceptual frameworks of access, presents a new access framework, provides recommendations for how to measure access in the new framework, and discusses future directions for research on access.
Objective
The objective of this research was to apply a new methodology (population‐level cost‐effectiveness analysis) to determine the value of implementing an evidence‐based practice in routine ...care.
Data Sources/Study Setting
Data are from sequentially conducted studies: a randomized controlled trial and an implementation trial of collaborative care for depression. Both trials were conducted in the same practice setting and population (primary care patients prescribed antidepressants).
Study Design
The study combined results from a randomized controlled trial and a pre‐post‐quasi‐experimental implementation trial.
Data Collection/Extraction Methods
The randomized controlled trial collected quality‐adjusted life years (QALYs) from survey and medication possession ratios (MPRs) from administrative data. The implementation trial collected MPRs and intervention costs from administrative data and implementation costs from survey.
Principal Findings
In the randomized controlled trial, MPRs were significantly correlated with QALYs (p = .03). In the implementation trial, patients at implementation sites had significantly higher MPRs (p = .01) than patients at control sites, and by extrapolation higher QALYs (0.00188). Total costs (implementation, intervention) were nonsignificantly higher ($63.76) at implementation sites. The incremental population‐level cost‐effectiveness ratio was $33,905.92/QALY (bootstrap interquartile range −$45,343.10/QALY to $99,260.90/QALY).
Conclusions
The methodology was feasible to operationalize and gave reasonable estimates of implementation value.
PDF
