Impact Evaluation in Practice Gertler, Paul; Martinez, Sebastian; Premand, Patrick ...
2010, 12-13-2010, 20110101
eBook
Odprti dostop
This book offers an accessible introduction to the topic of impact evaluation and its practice in development. While the book is geared principally towards development practitioners and policymakers ...designing prospective impact evaluations, we trust that it will be a valuable resource for students and others interested in using impact evaluation. Prospective impact evaluations should be used selectively to assess whether or not a program has achieved its intended results, or to test alternatives for achieving those results. We consider that more and better impact evaluation will help strengthen the evidence base for development policies and programs around the world. If governments and development practitioners can make policy decisions based on evidence - including evidence generated through impact evaluation - our hope is that development resources will be spent more effectively, and ultimately have a greater impact on reducing poverty and improving people?s lives. The three chapters in this handbook provide a non-technical introduction to impact evaluations, including ?Why Evaluate? in Chapter 1, ?How to Evaluate? in Chapter 2 and ?How to Implement Impact Evaluations? in Chapter 3. These elements are the basic ?tools? needed in order to successfully carry out an impact evaluation. From a methodological standpoint our approach to impact evaluation is largely pragmatic: we think that the most appropriate methods should be identified to fit the operational context, and not the other way around. This is best achieved at the outset of the program, through the design of prospective impact evaluation that can be built into the project?s implementation. We argue that gaining consensus between key stakeholders and identifying an evaluation design that fits the political and operational context is as important as the method itself. We also believe
strongly that impact evaluations should be upfront about their limitations and caveats. Finally, we strongly encourage policymakers and program managers to consider impact evaluations in a logical framework that clearly sets out the causal pathways by which the program works to produce outputs and influence final outcomes, and to combine impact evaluations with monitoring and selected complementary evaluation approach to gain a full picture of performance.This book builds on a core set of teaching materials developed for the ?Turning Promises to Evidence? workshops organized by the office of the Chief Economist for Human Development (HDNCE) in partnership with regional units and the Development Economics Research Group (DECRG) at the World Bank.
Several developing economies have recently introduced conditional cash transfer programs, which provide money to poor families contingent on certain behavior, usually investments in human capital, ...such as sending children to school or bringing them to health centers. The approach is both an alternative to more traditional social assistance programs and a demand-side complement to the supply of health and education services. Unlike most development initiatives, conditional cash transfer programs have been subject to rigorous evaluations of their effectiveness using experimental or quasi-experimental methods. Evaluation results for programs launched in Colombia, Honduras, Jamaica, Mexico, Nicaragua, and Turkey reveal successes in addressing many of the failures in delivering social assistance, such as weak poverty targeting, disincentive effects, and limited welfare impacts. There is clear evidence of success from the first generation of programs in Colombia, Mexico, and Nicaragua in increasing enrollment rates, improving preventive health care, and raising household consumption. Many questions remain unanswered, however, including the potential of conditional cash transfer programs to function well under different conditions, to address a broader range of challenges among poor and vulnerable populations, and to prevent the intergenerational transmission of poverty.
Sugira Muryango is a father-engaged early child development and violence-prevention home-visiting programme delivered by trained lay workers. This cluster-randomised trial evaluates whether families ...living in extreme poverty (Ubudehe 1, the poorest category in the Government of Rwanda's wealth ranking) who receive Sugira Muryango in combination with a government-provided social protection programme demonstrate greater responsive, positive caregiving, nutrition, care seeking, hygiene, and father involvement compared with control families receiving usual care (UC).
Using detailed maps, we grouped closely spaced villages into 284 geographic clusters stratified by the type of social protection programmes operating in the village clusters; 198 clusters met all enrolment criteria. Sugira Muryango was delivered to n = 541 families in 100 treatment clusters with children aged 6-36 months living in extreme poverty. We assessed changes in outcomes in intervention and n = 508 UC control families using structured surveys and observation. Analyses were intent to treat using mixed models to accommodate clustering.
Families receiving Sugira Muryango improved on core outcomes of parent-child relationships assessed using the Home Observation for Measurement of the Environment (Cohen's d = 0.87, 95% CI: 0.74, 0.99) and the Observation of Mother-Child Interaction (Cohen's d = 0.29, 95% CI: 0.17, 0.41). We also saw reductions in harsh discipline on items from the UNICEF MICS (OR = 0.30: 95% CI: 0.19, 0.47) and in violent victimisation of female caregivers by their partners (OR = 0.49, 95% CI: 0.24, 1.00) compared with UC. Moreover, children in families receiving SM had a 0.45 higher increase in food groups consumed in the past 24 h (Cohen's d = 0.35, 95% CI: 0.22, 0.47), increased care seeking for diarrhoea (OR = 4.43, 95% CI: 1.95, 10.10) and fever (OR = 3.28, 95% CI: 1.82, 5.89), and improved hygiene behaviours such as proper treatment of water (OR = 3.39, 95% CI: 2.16, 5.30) compared with UC. Finally, Sugira Muryango was associated with decreased caregiver depression and anxiety (OR = 0.58, 95% CI: 0.38, 0.88).
Sugira Muryango led to improvements in caregiver behaviours linked to child development and health as well as reductions in violence.
ClinicalTrials.gov number NCT02510313.
There is growing interest in how best to leverage cash transfers to foster positive impact on children in beneficiary households. We evaluate the effects of interventions based on behavioral science ...on measures of early childhood socio-cognitive development (and related household-level outcomes) for children from households receiving cash transfers in Madagascar using a multi-arm cluster-randomized trial, where communities were randomized into arms, with 77 communities in each arm and approximately 950 and 1200 households sampled at baseline and midline, respectively. Three behavioral interventions (a ‘Mother Leaders’ group, either by itself or augmented with a ‘self-affirmation’ or a ‘plan-making’ nudge) are layered onto a child-focused cash transfer program targeting the rural poor in Madagascar with children aged 0–6. Approximately 18 months into the implementation of these interventions, we find evidence that households in the behaviorally enhanced arms undertake more desirable parenting behaviors, interact more with their children, prepare more (and more diverse) meals at home, and report lower food insecurity than households that received only cash, and children in these arms perform better than children from households in the cash-only arm on several measures of socio-cognitive development including language learning and social skills. This is promising evidence that behavioral interventions can add significant value to cash transfer programs that aim to improve human development outcomes. (AEARCTR-0000957).
Context: Demographic differences can influence the safety and effectiveness of medical devices; however, clinical trials of devices for adults have historically underrepresented women, people of ...color, and patients over age 65. The US Food and Drug Administration (FDA) Safety and Innovation Act became law in 2012, encouraging greater diversity and subgroup analyses. In 2013, the FDA reported that there was diversity in clinical trials considered "pivotal" for approval decisions and that subgroup analyses were conducted for most applications for the highest-risk medical devices. However, the FDA's report did not specify whether analyses included sufficient numbers to be meaningful, whether analyses were conducted for most major subgroups, or whether analyses included safety, effectiveness, or accuracy. Methods: We examined publicly available documents for all 22 medical devices that the FDA designated "highest risk" or "novel," were reviewed through the premarket approval pathway, and were scrutinized at FDA public meetings from 2014 to 2017. We evaluated patient demographics and subgroup analyses for all pivotal trials. Findings: Only 3 (14%) of the devices provided subgroup analyses for both effectiveness and safety or both sensitivity and selectivity for gender, race, and age. However, 55% of the devices reported both of those subgroup analyses for at least 1 of the 3 subgroups. Whether analyses were reported or not, the number of patients in most subgroups was too small to draw meaningful conclusions. Subgroup analyses were more likely to be reported to the FDA's Advisory Committees than in the FDA's public reviews or labeling. Conclusions: Despite a law encouraging more diversity and subgroup analyses in pivotal trials used as the basis for FDA approval, the results of our study indicate relatively few subgroup analyses are publicly available for the highest-risk and novel medical devices. The lack of subgroup analyses makes it impossible to inform patients or physicians as to whether many newly approved medical devices are safe and effective for specific demographic subgroups defined by gender, race, and age.
IntroductionFamilies living in extreme poverty require interventions to support early-childhood development (ECD) due to broad risks. This longitudinal cluster randomised trial examines the ...effectiveness of Sugira Muryango (SM), a home-visiting intervention linked to Rwanda’s social protection system to promote ECD and reduce violence compared with usual care (UC).MethodsFamilies with children aged 6–36 months were recruited in 284 geographical clusters across three districts. Cluster-level randomisation (allocated 1:1 SM:UC) was used to prevent diffusion. SM was hypothesised to improve child development, reduce violence and increase father engagement. Developmental outcomes were assessed using the Ages and Stages Questionnaire (ASQ-3) and the Malawi Development Assessment Tool (MDAT) and anthropometric assessments of growth. Violence was assessed using questions from UNICEF Multiple Indicators Cluster Survey (MICS) and Rwanda Demographic and Health Surveys (DHS). Father engagement was assessed using the Home Observation for Measurement of the Environment. Blinded enumerators conducted interviews and developmental assessments.ResultsA total of 541 SM families and 508 UC families were enrolled and included in the analyses. Study attrition (2.0% children; 9.6% caregivers) was addressed by hot deck imputation. Children in SM families improved more on gross motor (d=0.162, 95% CI 0.065 to 0.260), communication (d=0.081, 95% CI 0.005 to 0.156), problem solving (d=0.101, 95% CI 0.002 to 0.179) and personal-social development (d=0.096, 95% CI −0.015 to 0.177) on the ASQ-3. SM families showed increased father engagement (OR=1.592, 95% CI 1.069 to 2.368), decreased harsh discipline (incidence rate ratio, IRR=0.741, 95% CI 0.657 to 0.835) and intimate partner violence (IRR=0.616, 95% CI:0.458 to 0.828). There were no intervention-related improvements on MDAT or child growth.ConclusionSocial protection programmes provide a means to deliver ECD intervention.Trial registration numberNCT02510313.
Early childhood development (ECD) programmes are heralded as a way to improve children’s health and educational outcomes. However, few studies in developing countries calculate the effectiveness of ...quality early childhood interventions. This study estimates the cost and cost-effectiveness of the Sugira Muryango (SM) trial, a home-visiting intervention to improve ECD outcomes through positive parent-child relationships. Cost-effectiveness analysis of ECD interventions is challenging given their potential to have multiple benefits. We propose a cost-effectiveness method using a single outcome, in this case the improvement in cognitive development per home-visit session, as an indication of efficiency comparable across similar interventions. The trial intervention cost US$456 per family. This cost will likely fall below US$200 if the intervention is scaled through government systems. The cost-effectiveness analysis suggests that while SM generated a relatively small impact on markers of early development, it did so efficiently. The observed improvements in cognitive development per home-visit are similar to other home-visiting interventions of longer duration. SM by focusing on the family had benefits beyond ECD, including reductions in violence against children and intermate partner violence, further analysis is needed to include these returns in the economic evaluation.
Over 250 million children under five in the developing world risk not reaching their full development potential because of deficient investments in nutrition, early stimulation, early learning, and ...nurturing care, as well as due to exposure to stress. Investing in young children can be a cost-effective strategy not only to promote a healthier and more prductive population but also as a powerful promoter of opportunity for disadvantaged children.
Early child development (ECD) programs are increasingly combined with targeted cash transfers for poor households to break intergenerational poverty. However, few evidence-based, scalable, and ...sustainable ECD programs that complement cash transfer programs exist in in low- and-middle-income countries. We conducted a cluster-randomized pilot study to assess whether Sugira Muryango, a strengths-based home-visiting intervention to promote child development and prevent violence among children aged 6–36 months, could be delivered by community-based lay workers to poor families participating in Rwanda’s cash-for-work Vision Umurenge Program (VUP). Data collection occurred among 38 families at baseline, endline, and 6 months after the intervention and included child-level (child engagement, caretaking, and health and development), caregiver-level (family unity and mental health) and household-level (water and sanitation practices and family conflict) outcomes. We compared trajectories of Sugira Muryango families vs. families receiving the cash transfer only over time using mixed-effect models. Sugira Muryango children experienced significantly greater ECD engagement than children in control families and marginally significant reductions in exposure to violent disciplinary methods. Sugira Muryango caregivers reported greater shared decision-making between parents and marginally significant improvements in family unity and anxiety. Conflict within intervention households halved between baseline and follow-up. Satisfaction was high. This randomized pilot demonstrates that Sugira Muryango can be delivered by community-based lay workers, improves access to nurturing care and stimulation among children living in poverty, and may reduce intra-family conflict. A large-scale effectiveness study is underway to assess the intervention’s impact on child development and health outcomes.
Highlights
This study describes a randomized pilot study of Sugira Muryango, a strengths-based home-visiting intervention to promote child development among vulnerable children in Rwanda.
The pilot study demonstrated that Sugira Muryango can be successfully delivered by community-based lay workers.
This pilot study also provides evidence that Sugira Muryango improves access to nurturing care and stimulation among children living in poverty and may reduce intra-family conflict.
Conditional cash transfers are a departure from more traditional approaches to social assistance that represents an innovative and increasingly popular channel for the delivery of social services. ...Conditional cash transfers provide money to poor families contingent upon certain behaviour, usually investments in human capital such as sending children to school or bringing them to health centres on a regular basis. They seek both to address traditional short‐term income support objectives and promote the longer‐term accumulation of human capital by serving as a demand‐side complement to the supply of health and education services. Evaluation results reveal that this innovative design has been quite successful in addressing many of the failures in delivering social assistance such as poor poverty targeting, disincentive effects and limited welfare impacts. There is clear evidence of success from the first generation of programmes in Brazil, Colombia, Mexico and Nicaragua in increasing enrolment rates, improving preventive healthcare and raising household consumption. Despite this promising evidence, many questions remain unanswered about conditional cash transfer programmes, including the replicability of their success under different
PDF
