•Implementation of a values-engaged evaluation approach was examined.•Evaluators explicitly attended to culture, diversity, and equity.•The case setting was the evaluation of a multi-year STEM ...program.•Evaluators were most successful when using formal and informal communication.
Evaluation must attend meaningfully and respectfully to issues of culture, race, diversity, power, and equity. This attention is especially critical within the evaluation of science, technology, engineering, and mathematics (STEM) educational programming, which has an explicit agenda of broadening participation. The purpose of this article is to report lessons learned from the implementation of a values-engaged, educative (Greene et al., 2006) evaluation within a multi-year STEM education program setting. This meta-evaluation employed a case study design using data from evaluator weekly systematic reflections, review of evaluation and program artifacts, stakeholder interviews, and peer review and assessment. The main findings from this study are (a) explicit attention to culture, diversity, and equity was initially challenged by organizational culture and under-developed evaluator–stakeholder professional relationship and (b) evidence of successful engagement of culture, diversity, and equity emerged in formal evaluation criteria and documents, and informal dialogue and discussion with stakeholders. The paper concludes with lessons learned and implications for practice.
While evaluators have many intersecting identities, ethnicity remains the most salient identity for evaluators of colour. As formal graduate training in evaluation continues to expand, so too does ...the number of students from ethnic minoritized populations, who are in need of specialized mentoring and advising. Drawing from previous research on evaluation, higher education literature, and personal reflections from the author, an Afro-Latina faculty member, this practice note outlines five strategies for mentoring and advising evaluation graduate students of colour. These include considering the impact of vicarious trauma; assisting with the facilitation of peer and mentor “squads”; respecting, honouring, and celebrating students’ culture, religion, and families; being vigilant of microaggressions and practicing microvalidations; and developing mentoring competence. Each strategy is presented along with reflections and practical examples for implementation.
Violence, marginalization, oppression, exploitation, erasure, and injustice are cornerstones of the Black experience in the United States. Despite seemingly insurmountable challenges, Black people ...have worked diligently and competently to earn spaces within the present-day professional arena. While the experiences of Black professionals have been investigated in multiple fields, little is known about Black evaluators. This study aimed to investigate Black evaluators’ experiences in an attempt to understand the intersectionality of their identities, roles, and practice and to ensure voices and contributions of Black evaluators are highlighted. We conducted 26 interviews with Black evaluators across academic, government, nonprofit, philanthropic, and private sectors. We found race and advocate for social justice are central to Black evaluators’ identities, roles, and practice. We developed a thematic framework as a result of our analysis and interpretations of the ways in which Black evaluators’ identities impacted their perceptions of their professional role and practice.
Postcolonial theory can be paired with a values-engaged educative evaluation approach to resist imperialism and colonialism in evaluation spaces, de-Westernize conceptualizations of diversity, ...equity, and inclusion, and extend the capacity of culturally responsive evaluation. Beyond the joining of the values-engaged educative approach and postcolonial theory, Collaboration Research Network theories offer another distinct but complementary avenue for exploring structural power imbalance and invisible hierarchies in science, technology, engineering, and mathematics evaluation contexts. Using these three theories, values-engaged educative, Collaboration Research Network, and postcolonial theory, this qualitative study examined the perceptions and experiences of faculty and students at three minority-serving institutions participating in a science, technology, engineering, and mathematics Center. This study aims to bridge theories traditionally used in research with evaluation frameworks to reconceptualize concepts of diversity, equity, and inclusion.
More evaluators have anchored their work in equity-focused, culturally responsive, and social justice ideals. Although we have a sense of approaches that guide evaluators as to how they should attend ...to culture, diversity, equity, and inclusion (DEI), we have not yet established an empirical understanding of how evaluators measure DEI. In this article, we report an examination of how evaluators and principal investigators (PIs) funded by the National Science Foundation's Advanced Technological Education (ATE) program define and measure DEI within their projects. Evaluators gathered the most evidence related to diversity and less evidence related to equity and inclusion. On average, PIs’ projects engaged in activities designed to increase DEI, with the highest focus on diversity. We believe there continues to be room for improvement and implore the movement of engagement with these important topics from the margins to the center of our field's education, theory, and practice.
The Evaluation Journal of Australasia special issue on Values in Evaluation published in December 2019 gave an overview of current discourses associated with valuing in evaluation with the debut of ...'Praxis' papers. This introduction summarizes the three praxis papers in this second Values in Evaluation special issue. The praxis papers contribute to knowledge about the ways in which we can and should engage with values across a variety of evaluation contexts with varying stakeholders.
Examinations of culture and cultural responsiveness within evaluations have gained traction, successes and challenges working with Latino/a/e communities have been well‐documented, and considerations ...for working with this community have been put forth. However, evaluation literature examining culturally responsive evaluation (CRE) and social justice approaches within Latino/a/e contexts is still relatively nascent. In this article, we reflect upon an evaluation of a project at the University of Puerto Rico at Humacao. We present three key lessons learned. (1) Having the right evaluation team is critical. There is a need for team members with evaluation expertise and cultures and lived experiences similar to those of the participants. (2) Developing an understanding of the context is an important and continuously evolving task. (3) Culturally commensurate instruments and reports are a non‐negotiable aspect of culturally responsive evaluation work within Latino/a/e communities. We conclude with actionable strategies when working in similar contexts.
As the field of evaluation continues to expand, so do evaluation course offerings. However, little is known about the backgrounds of evaluation educators, how they go about planning their courses, ...challenges experienced, or how they attain competency as teachers. This study offers an initial understanding of novice evaluation educators’ experience in developing and teaching introduction to evaluation courses within a university setting. Factors that influence decisions around course content and pedagogical approaches are also examined. Data collection methods include dialogic conversation notes, reflective journals, course syllabi, and student evaluative feedback. Results indicate that perspectives on the purpose of evaluation and how evaluation should be taught are influenced by evaluation educator professional affiliation, experience with evaluation, and student audience. Implications for teaching and training in evaluation are discussed.
There has yet to be consensus within the field on whose and what criteria, or values, should be used to judge program worth. Furthermore, evaluators continue to grapple with decisions about which ...stakeholder values are to be included in evaluation, and if evaluators should be advocating or prescribing certain values or not. This introduction very briefly highlights the discourse associated with valuing in evaluation and summarizes the four articles in the Values in Evaluation Praxis section.
In this chapter, we reflect on the aims and gaps of this volume and the positionality of the co‐editors and contributors. We invite readers to consider how education and training for evaluators are ...or should be conceptualized and actualized, and detail future areas for consideration and attention.
