PMGE: Blueprint for Culturally Responsive Evaluations

Long known for its cutting-edge studies, the Office of Planning, Research, and Evaluation (OPRE), a department of the Administration for Children and Families (ACF), has required evaluations of its programs to be grounded in the principles of its Program Manager’s Guide to Evaluation (PMGE). Now, 14 years after its 2nd update, Westat has updated the guide making it relevant and accessible for current programs. In addition to being easier to navigate, this 3rd edition offers fresh visuals and new topics, including approaches to integrating equity principles at each phase of evaluation.

Westat’s Allison Hyra, PhD, an Associate Vice President, led the task, and Lila Gutuskey, PhD, a Principal Research Associate, developed culturally responsive and equitable evaluation (CREE) content; both Hyra and Gutuskey are in the Social Policy and Economics Research practice. Along with Westat’s Amanda Hare, PhD, a Principal Research Associate in Education Studies, Hyra and Gutuskey will form a panel to discuss the guide’s development at the 22nd Research and Evaluation Conference on Self-Sufficiency (RECS) in late May 2024 in Washington, DC.

Here, Hyra and Gutuskey discuss the revised guide’s new content, challenges faced in updating it, and the benefits it offers.

Q. What is different in the 3rd edition of the Program Manager’s Guide to Evaluation (PMGE)?

A. Allison Hyra: This PMGE is shorter and stylistically easier to navigate. Each chapter opens with a “What’s Inside?” box describing what the chapter addresses and who might be interested in reading it. We hyperlinked relevant subsections so the reader can access more detailed information, and we’ve included examples throughout describing the application of the techniques we describe. Vitally important is the content Lila developed providing clear, actionable guidance on how to incorporate culturally responsive and equitable evaluation principles into all evaluations. We also updated the guide to share information about technique and approaches that have become standard practice in the last 15 years, such as using administrative data to power evaluations and evidence standards clearinghouses use to determine the credibility of impact evaluation findings.

Q. You mention including approaches to integrate equity principles at each phase of evaluation? Can you provide an example?

A. Lila Gutuskey: What’s great about culturally responsive and equitable evaluation approaches are the variety of ways you can include them. When you’re putting together your evaluation team, you can take a participatory approach and include members from the program’s audience as consultants or advisors.

Q. How did you determine what new topics to include in the PMGE?

A. Allison Hyra: We sought input from a panel of evaluation experts and practitioners and a cadre of consultants as well as from ACF management, program staff, and intermediary organizations that work with ACF programs. To validate what we learned from these professionals, we recruited 4 experts to provide insight on strategic content priorities and identify current and relevant resources. Of course, we brought our own evaluation expertise to the guide as well.

Q. Why does this edition emphasize culturally responsive and equitable evaluation (CREE) principles?

A. Allison Hyra: This aspect of research is a relatively new approach for federally funded evaluations. We know that it is important for evaluators to engage the communities affected by the programs they were studying—in essence, for their research to reflect reality.

Lila Gutuskey: Including CREE principles makes an evaluation better and more accurate. Evaluators who understand how their own cultures, experiences, and biases influence their work can make better decisions. Evaluations that include a variety of perspectives and voices are more thoughtfully designed and result in findings more meaningful and relevant for their audiences.

Q. What sources did you use to develop the CREE content?

A. Lila Gutuskey: We synthesized information from across different resources, including the Centers for Disease Control and Prevention’s Practical Strategies for Culturally Competent Evaluation. Then, we augmented the content with smaller research briefs, OPRE’s expertise, and our own knowledge and experience.

Q. What challenges did you face in updating the PMGE?

A. Allison Hyra: We constantly grappled with balancing giving our readers enough information to get an introduction to a concept or evaluation element with not creating an enormous textbox-size guide that provided a thorough explanation. We also needed to ensure the language was explicit and transparent, that the visuals clearly communicated ideas, and that our multiple writers maintained the same style and language consistency.

Lila Gutuskey: For the CREE content, we really wanted to provide clear and actionable approaches that would be relevant for a long time. Considering how rapid this field is growing, we had to balance how we included emerging techniques with what was known to work.

Q. What benefits does this new edition offer?

A. Allison Hyra: This edition serves as an accurate and reliable foundation for a shared vocabulary about program evaluation. It’s a one-stop shop for learning about the evaluation process, and it offers the basis for program staff to have educated conversations about evaluations with their internal or external evaluation experts. It also hyperlinks to high-quality resources we’ve vetted so readers can learn more about any topic of interest.

Lila Gutuskey: Each chapter provides a variety of concrete ways CREE principles can be woven into the evaluation across its different phases. By providing a variety of CREE approaches, evaluation teams can choose which ways are most relevant and feasible for the context of their program and evaluation. The CREE sections also include real-world evaluation examples and enough additional resources for readers to dig into any topic they find interesting.

Q. What is a key takeaway from the CREE content?

A. Lila Gutuskey: I have 3! A program doesn’t have to be about equity for CREE approaches to be relevant. Including CREE principles in your evaluation can make it better. Evaluators should see program participants and staff as having knowledge and perspectives that will improve the evaluation and engage them as partners in the evaluation process.

Q. What feedback has the guide received from OPRE?

A. Allison Hyra: Although the guide was only published on OPRE’s website in late fall 2023, it made the list of OPRE’s 10 most downloaded resources for all of 2023. We are very happy to have contributed to creating a resource that serves various audiences, including ACF grant recipients, evaluators, technical assistance providers, other federal agencies’ staff, and the general public seeking knowledge on program evaluation.