Despite the surge in interest among foundations and nonprofit organizations in measuring effectiveness, there is a growing consensus that we do not always measure what matters.  Chaletta Huertas of GMA Foundations hosted a discussion to explore what does and does not work in measuring outcomes in out-of-school-time programs.

The November 18th briefing featured Manosi Datta, program director of the Build the Out-of-School Time Network (BOSTnet), a technical assistance provider for dozens of programs throughout the state, Dr. John Zuman, executive director of the Intercultural Center for Research in Education, which conducts small- and large-scale evaluations in the U.S. and internationally, and Dr. Christine Green, president of the Trefler Foundation.

The briefing underscored the persistent criticism that most organizations, including foundations, focus their efforts on monitoring performance rather than on creating a culture of learning.

“We spend 90 percent of our time on monitoring…determining if outcomes have been achieved, but only 10 percent looking at the results and learning from them. True evaluation is a reflective process that helps stakeholders to analyze and use data to inform decisions for program improvement,” said Dr. Zuman.

Ms. Datta confirmed this point and noted that while more nonprofit organizations are using the many data collection tools available, nonprofits or foundations rarely have a chance to use their data to spur programmatic and organizational changes.

To end this imbalance, the speakers challenged funders to move from requiring a grant report (a monitoring instrument) to encouraging efforts that integrate outcomes measures with decision making and strategic planning.

This paradigm shift in evaluation may require foundations to support intermediary organizations over several years help to nonprofit leaders approach evaluation in the manner of many for-profit companies that use monitoring and evaluation as a tool for continuous improvement.

The practice of evaluation also may require additional training for nonprofit leaders. Dr. Green learned from the Trefler foundation’s work to support the leadership development of nonprofit directors that, while they clearly understand the importance of both monitoring and evaluation, they do not view them as a priority for professional development.

“If we want to build an out-of-school-time field, we must equip our leadership…and include intentional trainings on how to build a culture of evaluation.”

In addition, the discussion touched on the following related topics:

• Recent efforts to measure quality at point-of-service
• The pitfalls of translating data to program changes/public policy initiatives
• How a foundation’s evaluation efforts and risk tolerance may vary if its focus is on building the field of out-of-school-time or simply supporting individual programs/organizations
• How differently a foundation’s portfolio and activities would look if the focus were less on monitoring and more on building a culture of learning among grantees and within the foundation itself.

Speakers:

Manosi Datta, Build the Out-of-School Time Network (BOSTnet)
Dr. Christine Green, Trefler Foundation
Dr. John Zuman, Intercultural Center for Research in Education (INCRE)