Like many organizations working towards social change, Aahung is interested in better understanding what works and doesn’t work in our programs. To do this, in 2013 we conducted a survey with young people who had participated in our comprehensive sexuality education (CSE) program in selected schools. We also collected qualitative data from young people in schools through testimonials, focus group discussions, in-depth interviews, and observation of CSE classroom sessions.
When we spoke to young people about their experiences in the program, we found a number of interesting—and positive—things. Firstly, girls shared their experiences of helping their cousins and friends delay marriage after they knew more about their rights and had better communication with their parents and other adults. They also told us that they felt more confident, and that they were better equipped to make smarter and more responsible decisions about their lives. Lastly, they felt more comfortable talking to their parents and friends about difficult topics such as marriage, puberty, street harassment, and mental health issues.
Yet our survey results told a different story. Our results showed that there had been little or no change in students’ knowledge, attitudes, and behavior after they had participated in the program. Positive change was only found in a few outcomes, such as knowledge about pubertal changes, hygiene practices, and sexual violence. This made us question our approach and the types of questions we were asking.
The goals of the program included many of the themes mentioned by the girls in our conversations with them: empowerment, a healthier and more equitable view of gender, increased capacity to make decisions, better communication skills, positive body image, and enhanced self esteem. However, we did not know which questions to include in the survey to find out more about whether these outcomes were being achieved.
We faced other challenges too, of course. How could we get all of the information we needed without boring young people? Also, because of high drop-out rates, how could we follow students to measure our program’s impact over time? One of the hardest questions we faced was how to measure longer-term behavior change such as choice of a marriage partner, ability to negotiate contraception use, and the capacity to delay first pregnancy.
After speaking with my peers at other organizations at a convening on documenting and learning from CSE programs, organized by IWHC and CREA in April 2015, I realized we were not the only ones facing such dilemmas. Many like-minded organizations struggle with these same questions on how to effectively measure their work. Listening to others discuss their approaches and challenges provided me with a lot of insight into improving my own measurement strategies.
I learned that it is important to use a variety of different tools when measuring complex social change. To get the full picture, it may be necessary to move away from traditional methods of evaluation. Participatory methods—in which we help participants critically analyze concepts such as communication, gender, and rights—can be highly effective. Such methods provide high quality information and better reflect the transformation that students in such programs often experience.
An interesting and effective way that we looked at impact was through artwork. Below are examples of paintings done by the young people who participated in the program. This provided more insight into our programs than we anticipated.
We also compiled a video demonstrating some of our findings and illustrating how girls have brought positive change into their own lives and those of others.
In the future, we plan to move out of our comfort zone and experiment with new tools and methods of measuring the impact our programs. This includes body-mapping, a board game, role-playing exercises, and a revised, shorter, and more interactive survey. Because using art was so successful, we plan to continue to use it to gauge the extent to which students have understood, accepted, and internalized the CSE content. We’ve realized measuring social change can happen in many different forms.