Learning from Challenges in Measuring Our Work

Like many organizations working towards social change, Aahung is interested in better understanding what works and doesn’t work in our programs. To do this, in 2013 we conducted a survey with young people who had participated in our comprehensive sexuality education (CSE) program in selected schools. We also collected qualitative data from young people in schools through testimonials, focus group discussions, in-depth interviews, and observation of CSE classroom sessions.

When we spoke to young people about their experiences in the program, we found a number of interesting—and positive—things. Firstly, girls shared their experiences of helping their cousins and friends delay marriage after they knew more about their rights and had better communication with their parents and other adults. They also told us that they felt more confident, and that they were better equipped to make smarter and more responsible decisions about their lives. Lastly, they felt more comfortable talking to their parents and friends about difficult topics such as marriage, puberty, street harassment, and mental health issues.

Yet our survey results told a different story. Our results showed that there had been little or no change in students’ knowledge, attitudes, and behavior after they had participated in the program. Positive change was only found in a few outcomes, such as knowledge about pubertal changes, hygiene practices, and sexual violence. This made us question our approach and the types of questions we were asking.

The goals of the program included many of the themes mentioned by the girls in our conversations with them: empowerment, a healthier and more equitable view of gender, increased capacity to make decisions, better communication skills, positive body image, and enhanced self esteem. However, we did not know which questions to include in the survey to find out more about whether these outcomes were being achieved.

We faced other challenges too, of course.  How could we get all of the information we needed without boring young people?  Also, because of high drop-out rates, how could we follow students to measure our program’s impact over time?  One of the hardest questions we faced was how to measure longer-term behavior change such as choice of a marriage partner, ability to negotiate contraception use, and the capacity to delay first pregnancy.

After speaking with my peers at other organizations at a convening on documenting and learning from CSE programs, organized by IWHC and CREA in April 2015, I realized we were not the only ones facing such dilemmas. Many like-minded organizations struggle with these same questions on how to effectively measure their work. Listening to others discuss their approaches and challenges provided me with a lot of insight into improving my own measurement strategies.

I learned that it is important to use a variety of different tools when measuring complex social change. To get the full picture, it may be necessary to move away from traditional methods of evaluation. Participatory methods—in which we help participants critically analyze concepts such as communication, gender, and rights—can be highly effective. Such methods provide high quality information and better reflect the transformation that students in such programs often experience.

An interesting and effective way that we looked at impact was through artwork. Below are examples of paintings done by the young people who participated in the program. This provided more insight into our programs than we anticipated.

‘Mera bhi haq hai mein astronaut banu’ (I also have the right to become an astronaut) In this painting, the artist depicts how girls have the right to be able to choose their careers, especially careers which are considered male-dominated. Interestingly, the artist uses the word ‘Haq’ which means ‘right.’
‘Mera bhi haq hai mein astronaut banu’ (I also have the right to become an astronaut)
In this painting, the artist depicts how girls have the right to be able to choose their careers, especially careers which are considered male-dominated. Interestingly, the artist uses the word ‘Haq’ which means ‘right.’

 

This image illustrates traditional roles of men and women and how young boys and girls envision their future. Even though they start out equally, social expectations will determine the paths their lives take.
This image illustrates traditional roles of men and women and how young boys and girls envision their future. Even though they start out equally, social expectations will determine the paths their lives take.

 

Here, the artist has shown equality and respect between boys and girls, especially in sharing responsibilities.
Here, the artist has shown equality and respect between boys and girls, especially in sharing responsibilities.

We also compiled a video demonstrating some of our findings and illustrating how girls have brought positive change into their own lives and those of others.

In the future, we plan to move out of our comfort zone and experiment with new tools and methods of measuring the impact our programs. This includes body-mapping, a board game, role-playing exercises, and a revised, shorter, and more interactive survey. Because using art was so successful, we plan to continue to use it to gauge the extent to which students have understood, accepted, and internalized the CSE content. We’ve realized measuring social change can happen in many different forms.

Neha Mankani

Neha Mankani

Research, Monitoring and Evaluation ManagerAahung

Aahung is an organization based in Pakistan that helps public and private school teachers integrate quality Comprehensive Sexuality Education (CSE) into the school curriculum. Aahung has been working on CSE for more than a decade with the aim of empowering adolescents by providing them with knowledge, information, and skills on sexual and reproductive health and rights.

One response to “Learning from Challenges in Measuring Our Work

  1. Dear Sir /Madam
    Greetings
    Thanks for sharing your approach to young generation on their issues which reflect to society at large and development of country as they are the future resource of any country.
    We organized ,managed and conducted “Mission Kishoreuday ” program in India in 52 cities -across country in 7-8 month duration.
    We involved Parents,teachers ,students and pediatricians of country in all 52 cities.
    We had extraordinary response to our program . Program is still in demand across country.
    We learned from this need base program that it should run and primary focus should be to train parents,teachers and religious leaders alngwith practicing pediatricians.
    In this program we have own module,PPTs and pamphlet message for parents and teachers .
    Requirement for adolescent is need base program and teaching by IEP-Individualized education program by the same gender.
    Dr JS Tuteja
    Chairperson
    Adolescent health academy-IAP
    2012-2015
    Misson Kishore uday
    2013
    National convener2012
    Difficult adolescent.

Leave a Reply

Your email address will not be published. Required fields are marked *