Reflections from SIMNA Ethics in Impact Measurement Event (Sydney, March 2019)

At the end of March Social Impact Measurement Network Australia (SIMNA) held an event focused on ethics in impact measurement. Our Impact Specialist, Andrew Callaghan, was invited to be on a panel to share his experiences and advice on how to tackle a crucial yet complex part of any organisation’s impact measurement approach. Andrew reflects on the event discussions:

Members of the panel had some differences in opinion regarding a couple of key points on ethics, which really stood out for me. These two points were:

1.     Whether there is a difference between research and program evaluation?

2.    Do you need ethics approval from an official ethics committee to carry out program evaluation?

The first question around what the difference is, if any, between academic research and program evaluation. This is a question which I have extensively researched and truthfully grappled with for at least 12 months in a previous role before I was confident in where I stood on the issue.

The view of another member of the panel was that fundamentally if you plan on publishing the results of your evaluation anywhere, it enters the body of work available on the subject in the public domain as grey literature*, and therefore there is no difference from an ethics perspective. If you accept this view, the argument would go that you need ethical approval from an official ethics committee for every evaluation you carry out including annual impact measurement activities.

*Grey literature are materials and research produced by organizations outside of the traditional commercial or academic publishing and distribution channels. (https://en.wikipedia.org/wiki/Grey_literature)

Personally, I disagree with this view for the following reasons.

Firstly, research starts off with a hypothesis which the researcher is looking to prove/disprove where an impact measurer is looking to measure the effectiveness of an intervention which has been implemented. It could be perceived as a subtle difference, yet it is actually a fundamental difference.

I will point out that if your intervention is not based on sound academic research which has already proven your intended intervention to be effective then you may indeed be carrying out research.

This infographic is a great illustration of the differences and therefore my argument as to why they are different activities.

On the second question of whether you need ethics approval from an official ethics committee to carry out program evaluations, my answer is, I don’t believe you necessarily do.

Other panel members disagree, and they believe that every program evaluation needs the oversight of an official ethics committee. The risk of doing damage to participants or releasing false evidence in the grey literature is too great to not to have it.

There are many factors which may mean you do require ethics approval,  such as; it is a condition of your funding or grant, you are working with highly vulnerable participants or your evaluation is likely to address sensitive subjects which may cause harm to the participants.

This is why ethics is a really complex area where the answer is never straight forward.

Outside of academic institutions, there is only one organisation, which I am aware of in Australia, who can provide recognised ethics approval. The cost and time involved in getting ethics approval in this way can be worth it, depending on the budget for evaluation, but for most organisations, unless specifically provided extra funding by a funder/grant to get ethics approval, it is simply too expensive and time-consuming, no matter the size of the organisation.

Speaking from my own experience, the reality is that most organisations do not have the knowledge, experience or resources to gain ethics approval. These organisations at the same time are being asked to measure their impact, report on outcomes and make strategic decisions based on evidence of impact.

We need to be pragmatic in our expectations and try to simplify the decision process for them.

These differing views on ethics from the SIMNA event all have merit, and the public discussion was definitely worthwhile. But if we are to define program evaluations as research, then funders need to provide the extra money and time required, to gain ethics approval at the beginning of programs. Or we need to accept that program evaluation is not research and the realities of running programs mean it will be rare for organisations to have the time, expertise or money to have ethics board approval for program evaluations.

For those grappling with what to do about ethics in their own work, my advice would be to ensure you have considered and agreed on your ethical approach to evaluation internally, and that it meets the expectations of your stakeholders.

I would like to thank SIMNA for inviting me to speak and be part of the panel. It made me reflect on my own position on these questions and even doubt myself at times during the panel discussion! But I truly believe there is a difference between research and program evaluation, and that ethics approval from an ethics board is not necessary for all program evaluations.

READ: DO I MEASURE IMPACT OR EVALUATE?