Evaluating the impact and effectiveness of advocacy

The other week I attended a workshop organised jointly by 3ie, the Water Supply and Sanitation Collaborative Council (WSSCC) and the Children’s Investment Fund Foundation (CIFF) to look at how to evaluate the impact and effectiveness of advocacy.

In this meeting there was a broad definition of the aims of advocacy. The aims included trying to change policy, social norms or individual behaviour knowledge or attitudes.  The reasoning was that if one of these changes it can contribute to changing some of the others.  For example if we think about anti-smoking campaign in the UK, there has been increasing awareness of the health implications of smoking, changes in individual behaviour and social norms and changes in policy which has led to further changes in individual behaviour and social norms.

I found it really interesting to think about the statistical aspects of this topic and how these might be different to those used for impact evaluation in international development….

Measuring the outcomes of advocacy

If the aim of your advocacy is to change policy then your outcome measure is a binary measure depending on whether or not the policy changed (of course there may be more subtle points regarding the policy so you could have a set of indicators relating to different aspects of policy change). In comparison if you wish to change individual behaviour, knowledge, attitudes or social norms there is the potential to use surveys and opinion polls to measure whether a change has happened. By that I mean that you can measure views of many individuals and summarise these – so your outcome measure is more than just a binary response.  This is also the case if your final outcome is not the policy change but the result of the policy change – for example the number of people with access to proper toilets.

Intervention and counterfactuals

Depending on your intervention there may not be a counterfactual to assess what would have happened if there was no intervention.  For example, if the intervention is based on a series of meetings with policy makers then it is not clear what the counterfactual would be. If however the intervention is related to individual behaviour change it may be possible to carry out a RCT and randomly intervene with some individuals, or groups of individuals, and not others.    There are really two things at play here: (1) the amount of control you have over how the environment within which your advocacy interventions happen and (2) how many separate independent interventions are possible.

Assessing the contribution or impact of your advocacy

If you can have lots of control over how multiple interventions are applied, for example in an RCT, it is relatively straightforward to assess the impact of the advocacy – here we are back to more familiar problems of impact evaluation. But advocacy often operates within an environment where you have little control over what else is going on. Other organisations may be advocating for similar or different things. In these situations, it is important to be able to identify (and measure) the other factors that may be influencing the outcomes you are measuring. Without this it is difficult to know attribute the relative contribution of your advocacy. This is the true whether you are measuring the impact of advocacy or the impact of a policy intervention.

Overall

If the advocacy is represented by one main intervention and your measure of success is whether or not a policy has been changed, or introduced, there is very little quantitative work that can be done. You really have a case study which you would need to investigate and understand.  Here, the principle of being able to identify and account for different sources of variability is important to help assess the relative contribution of your advocacy is possible.

If however, you are either able to measure how a single intervention is changing one or more outcomes over time (for example number of toilets being built, or social outcomes) or if there are multiple independent interventions that are occurring there are then possibilities for using appropriate statistical methods and principles to help understand the relative contribution of the advocacy. For example RCTs, designed surveys (such as case control studies), natural experiments, observational studies and time series analyses might be useful depending on the amount of control you have over the environment and the number of independent interventions there are. I have summarised this in the graph below, which I think holds true whether we are talking about advocacy or other types of interventions. At the meeting, Jo Puri, from 3ie, suggested that small-n designs would be appropriate in some cases – I think this would fit in the top left-hand corner of the graph.

I’ll be thinking about this some more, along with others, so watch this space….