Combining expert knowledge with data

Evidence helps inform humanitarian action so that it is impartial, effective and ethical.

The challenges to obtaining and using this evidence are the motivation behind the extremely informative paper “Insufficient evidence? The quality and use of evidence in humanitarian action” written by Paul Knox Clarke and James Dacy for ALNAP*. There are many statistical issues that are raised within this report.

Here I want to focus on just one aspect of the report. This is the question: How do you combine experts’ knowledge with data?

Why is this important? Well, the authors describe some of the difficulties experts face when using incomplete evidence. How can experts link their experience and knowledge with this evidence?

For example, organisations often try to apply the lessons they have learnt from one situation, for example from an evaluation, to another situation. Or else, they may have a needs assessments for one location and want to use this to help inform the needs assessment in another location. But the situations or locations are not exactly the same. Experts fill in the gaps with their knowledge. But this is not carried out in a systematic, or rigorous, manner.

Decision making may also fail to be properly informed by the available evidence. This might be especially true when evidence is incomplete. Experts and decision makers own beliefs, prior knowledge and assumptions may over-ride or compete with the evidence. This can make it difficult to justify decisions and the decision making process. It may be that it is the voice that shouts loudest that prevails.

What is needed is a systematic and formal methodology that includes expert knowledge in data processing and analysis. Then the final outcome will be a synthesis of data and knowledge. This synthesis should reflect the relative importance and precision of all available information.

A Bayesian approach is a way of doing just this.

A Bayesian framework provides a robust way to synthesize knowledge and data.

First, elicitation methods provide strategies for formalising and capturing expert knowledge. The elicitation transforms expert knowledge into probability distributions. Second, the Bayesian framework combines these probability distributions with data. Final estimates of those things that matter and are of interest are then calculated.

One of the nice things about a Bayesian approach is that the elicitation process captures what experts know and what experts don’t know. The more strongly and precisely experts know about something the greater the influence this can have on final estimates. This also makes it possible to explore how different views and beliefs might affect the final estimates of interest.

A typical interpretation of the Bayesian approach is that it is a mechanism for updating someone’s beliefs with data, or new evidence.  This is how we naturally revise our opinion about something when we are given new facts. Depending on how strongly we feel or have experienced something the harder it may be to move us from our initial viewpoint. The more compelling the evidence the more likely it is that this will over-ride or dominate our previous beliefs.

But that’s not all it is useful for. A Bayesian approach makes it possible to estimate something of interest in complex situations with limited data. For example, in an early warning system there may be data for some indicators but others where no data exists at a particular location. But for the missing indicators there may be data from other locations. Experts may have knowledge about how similar or different these locations are to the location of interest.  Elicitation methods can help experts interpret these data and their knowledge to provide a range of potential values for the missing indicator and how likely different values in this range are. These are translated into probability distributions. A Bayesian analysis could then include this information in the outputs of the early warning system.

This is only a very brief description of why a Bayesian framework might be useful. There is much to be done and explored to see if there is potential to use such an approach. Amongst other things there is the question of how to carry out such exercises in a timely manner. I’d be interested to know what you think either in the comments below or if you want to get in touch to explore the issue more.

* I know this paper came out in 2013 and I am somewhat behind. I recently reread it, whilst preparing for my previous post where I touched on many of the issues. I felt there was more to discuss.  In particular I felt that this may not be a topic that is usually associated with statistics or with what statisticians do.