Probabilities from Experts

(27th-May-2020)

• The use of pseudocounts also gives us a way to combine expert opinion and data. Often a single agent does not have good data but may have access to multiple experts who have varying levels of expertise and who give different probabilities.

• There are a number of problems with obtaining probabilities from experts:

• experts' reluctance to give an exact probability value that cannot be refined,

• representing the uncertainty of a probability estimate,

• combining the numbers from multiple experts, and

• combining expert opinion with actual data.

• Rather than expecting experts to give probabilities, the experts can provide counts. Instead of giving a real number such as 0.667 for the probability of A, an expert can give a pair of numbers as ⟨n,m⟩ that can be interpreted as though the expert had observed n A's out of m trials. Essentially, the experts provide not only a probability but also an estimate of the size of the data set on which their opinion is based.