Scoring Rule Results & Logarithmic Loss Values
Scoring Rule Results: The third section of the Test Net with Cases report is titled "Scoring Rule Results:" This doesn't just take the most likely state as a prediction, but rather considers the actual belief levels of the states in determining how well they agree with the value in the case file. These results are calculated in the standard way for scoring rules.
For more information see any reference on scoring rules, such as:
Morgan, M. Granger and Max Henrion (1990) Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis, Cambridge Univ. Press, New York.
Pearl, Judea (1978) "An economic basis for certain methods of evaluating probabilistic forecasts" in International J. of Man-Machine Studies, 10, 175-183.
Logarithmic loss values were calculated using the natural log, and are between 0 and infinity inclusive, with zero indicating the best performance. Quadratic loss (also known as the Brier score) is between 0 and 2, with 0 being best, and spherical payoff is between 0 and 1, with 1 being best.
The equations are:
Logarithmic loss = MOAC [- log (Pc)]
Quadratic loss = MOAC [1 - 2 * Pc + sum[j=1 to n] (Pj ^ 2)]
Spherical payoff = MOAC [Pc / sqrt (sum[j=1 to n] (Pj ^ 2))]
where Pc is the probability predicted for the correct state, Pj is the probability predicted for state j, n is the number of states, and MOAC stands for the mean (average) over all cases (i.e. all cases for which the case file provides a value for the node in question).
Other sections of the Test Net with Cases Report:
Calibration & Times Surprised Table