Quality of A Test
Quality of Test: The final section of the Test Net with Cases report is the "Quality of Test" table for binary nodes, or "Test Sensitivity" table for nodes with more than 2 states. These are useful when the output of the network is going to be used to decide an action, with one action corresponding to each state of the node.
As a medical example, the node may be "Disease-A" and have the two states "Present" and "Absent". If, after updating for a case, the network reports "Present", then a particular treatment will be started, but if it reports "Absent" then the treatment won't be started. The question is, at what probability for "Present" should we say that the network is reporting Present? The confusion matrix and error rate discussed above were determined using the maximum likelihood state (i.e. the one with highest belief after updating).
For a binary variable, this means choosing the first state only if its belief is higher than 50%. But if each state has a different cost of misclassification, you may not want the cutoff probability to be 50%. In the medical example, it may be disastrous to not treat a patient who has the disease, but not that serious if he is treated unnecessarily. So you would like the network to report "Present" if the probability of the disease is above some small number, like 2%. It is a matter of trading off the rate of false positives against the rate of false negatives. Ideally you would just convert the network to a decision network, by adding a decision node for the action to be taken and a utility node for the cost of misclassification. However, at the time the network is constructed and being graded as to its usefulness, the utilities may not be known.
The Quality of Test section has performance results for a series of cutoff threshold probabilities (which run vertically in the first column). For each case, the beliefs given by the network are converted to a "prediction". The prediction is "first state" if the belief for the first state is higher than the cutoff probability, and "second state" if it's lower. You may want to change the order of the states, so that the first state is the "positive" one, to better match conventional meanings.
The meanings of the columns are:
Sensitivity = Of the cases whose actual value was the first state,
the fraction predicted correctly.
Specificity = Of the cases whose actual value was the second state,
the fraction predicted correctly.
Predictive Value = Of the cases the network predicted as first state,
the fraction predicted correctly.
Predictive Value Negative = Of the cases the network predicted as
second state, the fraction predicted correctly.
Often this data is summarized with a graph called the ROC (receiver operating characteristic) curve. To use Excel (available from Microsoft) to create the ROC curve from this data, select the whole table (except headings) and while holding down the <Ctrl> key, type tcz. Then open the Excel file called "Graph_ROC.xls" (available from the Norsys ftp site), paste into the indicated cell, and the graph will be drawn. If the node has more than 2 states, instead you will get a Test Sensitivity section. The first number of each "column" is the cutoff threshold probability. The second number of each column is the number of cases whose actual value was the state given at the left hand side of the row, and which the network correctly predicted to be in that state (i.e. its belief was greater than cutoff probability), divided by the total number of cases whose actual value was that state.
It may seem awkward that the cutoff probability changes in strange sized jumps. The reason is that Netica only reports on values for which it was able to gather enough data. So running the test using a greater number of cases generally results in finer divisions of the cutoff column.
Other sections of the Test Net with Cases Report:
Calibration & Times Surprised Table