The quality of fit in Generalized Linear Models (GLMs) is usually quantified by the deviance, or twice the negative log-likelihood. When there’s a high level of noise in the data, it’s difficult to interpret the deviance directly; the lower bound for the deviance doesn’t take into account noise, and is much too low. I had come up with a way of estimating a bound on the accountable deviance, in the same vein as Sahani & Linden (2003), for the case of a logistic regression GLM for our classification image paper. It didn’t make the cut, but I still think it’s interesting and with some beefing up it could be useful in both classification image and reverse correlation experiments.

The idea is to find out the expected deviance of an oracle (EDO), which knows the internal decision process of the observer with perfect accuracy, minus the value of the internal noise. As it turns out, the only thing that matters for the EDO is the distribution of the internal variable (in the case logistic regression, 1/(1+exp(-eta))) across trials. This distribution can be estimated from observer-observer consistency data (repeated trials). I considered three cases: the internal variable has a beta distribution or one of two discrete distribution which I conjecture give lower and higher bounds for the EDO. I could never prove that these conjectured bounds are real bounds but it’s supported by some simulations.

Here’s the appendix (the references are broken, and it will make more sense in the context of our classification image paper).