This paper presents the Bayesian evidence framework for both model-fitting and model-comparison that enables a unified and mathematically sound wayof constructing and training committees of an arbitrary number of non-parameterized models. The main contribution of this paper is that this framework is expanded for recurrent neural networks, which involves analyticallyderiving the gradient and the Hessian of the error function to the weights in the network. We then compare State Space Neural Networks (SSNN), a special type of recurrent neural networks, to Feed Forward Neural Networks (FFNN) and investigate the effect of using the Bayesian framework on both types on a densely used freeway in The Netherlands and to compare the SSNN to the FFNN. On the basis of real data from inductive loops and license plate cameras we find with a cross-validation procedure that for a short time horizon, it is shown that both the Bayesian training and the recurrency do not lead to improvements, but that for the a longer horizon both techniques are beneficial. It is shown that the use of a committee indeed leads to improved performance and the correlation between the evidence factor, which follows from Bayesian model-fitting, with the generalization performance is compared versus the training error and the test error. It is found that the evidence has lower correlation, which is an indication that (1) the dataset may be too small, (2) bias exists in the networks, (3) the mapping between the input and output data is difficult and (4) the approximation of the evidence is imperfect. Future research will need to resolve these issues. However, the Bayesian framework will already be beneficial to more complex problems, and moreover leads to estimations of error bars on the predictions, which may be useful for many applications.
Abstract