What the Dutch can do with prior information (and you too) – to be presented at the Modern Modeling Methods (M3) conference 

Bayes is growing in all disciplines! This is one of the results found by van de Schoot, Winter, Ryan, Zondervan-Zwijnenburg and Depaoli (in press) in an extensive systematic review. There are many different reasons why one might choose to use Bayes (e.g., the use of priors, estimating otherwise intractable models, modeling uncertainty, etc.). They found in this review that the use of Bayes has increased and broadened in the sense that this methodology can be used in a flexible manner to tackle many different forms of questions.

In this symposium we will show a broad range topics that can be tackled by using prior information. We start off with an evaluation of Bayesian estimation for small sample problems, when is it a solution and what are some pitfalls? Thereafter we discuss a new method to judge experts based on their elicited prior beliefs. To provide proof of concept an application is presented ranking regional directors in a large financial institution. To conclude, an innovative way of testing replication of hypothesis using prior predictive p-values is presented and illustrated. Online tools are made available so that you too can start using this method.

Bayesian Structural Equation Models with Small Samples
Authors: Sanne Smid, Dan McNeish, Rens van de Schoot
Presenting author: Sanne Smid

Bayesian estimation is frequently mentioned in the literature as a solution for small sample problems. When the Bayesian analysis settings are adapted to the specific research situation, problems can be overcome and Bayesian estimation can indeed be a solution for small sample problems. However, if just default settings are used, Bayesian estimation can perform worse than Maximum Likelihood estimation: default non-informative priors can become highly informative priors when samples are small and variance estimates can become extremely high. This are some of the main findings of the systematic review we carried out. In this review, we included papers in which a simulation study was used to investigate and compare the performance of Bayesian parameter estimation to Maximum Likelihood estimation in structural equation models with small sample sizes. The goal was to investigate whether Bayes should be used instead of Maximum Likelihood for SEM when the sample size is small. A total of n = 4977 records was identified in different searches. After removal of duplicates, n = 3548 abstracts were screened and n = 475 full-text articles were retrieved. In the end, we were left with n = 24 papers, in which a total of n = 29 simulation studies are described that met our inclusion criteria. We conclude that Bayesian estimation can have advantages for small samples in comparison to Maximum Likelihood estimation. However, researchers should always adapt the settings for the analysis to the specific research situation. This also includes thinking about priors: never rely on default priors when the sample size is small.

Keywords: Bayesian estimation, small samples, systematic review

Using the Data Agreement Criterion to Rank Experts’ Beliefs
Authors: Duco Veen, Diederick Stoel, Rens van de Schoot
Presenting Author: Duco Veen

Experts possess information and predictions can be a manifestation of this knowledge. In this presentation the use of priors to evaluate the relative merit of multiple sources of expert information in a decision making process will be discussed. We show how to elicit experts’ beliefs and represent these in the probabilistic form of a prior distribution, one for each expert. Representing beliefs in the form of a prior distribution allows experts to express appropriate (un)certainty, capturing aleatory and epistemic uncertainty. For this purpose a two-step digitized elicitation procedure was designed and evaluated using a pilot study and user feasibility study. The Data Agreement Criterion (DAC), a prior-data conflict measure as developed by Bousquet (2008), was adapted to allow simultaneous evaluations of multiple priors, i.e., the experts’ beliefs, in the light of new data. We show that the adapted version of the DAC can be used to rank experts based on their level of prior-data conflict. The resulting ranking can be used in the evaluation of expertise and assist decision makers in quantifying the relative importance sources of expert information. Behavior of the adapted DAC will be evaluated. Proof of concept by means of an empirical study, ranking regional directors in a large financial institution on their predictions of average turnover per professional will be presented. In conclusion, the two-step elicitation procedure and the adapted DAC are shown to be valid for the evaluation and ranking of experts’ beliefs.

Keywords: Bayesian statistics, Data Agreement Criterion, decision making, elicitation, experts, prior-data conflict, ranking

Testing ANOVA replications by means of the prior predictive p-value
Authors: Mariëlle Zondervan-Zwijnenburg, Rens van de Schoot, Herbert Hoijtink
Presenting author: Mariëlle Zondervan-Zwijnenburg

Replication concerns an original study and a new study. The original study leads to hypotheses for the new study. These hypotheses can, for example, concern the ordering of group means, specific differences between group means, or specific values for the group means based on the original study. The current study explains how we can test replication of these hypotheses by means of the prior predictive p-value. We illustrate the application of the prior predictive p-value with an example. Additionally, we explain how to calculate the required sample size for the new study such that power for the replication test is sufficient. Both the replication test, and the sample size calculator are made available as online applications. As such, the current study supports researchers that want to adhere to the call for replication studies in the field of psychology.

Keywords: ANOVA, comparison of means, power analysis, prior predictive p-value, replication study

With many thanks to Duco Veen who coordinated the submission procedure!