Increasingly, communcation experiments are incorporated replication factors for the purpose of controlling confounds and increasing generalizability. If replications are considered to be samples of possible treatment implementations, treating the replication factor as random is more appropriate than treating is as fixed. Study 1 shows that treating sampled replications as a fixed effect leads to potentially serious alpha inflation in the test of the treatment effect while treating sampled replications as random controls alpha at its nominal level. Study 2 addresses a common objection to treating replications as random; the argument that to do so will lead to unacceptably low power in statistical testing. Although experiments with very few replications are likely to be deficient in power, the results of Study 2 establish that power can be improved to an unexpected degree by a relatively modest increase in the number of replications.
Abstract