SIMSTAT: Statistical Tests
How to perform a statistical test
The correct way to perform any statistical test is first to choose
a null hypothesis, decide which test to use, select either a
one or two tail alternative if
the option is available, fix the significance level, do the test,
then accept the outcome. Multiple testing, especially on the same data,
is not allowed, since this will necessarily increase the chance of
a Type 1 error.
Multiple testing
Although the previous comments are accepted, it must be pointed out
that SIMFIT is not a formal statistics package; it is a package for
data exploration. So, in order to cut down the number of options and make
such data exploration as easy as possible, program Simstat does
multiple testing wherever possible. For instance, if you request a t test,
Simstat does tests for normality and variance equality before
doing the t test. It also reports lower tail, two tail and upper tail
p values, displays the results for the unequal variance correction,
does a paired test on demand, then allows the same data to be used
for a Mann-Whitney-U or Kolmogorov-Smirnov 2-sample test. However, after the
stage of preliminary data exploration is over, there is nothing to stop you
confining your analysis to just one of the tests where several are offered.
Alernatively, you can use the Bonferroni principle whereby, with n tests,
rejection is only indicated if p is less than alpha/n.
The 1-sample t test
You input a single sample and your estimate for the theoretical
i.e. population mean mu_0. A Shapiro-Wilks test for normality is
done followed by a 1-sample t test to see if the sample mean
differs significantly from the theoretical mean. You can then
change mu_0 interactively or read in a new sample.
The 1-sample Kolmogorov-Smirnov test
Use this option, from SIMSTAT or from the individual dedicated
programs, to test if your sample is consistent with one of
the standard distributions, e.g.
beta, binomial, chi-square, exponential, F, gamma, lognormal, normal,
Poisson, t, uniform or Weibull.
You can allow the program to estimate the parameters or you can
fix the parameters for the null hypothesis, but you should remember
that the test is much more reliable and powerful when the parameters
are supplied independently
rather than being estimated from the sample.
Try the test file normal.tf1 and select the normal distribution.
Always examine the best-fit curve to the cdf to assess goodness of fit.
Note that this option will refuse to analyse samples where values
violate the model assumptions, e.g. supposed beta distribution
data with negative values or values greater than unity.
Note that, using this test for testing data for conformity with an
assumed distribution, is usually only reliable with fairly large
sample sizes; say greater than twenty.
The 1-sample Shapiro-Wilks test for a normal distribution
This is a very useful test for a nomal distribution using the
sample mean and variance and the expected normal scores, but it can
only be used for sample sizes less than 2000. The procedure is to supply
a sample vector, then the routine does a Kolmogorov-Smirnov test
using the parameters estimated from the sample, and a Shapiro-Wilks
test. After these tests have been done, goodness of fit can be
explored by plotting the sample cdf and histogram, as well as a
normal scores plot, which should be approximately linear if the
sample does come from a normal distribution.
The 1-sample Dispersion and Fisher exact tests for a Poisson distribution
This is an additional procedure to test if a sample of nonnegative
integers are consistent with a Poisson distribution. The Fisher
exact test can only be done successfully with fairly small samples
and mean values. After testing, a histogram is displayed to compare
the data with best-fit theory. ÿ
The 2-sample paired/unpaired t test
You input two samples and get the results for a t test, and also you
are given the results corrected for unequal variances. Note that
tests for normality and variance equality precede the t test and
only if these are satisfactory should you consider relying on
the t test. If the two sample sizes are equal, since they represent
paired data, you can also do a paired
t test on request. Try the test files ttest.tf2/ttest.tf3 and also
ttest.tf4/ttest.tf5 for a paired test and consult program TTEST for
more details about the t distribution.
The 2-sample Kolmogorov-Smirnov test
This nonparameteric test can also be done when comparing two samples
for equality of medians, but it is only approximate and not usually so
reliable as the Mann-Whitney U test. Try the same test files recommended
for a t test.
The 2-sample Wilcoxon-Mann-Whitney-U test
This nonparameteric test should always be done when comparing two samples
for equality of medians. Note that, as with many other tests, the
p-values calculated by Simstat are exact, not approximate, and they
are given for upper, lower and two tail test statistics, so it is up to
you to decide which to use.
Try the same test files recommended for a t test.
The Wilcoxon pared-sample signed-ranks test
Just as Wilcoxon-Mann-Whitney-U test is the nonparametric analogue of
the t test, this is the nonparametric analogue of the paired t test. If
your data are normally distributed with the same variance but possibly
different means, then the more powerful t and paired t tests should be
used. However, if you have counts, scores, proportions, percentages,
or any other type of non-normal data, then these tests are the safer
alternatives. Note that ties degrade the performance of these tests,
and it is usual to set the test median to zero and also to suppress
zero differences in this test.
The chi-square and Fisher Exact Tests
Use this to test for homogeneity in a m by n contingency table and
consult program CHISQD for more details about the chi-square
distribution.
Simply supply the contingency table as a m by n matrix as, for
instance, in the test files chisqd.tf4 and chisqd.tf5. Note that
contraction is used for sparse matrices and the Fisher Exact cell
p-values are only calculated for 2 by 2 matrices with modest row
sums. The Yates correction will be used where appropriate.
A log-linear model can also be fitted to contingency tables.
This involves the automatic generation of dummy indicator variables
and equations of constraint to make both the sum of row and the sum of
column coefficients equal to zero, followed by fitting a generalized
linear model with Poisson error and log link.
The McNemar test
This test is used when it is suspected that data are intentionally paired.
The row categories and column categories are the same, but the cell
frequencies (which must be nonnegative integers) refer to frequencies
with natural pairing. Data must be in a square matrix format, as in
the test file mcnemar.tf1.
The Cochran Q test
For this you need a matrix of n values across (either 0 or 1) for
groups and k blocks down for the dichotomous repeat measures data.
You can put successive integers (1 to k)
down the first column to identify the cases if you want and they will
be ignored, as will any rows that add up to 0 or n (i.e. all 0 or all 1).
Nonparametric Tests
In addition to the Kolmogorov-Smirnov (1 and 2 sample), Mann-Whitney U,
Kruskal-Wallis, Friedman, Kendall tau, Spearman rank and chi-square
tests,
the sign test and run test can also be done interactively by
entering the numbers of positives, negatives and runs (instead
of sample values as in exhaustive analysis).
Additional tests for goodness of fit, such as runs up and down,
and analysis of two samples (e.g. best-fit and theory),
can be done using program RSTEST, which also provides the median
test, Mood's and David's tests for equal dispersion, and Kendall's
coefficient of concordance.
The F test for excess variance
You input the (weighted) sums of squares (Q1, Q2), the number of
parameters estimated (M1, M2) and the number of experimental points
(NPTS). Then the program calculates the F test statistic, appropriate
p-values and critical values, and outputs comments about which
of the two models should be
accepted on the principle of parsimony for a nested set.
The Mantel-Haenszel Test
This test and the related analysis of censored survival data
by fitting survival models, such as the Weibull distribution,
can be done using program GCFIT.
Back to Help Menu or End Help