SIMSTAT: Statistical Tests

How to perform a statistical test
The correct way to perform any statistical test is first to choose a null hypothesis, decide which test to use, select either a one or two tail alternative if the option is available, fix the significance level, do the test, then accept the outcome. Multiple testing, especially on the same data, is not allowed, since this will necessarily increase the chance of a Type 1 error.

Multiple testing
Although the previous comments are accepted, it must be pointed out that SIMFIT is not a formal statistics package; it is a package for data exploration. So, in order to cut down the number of options and make such data exploration as easy as possible, program Simstat does multiple testing wherever possible. For instance, if you request a t test, Simstat does tests for normality and variance equality before doing the t test. It also reports lower tail, two tail and upper tail p values, displays the results for the unequal variance correction, does a paired test on demand, then allows the same data to be used for a Mann-Whitney-U or Kolmogorov-Smirnov 2-sample test. However, after the stage of preliminary data exploration is over, there is nothing to stop you confining your analysis to just one of the tests where several are offered. Alernatively, you can use the Bonferroni principle whereby, with n tests, rejection is only indicated if p is less than alpha/n.

The 1-sample t test
You input a single sample and your estimate for the theoretical i.e. population mean mu_0. A Shapiro-Wilks test for normality is done followed by a 1-sample t test to see if the sample mean differs significantly from the theoretical mean. You can then change mu_0 interactively or read in a new sample.

The 1-sample Kolmogorov-Smirnov test
Use this option, from SIMSTAT or from the individual dedicated programs, to test if your sample is consistent with one of the standard distributions, e.g. beta, binomial, chi-square, exponential, F, gamma, lognormal, normal, Poisson, t, uniform or Weibull. You can allow the program to estimate the parameters or you can fix the parameters for the null hypothesis, but you should remember that the test is much more reliable and powerful when the parameters are supplied independently rather than being estimated from the sample. Try the test file normal.tf1 and select the normal distribution. Always examine the best-fit curve to the cdf to assess goodness of fit. Note that this option will refuse to analyse samples where values violate the model assumptions, e.g. supposed beta distribution data with negative values or values greater than unity.
Note that, using this test for testing data for conformity with an assumed distribution, is usually only reliable with fairly large sample sizes; say greater than twenty.

The 1-sample Shapiro-Wilks test for a normal distribution
This is a very useful test for a nomal distribution using the sample mean and variance and the expected normal scores, but it can only be used for sample sizes less than 2000. The procedure is to supply a sample vector, then the routine does a Kolmogorov-Smirnov test using the parameters estimated from the sample, and a Shapiro-Wilks test. After these tests have been done, goodness of fit can be explored by plotting the sample cdf and histogram, as well as a normal scores plot, which should be approximately linear if the sample does come from a normal distribution.

The 1-sample Dispersion and Fisher exact tests for a Poisson distribution
This is an additional procedure to test if a sample of nonnegative integers are consistent with a Poisson distribution. The Fisher exact test can only be done successfully with fairly small samples and mean values. After testing, a histogram is displayed to compare the data with best-fit theory. ÿ

The 2-sample paired/unpaired t test
You input two samples and get the results for a t test, and also you are given the results corrected for unequal variances. Note that tests for normality and variance equality precede the t test and only if these are satisfactory should you consider relying on the t test. If the two sample sizes are equal, since they represent paired data, you can also do a paired t test on request. Try the test files ttest.tf2/ttest.tf3 and also ttest.tf4/ttest.tf5 for a paired test and consult program TTEST for more details about the t distribution.

The 2-sample Kolmogorov-Smirnov test
This nonparameteric test can also be done when comparing two samples for equality of medians, but it is only approximate and not usually so reliable as the Mann-Whitney U test. Try the same test files recommended for a t test.

The 2-sample Wilcoxon-Mann-Whitney-U test
This nonparameteric test should always be done when comparing two samples for equality of medians. Note that, as with many other tests, the p-values calculated by Simstat are exact, not approximate, and they are given for upper, lower and two tail test statistics, so it is up to you to decide which to use. Try the same test files recommended for a t test.

The Wilcoxon pared-sample signed-ranks test
Just as Wilcoxon-Mann-Whitney-U test is the nonparametric analogue of the t test, this is the nonparametric analogue of the paired t test. If your data are normally distributed with the same variance but possibly different means, then the more powerful t and paired t tests should be used. However, if you have counts, scores, proportions, percentages, or any other type of non-normal data, then these tests are the safer alternatives. Note that ties degrade the performance of these tests, and it is usual to set the test median to zero and also to suppress zero differences in this test.

The chi-square and Fisher Exact Tests
Use this to test for homogeneity in a m by n contingency table and consult program CHISQD for more details about the chi-square distribution. Simply supply the contingency table as a m by n matrix as, for instance, in the test files chisqd.tf4 and chisqd.tf5. Note that contraction is used for sparse matrices and the Fisher Exact cell p-values are only calculated for 2 by 2 matrices with modest row sums. The Yates correction will be used where appropriate. A log-linear model can also be fitted to contingency tables. This involves the automatic generation of dummy indicator variables and equations of constraint to make both the sum of row and the sum of column coefficients equal to zero, followed by fitting a generalized linear model with Poisson error and log link.

The McNemar test
This test is used when it is suspected that data are intentionally paired. The row categories and column categories are the same, but the cell frequencies (which must be nonnegative integers) refer to frequencies with natural pairing. Data must be in a square matrix format, as in the test file mcnemar.tf1.

The Cochran Q test
For this you need a matrix of n values across (either 0 or 1) for groups and k blocks down for the dichotomous repeat measures data. You can put successive integers (1 to k) down the first column to identify the cases if you want and they will be ignored, as will any rows that add up to 0 or n (i.e. all 0 or all 1).

Nonparametric Tests
In addition to the Kolmogorov-Smirnov (1 and 2 sample), Mann-Whitney U, Kruskal-Wallis, Friedman, Kendall tau, Spearman rank and chi-square tests, the sign test and run test can also be done interactively by entering the numbers of positives, negatives and runs (instead of sample values as in exhaustive analysis). Additional tests for goodness of fit, such as runs up and down, and analysis of two samples (e.g. best-fit and theory), can be done using program RSTEST, which also provides the median test, Mood's and David's tests for equal dispersion, and Kendall's coefficient of concordance.

The F test for excess variance
You input the (weighted) sums of squares (Q1, Q2), the number of parameters estimated (M1, M2) and the number of experimental points (NPTS). Then the program calculates the F test statistic, appropriate p-values and critical values, and outputs comments about which of the two models should be accepted on the principle of parsimony for a nested set.

The Mantel-Haenszel Test
This test and the related analysis of censored survival data by fitting survival models, such as the Weibull distribution, can be done using program GCFIT.

Back to Help Menu or End Help