23 Simulations, bootstrapping, permutations tests

23.1 Simulations in proportion estimation

Simulations provide a convenient way to approximate the solution to many problems, especially when the problem is mathematically complex, when we don’t know or there is no simple analytical form of the solution, or when we need to account for specific situations and adjust to the characteristics of the data.

Simulations allow us to estimate outcomes, and after calculating confidence intervals, we can also assess the precision of our estimates.

Example 23.1 Draw 10,000 points from a unit square. Check what percentage of these points are less than one unit away from the origin (the lower-left corner of the square). Based on this, estimate the value of \(\pi\) with 99% confidence. What would you need to do to double the precision of the estimate?

set.seed(124)
# Simulation:
nsim <- 1e4
x <- runif(nsim)
y <- runif(nsim)
w <- x^2+y^2<1
res <- binom::binom.confint(sum(w), nsim, conf.level=0.99, method="wilson")
c(estimate=res$mean*4, lower_bound=res$lower*4, upper_bound=res$upper*4, 
   interval_width = res$upper*4-res$lower*4)
##       estimate    lower_bound    upper_bound interval_width 
##     3.15880000     3.11605030     3.20001302     0.08396272
# Doubling the precision is possible by quadrupling the sample size:
nsim <- 4 * 1e4
x <- runif(nsim)
y <- runif(nsim)
w <- x^2+y^2<1
res <- binom::binom.confint(sum(w), nsim, conf.level=0.99, method="wilson")
c(estimate=res$mean*4, lower_bound=res$lower*4, upper_bound=res$upper*4, 
   interval_width = res$upper*4-res$lower*4)
##       estimate    lower_bound    upper_bound interval_width 
##     3.13390000     3.11249442     3.15492947     0.04243505

23.2 Bootstrapping

Bootstrapping methods are techniques for determining confidence intervals based on simulations from a single sample. In the simplest version of the method:

  1. Multiple samples are generated by sampling with replacement from a single sample ("resampling");

  2. The statistic of interest, which estimates the population parameter, is computed for each of these samples;

  3. A confidence interval is then constructed based on the appropriate quantiles of the generated statistic distribution.

This method typically works at least as well as standard confidence intervals. However, it does require a sufficiently large random sample from the population under study. The method has limited effectiveness if the data come from a highly skewed or otherwise extreme population.

23.3 Permutation tests

Permutation tests are flexible, non-parametric statistical tests based on permutations (understood here as transformations through shuffling values). This approach allows hypothesis testing without assuming a specific data distribution in the population (or homogeneity of variance, etc.).

Test permutacyjny składa się z następujących kroków:

  1. Define the null hypothesis. Depending on the situation, we assume there is no difference between the groups being studied or no association (statistical relationship) between the variables.

  2. Define the alternative hypothesis. In some tests, a one-tailed or two-tailed alternative hypothesis may be used.

  3. Calculate the test statistic (e.g., the difference in means, correlation, etc.) for the original data.

  4. Perform a permutation simulation. The data are shuffled randomly (between groups, between observations), keeping the number of observations in each group intact. For each permutation (a particular combination of the shuffled data), the test statistic is calculated. This creates a permutation distribution of the test statistic. In theory, for small samples, all permutations can be computed; in practice, however, a limited number of random permutations (e.g., 1,000 or 10,000) are generated due to computational constraints.

  5. Based on the permutation distribution, the p-value is calculated, which can then be compared with the chosen significance level \(\alpha\).

23.4 Exercises

Exercise 23.1 Using simulations, estimate the probability that in a group of 23 people, at least two share the same birthday.

Exercise 23.2 For Christmas, a class of 28 wants to exchange gifts with one another. The names of all the students are placed in a box, and each person draws one name. Using simulations, estimate the probability that no one draws their own name.

Exercise 23.3 Assume the population proportion is 0.25. Check if the 95% confidence interval for the proportion, based on a sample of 39, is truly 95%. How many replications are needed to estimate the actual confidence with a precision of plus or minus 0.1 percentage points? Perform the necessary calculations.

Exercise 23.4 Draw a random sample of 100 from a normal distribution and calculate the 95% confidence interval for the mean. Then, draw another 100-element random sample from the same population. What is the probability that the mean from the second sample falls within the confidence interval calculated from the first sample? Estimate this probability using simulations. Contrary to natural intuition, this probability is not 0.95.

Exercise 23.5 Generate 39 numbers from a normal distribution with a mean of 179 and a standard deviation of 7. Based on this synthetic sample:

  • Build a confidence interval for the mean using the standard method (z or t formula),

  • Build a confidence interval for the mean using bootstrapping,

  • Build a confidence interval for the standard deviation using bootstrapping.

Exercise 23.6 Generate 50 pairs of observations from two variables, X and Y, based on a bivariate normal distribution with a correlation of 0.4. Calculate the correlation for the sample.

  • Calculate the confidence interval for the correlation coefficient and perform the appropriate significance test using the formulas presented in the chapter 22.

  • Generate the confidence interval for the correlation coefficient using bootstrapping.

  • Generate the p-value for the correlation significance test using the permutation method.

Exercise 23.7 Use the data from task 11.6 to perform a permutation test to check the null hypothesis of equality of mean travel times to university by bicycle for Sz and Ko. Build a bootstrap confidence interval for the difference in means and for the effect size measure Cohen's d.

Exercise 23.8 Use the data from task16.2 to perform a permutation test.

Exercise 23.9 Use the data from task 22.2 to perform a permutation test for the correlation coefficient. Generate a confidence interval for the correlation coefficient using the bootstrapping method.