4.2 Examples and applications
4.2.1 Case study: The Challenger disaster
The Challenger disaster occurred on the 28th January of 1986, when the NASA Space Shuttle orbiter Challenger broke apart and disintegrated at 73 seconds into its flight, leading to the deaths of its seven crew members. The accident deeply shocked the US society, in part due to the attention the mission had received because of the presence of Christa McAuliffe, who would have been the first astronaut-teacher. Because of this, NASA TV broadcasted live the launch to US public schools, which resulted in millions of school children witnessing the accident. The accident had serious consequences for the NASA credibility and resulted in an interruption of 32 months in the shuttle program. The Presidential Rogers Commission (formed by astronaut Neil A. Armstrong and Nobel laureate Richard P. Feynman, among others) was created to investigate the disaster.
Figure 4.1: Challenger launch and posterior explosion, as broadcasted live by NBC in 28/01/1986.
The Rogers Commission elaborated a report (Presidential Commission on the Space Shuttle Challenger Accident 1986) with all the findings. The commission determined that the disintegration began with the failure of an O-ring seal in the solid rocket motor due to the unusual cold temperatures (-0.6 Celsius degrees) during the launch. This failure produced a breach of burning gas through the solid rocket motor that compromised the whole shuttle structure, resulting in its disintegration due to the extreme aerodynamic forces. The problematic with O-rings was something known: the night before the launch, there was a three-hour teleconference between motor engineers and NASA management, discussing the effect of low temperature forecasted for the launch on the O-ring performance. The conclusion, influenced by Figure 4.2a, was:
“Temperature data [are] not conclusive on predicting primary O-ring blowby.”
Figure 4.2: Number of incidents in the O-rings (filed joints) versus temperatures. Panel a includes only flights with incidents. Panel b contains all flights (with and without incidents).
The Rogers Commission noted a major flaw in Figure 4.2a: the flights with zero incidents were excluded from the plot because it was felt that these flights did not contribute any information about the temperature effect (Figure 4.2b). The Rogers Commission concluded:
“A careful analysis of the flight history of O-ring performance would have revealed the correlation of O-ring damage in low temperature.”
The purpose of this case study, inspired by Dalal, Fowlkes, and Hoadley (1989), is to quantify what was the influence of the temperature in the probability of having at least one incident related with the O-rings. Specifically, we want to address the following questions:
- Q1. Is the temperature associated with O-ring incidents?
- Q2. In which way was the temperature affecting the probability of O-ring incidents?
- Q3. What was the predicted probability of an incident in an O-ring for the temperature of the launch day?
To try to answer these questions we have the challenger dataset (download). The dataset contains (shown in Table 4.1) information regarding the state of the solid rocket boosters after launch28 for 23 flights. Each row has, among others, the following variables:
fail.field,fail.nozzle: binary variables indicating whether there was an incident with the O-rings in the field joints or in the nozzles of the solid rocket boosters.1codifies an incident and0its absence. On the analysis, we focus on the O-rings of the field joint as being the most determinants for the accident.temp: temperature in the day of launch. Measured in Celsius degrees.pres.field,pres.nozzle: leak-check pressure tests of the O-rings. These tests assured that the rings would seal the joint.
| flight | date | fail.field | fail.nozzle | temp |
|---|---|---|---|---|
| 1 | 12/04/81 | 0 | 0 | 18.9 |
| 2 | 12/11/81 | 1 | 0 | 21.1 |
| 3 | 22/03/82 | 0 | 0 | 20.6 |
| 5 | 11/11/82 | 0 | 0 | 20.0 |
| 6 | 04/04/83 | 0 | 1 | 19.4 |
| 7 | 18/06/83 | 0 | 0 | 22.2 |
| 8 | 30/08/83 | 0 | 0 | 22.8 |
| 9 | 28/11/83 | 0 | 0 | 21.1 |
| 41-B | 03/02/84 | 1 | 1 | 13.9 |
| 41-C | 06/04/84 | 1 | 1 | 17.2 |
| 41-D | 30/08/84 | 1 | 1 | 21.1 |
| 41-G | 05/10/84 | 0 | 0 | 25.6 |
| 51-A | 08/11/84 | 0 | 0 | 19.4 |
| 51-C | 24/01/85 | 1 | 1 | 11.7 |
| 51-D | 12/04/85 | 0 | 1 | 19.4 |
| 51-B | 29/04/85 | 0 | 1 | 23.9 |
| 51-G | 17/06/85 | 0 | 1 | 21.1 |
| 51-F | 29/07/85 | 0 | 0 | 27.2 |
| 51-I | 27/08/85 | 0 | 0 | 24.4 |
| 51-J | 03/10/85 | 0 | 0 | 26.1 |
| 61-A | 30/10/85 | 1 | 0 | 23.9 |
| 61-B | 26/11/85 | 0 | 1 | 24.4 |
| 61-C | 12/01/86 | 1 | 1 | 14.4 |
Let’s begin the analysis by replicating Figures 4.2a and 4.2b and checking that linear regression is not the right tool for answering Q1–Q3. For that, we make two scatterplots of nfails.field (number of total incidents in the field joints) versus temp, the first one excluding the launches without incidents (subset = nfails.field > 0) and the second one for all the data. Doing it through R Commander as we saw in Chapter 2, you should get something similar to:
scatterplot(nfails.field ~ temp, reg.line = lm, smooth = FALSE, spread = FALSE,
boxplots = FALSE, data = challenger, subset = nfails.field > 0)
scatterplot(nfails.field ~ temp, reg.line = lm, smooth = FALSE, spread = FALSE,
boxplots = FALSE, data = challenger)
There is a fundamental problem in using linear regression for this data: the response is not continuous. As a consequence, there is no linearity and the errors around the mean are not normal (indeed, they are strongly non normal). Let’s check this with the corresponding diagnostic plots:
mod <- lm(nfails.field ~ temp, data = challenger)
par(mfrow = 1:2)
plot(mod, 1)
plot(mod, 2)
Albeit linear regression is not the adequate tool for this data, it is able to detect the obvious difference between the two plots:
- The trend for launches with incidents is flat, hence suggesting there is no dependence on the temperature (Figure 4.2a). This was one of the arguments behind NASA’s decision of launching the rocket at a temperature of -0.6 degrees.
- However, the trend for all launches indicates a clear negative dependence between temperature and number of incidents! (Figure 4.2b). Think about it in this way: the minimum temperature for a launch without incidents ever recorded was above 18 degrees, but the Challenger was launched at -0.6 without clearly knowing the effects of such low temperatures.
Instead of trying to predict the number of incidents, we will concentrate on modeling the probability of expecting at least one incident given the temperature, a simpler but also revealing approach. In other words, we look to estimate the following curve:
\[
p(x)=\mathbb{P}(\text{incident}=1|\text{temperature}=x)
\]
from fail.field and temp. This probability can not be properly modeled as a linear function like \(\beta_0+\beta_1x\), since inevitably will fall outside \([0,1]\) for some value of \(x\) (some will have negative probabilities or probabilities larger than one). The technique that solves this problem is the logistic regression. The idea behind is quite simple: transform a linear model \(\beta_0+\beta_1x\) – which is aimed for a response in \(\mathbb{R}\) – so that it yields a value in \([0,1]\). This is achieved by the logistic function
\[\begin{align}
\text{logistic}(t)=\frac{e^t}{1+e^t}=\frac{1}{1+e^{-t}}.\tag{4.1}
\end{align}\]
The logistic model in this case is
\[
\mathbb{P}(\text{incident}=1|\text{temperature}=x)=\text{logistic}\left(\beta_0+\beta_1x\right)=\frac{1}{1+e^{-(\beta_0+\beta_1x)}},
\]
with \(\beta_0\) and \(\beta_1\) unknown. Let’s fit the model to the data by estimating \(\beta_0\) and \(\beta_1\).
In order to fit a logistic regression to the data, go to 'Statistics' -> 'Fit models' -> 'Generalized linear model...'. A window like Figure 4.3 will pop-up, which you should fill as indicated.
Figure 4.3: Window for performing logistic regression.
A code like this will be generated:
nasa <- glm(fail.field ~ temp, family = "binomial", data = challenger)
summary(nasa)
##
## Call:
## glm(formula = fail.field ~ temp, family = "binomial", data = challenger)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -1.0566 -0.7575 -0.3818 0.4571 2.2195
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 7.5837 3.9146 1.937 0.0527 .
## temp -0.4166 0.1940 -2.147 0.0318 *
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 28.267 on 22 degrees of freedom
## Residual deviance: 20.335 on 21 degrees of freedom
## AIC: 24.335
##
## Number of Fisher Scoring iterations: 5
exp(coef(nasa)) # Exponentiated coefficients ("odds ratios")
## (Intercept) temp
## 1965.9743592 0.6592539The summary of the logistic model is notably different from the linear regression, as the methodology behind is quite different. Nevertheless, we have tests for the significance of each coefficient. Here we obtain that temp is significantly different from zero, at least at a level \(\alpha=0.05\). Therefore we can conclude that the temperature is indeed affecting the probability of an incident with the O-rings (answers Q1).
The precise interpretation of the coefficients will be given in the next section. For now, the coefficient of temp, \(\hat\beta_1\), can be regarded the “correlation between the temperature and the probability of having at least one incident.” This correlation, as evidenced by the sign of \(\hat\beta_1\), is negative. Let’s plot the fitted logistic curve to see that indeed the probability of incident and temperature are negatively correlated:
# Plot data
plot(challenger$temp, challenger$fail.field, xlim = c(-1, 30), xlab = "Temperature",
ylab = "Incident probability")
# Draw the fitted logistic curve
x <- seq(-1, 30, l = 200)
y <- exp(-(nasa$coefficients[1] + nasa$coefficients[2] * x))
y <- 1 / (1 + y)
lines(x, y, col = 2, lwd = 2)
# The Challenger
points(-0.6, 1, pch = 16)
text(-0.6, 1, labels = "Challenger", pos = 4)
At the sight of this curve and the summary of the model we can conclude that the temperature was increasing the probability of an O-ring incident (Q2). Indeed, the confidence intervals for the coefficients show a significative negative correlation at level \(\alpha=0.05\):
confint(nasa, level = 0.95)
## 2.5 % 97.5 %
## (Intercept) 1.3364047 17.7834329
## temp -0.9237721 -0.1089953Finally, the probability of having at least one incident with the O-rings in the launch day was \(0.9996\) according to the fitted logistic model (Q3). This is easily obtained:
predict(nasa, newdata = data.frame(temp = -0.6), type = "response")
## 1
## 0.999604Be aware that type = "response" has a different meaning in logistic regression. As you can see it does not return a CI for the prediction as in linear models. Instead, type = "response" means that the probability should be returned, instead of the value of the link function, which is returned with type = "link" (the default).
Recall that there is a serious problem of extrapolation in the prediction, which makes it less precise (or more variable). But this extrapolation, together with the evidences raised by a simple analysis like we did, should have been strong arguments for postponing the launch.
To conclude this section, we refer to a funny and comprehensive exposition by Juan Cuesta (University of Cantabria) on the flawed statistical analysis that contributed to the Challenger disaster.Figure 4.4: The Challenger disaster and other elegant applications of statistics in complex problems.
References
After the shuttle exits the atmosphere, the solid rocket boosters separate and descend to land using a parachute where they are carefully analyzed.↩︎