Consider a litigation matter involving allegations of defects among a large population of units. The units might be homes within a residential development, manufactured products, building components, insurance claims, or other items relevant to the matter.
Inspecting every unit may be impractical. Instead, a random sample of 47 units is selected from the population and subjected to a defined inspection protocol.
Following the inspections, statistics are calculated from the results. These might include the proportion of units exhibiting a particular condition, the average number of defects per unit, the median repair cost, or some other measure relevant to the questions being addressed.
But an important statistical question remains:
How much would the results have changed if a different 47 units had been randomly selected for inspection?
This question concerns sampling uncertainty.
Traditional statistical methods often quantify this uncertainty using mathematical formulas based on probability models and assumptions about the underlying population. In many situations, those methods are entirely appropriate. In others, the statistic of interest may be complicated, its sampling distribution may be unknown, or the assumptions required for a conventional analytical method may be questionable.
One remarkably useful alternative is bootstrapping.
Bootstrapping uses the observed inspection results themselves to estimate how much a statistic might vary from one random sample to another.
Are inspection results being used to draw conclusions about a larger population? Request a Project Assessment with Praxis Reliability Consulting to evaluate whether the sampling design, inspection data, and statistical methods support those conclusions.
In this article, we examine how bootstrapping works and how it can help quantify uncertainty when inspection results are used to make statistical inferences about a larger population.
From 47 Inspections to a Population
Suppose a population contains hundreds or thousands of units, but only 47 units are randomly selected and inspected.
Assume that a particular condition is identified in 29 of those units. The observed proportion is therefore:
p^=2947=0.617\hat{p}=\frac{29}{47}=0.617
or approximately 61.7%.
That calculation is straightforward.
The more difficult question is whether 61.7% should be interpreted as the proportion of the entire population having the condition.
It should not—at least not without considering uncertainty.
The 47 inspected units represent only one possible random sample from the population. Had a different set of 47 units been selected, perhaps 27 would have exhibited the condition. Another random sample might have contained 31.
Each sample would produce a somewhat different estimate.
This sample-to-sample variation is a fundamental part of statistical inference.
What Is Bootstrapping?
Ideally, we might investigate this sampling variation by repeating the entire inspection program.
We could randomly select another 47 units, inspect them, calculate the statistic, and then repeat the process thousands of times.
In litigation, that is obviously unrealistic.
The inspections may be expensive, destructive, time-consuming, or no longer reproducible. We generally have only the 47 units that were actually inspected.
Bootstrapping provides a computational way to approximate this repeated-sampling process.
Starting with the inspection results from the 47 units:
- Randomly select 47 units with replacement from the original 47 inspection results.
- Calculate the statistic of interest from this new sample.
- Record the result.
- Repeat the procedure thousands of times.
- Examine the distribution of the resulting statistics.
Each simulated set of 47 observations is called a bootstrap sample, and the collection of calculated statistics forms the bootstrap distribution.
Why Sample “With Replacement”?
The phrase with replacement is essential.
Suppose the 47 inspected units are numbered 1 through 47.
A bootstrap sample might begin:
17, 4, 31, 17, 42, 8, 31,…17,\;4,\;31,\;17,\;42,\;8,\;31,\ldots
Notice that Units 17 and 31 have appeared more than once.
Other units may not appear at all.
After each unit is selected, it is effectively returned to the pool and is eligible to be selected again. This continues until a new sample containing 47 observations has been constructed.
Why do this?
If we selected all 47 units without replacement, every resample would contain exactly the same 47 inspection results. The statistic would therefore remain unchanged.
Sampling with replacement introduces the variation necessary to approximate what might have occurred had another random sample of 47 units been selected from the population.
What Does the Bootstrap Tell Us?
Suppose we generate 10,000 bootstrap samples, each containing 47 observations.
For every bootstrap sample, we calculate the proportion of units exhibiting the condition.
We might obtain estimates such as:
0.574,0.638,0.596,0.681,0.617,…0.574,\quad 0.638,\quad 0.596,\quad 0.681,\quad 0.617,\ldots
After 10,000 repetitions, these estimates form a distribution.
That distribution provides an empirical representation of the sampling uncertainty associated with the statistic.
If the bootstrap estimates are tightly concentrated around the observed 61.7%, the estimate is relatively precise.
If the estimates are widely dispersed, the inspection results provide less precise information about the population.
This is one of the central advantages of bootstrapping: rather than reporting only a single estimate, we can investigate how stable that estimate is to sampling variation.
From a Point Estimate to an Interval
A statement such as:
“61.7% of the inspected units exhibited the condition.”
is a statement about the sample.
A different statement is:
“Approximately 61.7% of the population exhibits the condition.”
The second statement is an inference about units that were not inspected.
That inference requires consideration of uncertainty.
Bootstrapping can be used to estimate a standard error and construct a confidence interval around the sample statistic.
For example, after generating thousands of bootstrap samples, we can examine the central portion of the bootstrap distribution to establish a range of plausible values for the population parameter.
This provides something much more informative than a single percentage.
Instead of simply asking:
What proportion of the inspected units exhibited the condition?
we can begin addressing the more important inferential question:
What do the inspection results reasonably tell us about the larger population, and how uncertain is that conclusion?
Why Bootstrapping Can Be Valuable in Litigation
Inspection data encountered in litigation are not always as simple as counting defective and non-defective units.
An expert may need to estimate:
- The average number of observed conditions per unit
- Median repair costs
- Differences between categories of units
- Ratios or rates
- Measures of association
- Regression model parameters
- Statistics derived from several inspection measurements simultaneously
For some statistics, well-established analytical formulas exist for calculating standard errors and confidence intervals.
For others, deriving the sampling distribution can become considerably more complicated.
Bootstrapping offers a general computational framework for investigating uncertainty without requiring a unique mathematical derivation for every statistic.
This does not mean that bootstrapping eliminates statistical assumptions or statistical theory. Rather, it provides another method for approximating the sampling behavior of a statistic when conventional analytical methods are difficult or inappropriate.
Bootstrapping Does Not Create 10,000 New Inspections
This distinction is particularly important when presenting bootstrap results in a legal setting.
If 47 units were inspected, the evidentiary data still consist of 47 inspected units.
Generating 10,000 bootstrap samples does not transform those 47 inspections into 470,000 independent observations.
The bootstrap samples are computational resamples of the same observed data.
Their purpose is to evaluate the uncertainty associated with the statistic calculated from the original sample.
Bootstrapping therefore does not increase the original sample size, nor does it compensate for an inadequate sampling design.
Bootstrapping Cannot Rescue a Poor Sample
This may be the most important limitation of bootstrapping in a litigation setting.
The method assumes that the observed sample provides useful information about the population from which it was drawn.
Suppose the 47 units were selected because they were:
- Known to have problems
- Convenient to access
- Identified by the claimants
- Selected from only one geographic region
- Selected using an undocumented or inconsistent procedure
Repeatedly resampling those 47 units will reproduce the characteristics—and potentially the biases—of that original sample.
The bootstrap cannot manufacture representativeness.
A biased sample bootstrapped 10,000 times remains based on the same biased sample.
This is why sampling design must come before statistical analysis.
Before using inspection results to make population-level conclusions, one must first determine whether the units were selected through a sampling process that permits such inference.
Inspection Protocol Matters Too
There is another issue that bootstrapping cannot resolve.
Even a properly selected random sample can produce unreliable information if the inspection protocol itself is inconsistent.
For example:
- Were all 47 units evaluated using the same inspection criteria?
- Were inspectors trained to classify conditions consistently?
- Were some units subjected to different tests?
- Were missing observations handled consistently?
- Did the observational unit remain the same throughout the inspection?
These are questions about measurement and data quality, not sampling uncertainty.
Bootstrapping quantifies uncertainty arising from the observed data. It cannot correct systematic errors introduced through inconsistent inspection procedures.
Statistical Methods Must Match the Question
Bootstrapping is not automatically preferable to conventional statistical methods.
For a simple proportion such as 29 of 47 inspected units, established methods exist for constructing confidence intervals directly from the binomial model.
In such a situation, bootstrapping may primarily serve as an intuitive way to demonstrate the concept of sampling variation.
Its greater value often appears when the statistic becomes more complicated and an analytical expression for its sampling distribution is difficult to obtain.
The appropriate question is therefore not:
“Can we bootstrap these data?”
but rather:
“What statistical method appropriately represents the sampling process, the data, and the question being addressed?”
The Role of Statistical Analysis in Litigation
When inspection results are presented in litigation, the numerical calculations are often the easiest part of the analysis.
The more fundamental questions concern how the evidence was generated.
Were the inspected units randomly selected?
Was the sample representative of the population?
Was the inspection protocol defined before the inspections were performed?
Were the observational units and inspection criteria clearly established?
And does the statistical method actually support the population-level conclusion being offered?
At Praxis Reliability Consulting, statistical analyses involving litigation matters, sampling, inspection data, reliability engineering, and statistical consulting are approached by examining the entire process—from population definition and sampling design through data collection, analysis, uncertainty, and interpretation.
The objective is not simply to calculate a statistic. It is to determine what conclusions the available evidence can reasonably support.
Conclusion
Suppose 47 units are inspected from a much larger population.
The observed inspection results provide information about those 47 units.
Statistical inference attempts to determine what those observations tell us about the larger population.
Bootstrapping provides one powerful way to investigate the uncertainty associated with that inference.
By repeatedly resampling the 47 observed inspection results with replacement, recalculating the statistic, and examining its resulting distribution, we can evaluate how sensitive our conclusion is to the particular sample that happened to be selected.
But bootstrapping has an important limitation:
It can quantify uncertainty in a sample. It cannot correct how that sample was obtained.
If population-level conclusions are going to be drawn from inspection data, sound sampling design, consistent inspection procedures, and appropriate statistical analysis remain essential.
Are inspection results being used to support conclusions about an entire population? Contact Praxis Reliability Consulting to discuss whether the sampling design and statistical analysis provide a defensible basis for those conclusions.