1 Introduction: Why Parallel Simulation?
This little book was written to solve a little problem. A scientist has data or is about to collect some. From it they compute a single number. Namely, an estimate of something they care about: the average effect of a drug, the size of a population, or the risk in a portfolio. The number itself is easy. The hard question is, How trustworthy is it? How much would it change if the experiment were repeated? Is it close to the truth? Answering such questions almost always takes a simulation experiment: generate data from a known model, and watch how that estimator behaves. For example:
Testing New Medicines Before a new drug is approved, researchers estimate how much it slows disease progression. With only a few dozen patients in early trials, they use those outcomes to compute an estimator of the “hazard ratio”. Simulating similar small-trial data sets allows them to assess the reliability of their estimator.
Counting Wildlife A team of ecologists wants to know the true number of deer in a forest. They walk a handful of fixed paths, count what they see, and take the average. By simulating many such walks under different population models, they can forecast how close their estimated average count is likely to be to the real population value.
Measuring Market Risk A financial analyst calculates the worst-case daily loss (Value-at-Risk) for a portfolio by looking at past returns. Because markets have “fat tails” (rare but huge swings), they simulate thousands of possible return scenarios to check whether their risk estimate under- or over-states the true exposure.
Evaluating Learning Experiments A psychologist tests a new memory exercise by measuring the difference in average recall between two small groups. Reaction times and memory scores aren’t perfectly normal, so they run simulated experiments to see if their estimated effect size will be biased or wildly variable with only 20 participants per group.
Evaluating a New Estimation Method A statistician has worked through tedious calculations to derive a new parameter estimation technique from a system of estimating equations whose solution defines the estimator: e.g., the normal equations for a least-squares regression fit. Before expending more effort following other theoretical pathways, simulation experiments can be used to assess the potential of the new methodology in some postulated model settings where it might be used.
Running serious simulation experiments like those that will be needed for our examples takes time: hours or maybe even days. Plus, a single run is generally not sufficient. We learn from experiments as we go, which translates into making changes in the experimental settings. And, of course, errors in the code or logic are not uncommon, in which case the whole experiment has to be discarded and run again.
One way to deal with the time issue is to run the simulations on a high-performance computer. Our scientist is assumed to be comfortable writing code in Python or R for use on a personal computer; at least sufficiently comfortable to have been able to write code to compute the estimator in question. But, the technical and possibly financial requirements for using a computer cluster fall outside their comfort zone. Moreover, a cluster may not be available and, realistically, would likely be overkill for the problem at hand.
So, what to do?
An answer is nearby sitting on the desk (or lap) top: namely, a personal computer. Almost all modern computers have multiple cores. These cores are individual numerical/logical processors that the CPU (central processing unit) uses as needed to perform its tasks. However, it is possible to program these cores individually and create a pool of worker processes that carry out their tasks in tandem. The result is an instance of parallel computing.
The speed-up from using multiple processes for a computational task is, in an ideal case, just the number of processes in the mix, provided each process gets a core of its own. While 10 workers could, in theory, make your code run 10 times faster, in practice this seldom happens. Modern machines have performance and efficiency cores with the former being most useful for parallel work. There are also communication and other overhead factors that make the ideal case generally unattainable.
Of course not every problem benefits from a parallel treatment. It must be possible to divide the problem into chunks that can be distributed across the processes. A case where this kind of splitting can be done is simulation experiments: the subject of our book. These fall under the heading of “embarrassingly parallel” problems where each chunk of code can be executed without knowing anything about the other. Here is a quick test for that property: if you could run the simulation loop backwards and still get the same answer, the work is almost certainly embarrassingly parallel. A computation in which each step depends on the previous one, such as a long Markov chain, fails that test and will not speed up just by adding workers.
In the next chapter we set the stage for developments throughout the remainder of the book. First, we describe serial and parallel random number generation and detail how to obtain random samples from any given probability distribution. Simulation is analogous to the repeated experiments that lie at the heart of the lab sciences and that is how we prefer to view it. Several examples are presented that will help cement this mental image.