8 Into the Beyond

The earlier chapters showed how much you can do with a single machine, and for most projects that genuinely is the end of the story. A modern laptop with eight or twenty cores, used the way this book describes, handles a surprising fraction of real simulation work.

This chapter is for the cases where it becomes necessary to step beyond that paradigm. First, how do you know you have actually outgrown your laptop, as opposed to merely written slow code and, once you are sure, what is the cheapest next step that fits the kind of limit you hit? Those are two different questions with different answers, and getting them in the wrong order can be expensive: people routinely rent a 64-core cloud machine to run code that some serious profiling would have made fast enough for the hardware they already possess. Sections 8.3 and 8.4 explore options for when the machine you have is no longer up to the task.

The basic premise throughout the book has been that we are running independent experiments. But, what if that is not the case. Are you destined to spend your time waiting on slow code? Is there nothing to be done? Section 8.5 discusses that scenario and, as a bit of a spoiler, there is hope if you adjust your thinking accordingly.

This chapter is also where the plug-and-play spirit of the book reaches its natural edge. Cloud machines and clusters still run the same code you have written, so the ideas carry over directly. On the other hand GPUs do not; they vary from one setup to the next, and demand a different programming model that sits outside this book’s “one formula you can drop in” promise. That is why the GPU material in Section 8.2 serves as a pointer rather than a recipe.

A good way to read the material that follows is to skim it now for the map. Then you can return to the relevant sections on the day you actually hit the wall.

8.1 When to Move Beyond a Personal Machine

A laptop is the right tool for a surprisingly long time. When you do outgrow it, the signal usually takes one of three forms. They are genuinely different problems with different fixes. So the first useful skill is telling them apart.

Before dealing with any of them, though, try harder on the machine you already have, because the cheapest speed-up is almost always in the code, not the hardware. Profile one run to find where the time actually goes; it is rarely where you guess. Look for a vectorized inner loop: generating the random numbers in one block instead of one at a time, computing the estimator without an unnecessary copy of the data, or simply trimming n_reps once the square-root law from Chapter 2 says the extra precision is not worth it. Each of these steps can be worth more than doubling your core count, and they cost nothing.

Renting a 64-processor machine to run code that a morning of profiling would have made fast enough is a common and expensive mistake. Reach for bigger hardware only once the laptop is genuinely the limit.

The first real limit is wall-clock time. Suppose the study still takes days even with every core working and the code already tightened. To confirm this is the bottleneck, open a system monitor while a run is going. If every core is pinned near 100% and it is still too slow, you are compute-bound, and the answer is more or faster cores. Be realistic about how much that buys you, though. As Chapters 3, 4 and 5 demonstrated, a machine with eight cores does not run eight times faster once efficiency cores and dispatch overhead are taken into account. A 5,000-resample bootstrap over a large dataset, or a 100,000-replication power study, is a typical case: pure compute, embarrassingly parallel, and simply larger than one machine can finish in a reasonable time. Estimate the speed-up you can actually expect by running some small experiments and then extrapolating to the case of interest.

The second limit is memory as described in Section 7.2 rather than speed. Each worker process gets its own copy of the data, so running on n_cores cores can mean n_cores copies in RAM at once. You will recognize this case because memory use climbs while adding cores and that actually makes things slower. The fix is therefore not more parallelism but more RAM, fewer simultaneous workers, or a leaner data layout: for example loading the shared data set once in a form the workers can read without duplicating it.

The third limit is breadth. You have a whole grid of parameter settings, say 500 combinations of effect size, sample size, and model assumptions for a power study and you would like them all running side by side rather than one after another. A single machine has to serialize that grid through its core budget, so even if each configuration is fast the wall-clock total is punishing. The work is naturally a collection of independent jobs, which is exactly what clusters and cloud job arrays are built for.

A fourth, quieter consideration is runtime risk. A computation that takes three days on a laptop is also a computation that can fail on day three, from a sleep event, an update, or a full disk, with nothing to show for it. Long runs want a machine that is managed, queued, and checkpointed, which is itself a reason to move even when raw speed is adequate. None of these limits means your code is wrong. The recipe from the earlier chapters stays the same; only the hardware has to change. Match the move to the symptom: compute-bound points to more or faster cores (cloud or cluster), memory-bound points to a bigger-RAM machine, and a breadth problem points to many machines or a job array.

8.2 Beyond Your CPU: GPU Programming

Actually, we have not completely exhausted what you can achieve with your personal machine. There is a bit more we can do if we look beyond computing with just the CPU. There is an as yet untapped resource: the graphics processing unit (GPU).

GPUs were originally developed to accelerate the rendering of images for video games and graphical interfaces. However, the recent growth of machine learning and artificial intelligence has accelerated GPU development to where modern GPUs now possess floating-point computational power that can also be exploited for simulation, numerical linear algebra and statistical computing.

The key architectural difference between a CPU and a GPU lies in how each approaches computational work. A CPU contains a small number of sophisticated cores optimized for sequential tasks, branching logic, and general-purpose operating system activity. CPUs excel when a program must make complicated decisions or rapidly switch between different types of operations. In contrast, a GPU contains a very large number of simpler arithmetic units designed to perform the same operation simultaneously across many pieces of data. In effect, a CPU is a small team of highly trained specialists while a GPU is a large workforce of less skilled laborers all performing the same repetitive calculation in tandem.

From a parallel programming perspective, GPUs are fundamentally different from ordinary multicore CPU programming. In CPU-based parallelism we typically launch a modest number of worker processes. In contrast, GPU programming involves launching many lightweight computational tasks simultaneously. Each task may perform only a tiny amount of work, but the aggregate effect can be substantial.

GPUs are especially attractive for scientific computing problems that exhibit data parallelism: the same arithmetic operation repeated independently many times. Embarrassingly parallel simulations are a prime example of data parallelism in that the requisite calculations are independent. The main challenge then becomes organizing the computation so that large blocks of work are sent efficiently to the GPU while minimizing communication overhead with the CPU.

This speed-up potential of GPUs comes with tradeoffs. GPUs are not universally faster than CPUs. Moving data between the CPU and GPU introduces overhead and many GPUs are not proficient at handling things like branching logic or memory-access patterns. Problems that are too small may actually run slower on a GPU because the startup and communication costs dominate the computational effort. Consequently, GPU computing is most effective for large, highly repetitive numerical workloads.

There are several methods available for programming with GPUs. Unfortunately, they are mostly platform dependent. For example, Apple has a language called Metal that can be used for the GPUs on their machines while computers with NVIDIA graphics are programmed in CUDA. These languages are C/C++ relatives and we prefer to avoid a detour in that direction.

Therefore, the case we will treat is a Windows machine with an NVIDIA graphics card. Fortunately, in that setting there is a package CuPy that will allow us to largely side-step CUDA and work in the more familiar Python environment.

To illustrate GPU computing, we return to the Cauchy scale-parameter simulation. Each experiment requires the same operations on a large array: generate uniform random numbers, transform them into Cauchy random variables, compute columnwise quartiles, and form the scale estimates. This regular, vectorized calculation is well suited to a GPU.

Our experiment was conducted on the ASUS laptop using its NVIDIA GeForce RTX 5060 GPU. The objective is not to develop a general GPU simulation template, but to provide a reproducible example of the setup and potential speedup.

Verify that Windows sees the NVIDIA GPU:

nvidia-smi

This should display the GPU model, driver, and supported CUDA version.

If necessary, download and install Miniconda from the official installation page. Then open an “Anaconda Prompt” from the Windows Start menu and create and activate an isolated environment via

conda create -n gpu python=3.12 -y
conda activate gpu

The new prompt should begin with (gpu). You install the CuPy package containing the CUDA 13 components with

python -m pip install "cupy-cuda13x[ctk]"

To verify CuPy start Python and at the Python prompt enter

import cupy as cp 
print(cp.__version__)
print(cp.cuda.runtime.getDeviceCount())

The final value should be at least 1.

We are almost ready to run a GPU program. The two programs we will use are cauchy_gpu_worker.py:

import cupy as cp

def worker(seed, n, n_reps, theta=1.0):

    bit_gen = cp.random.Philox4x3210(seed)
    rg = cp.random.default_rng(bit_gen)

    # Generate and transform the samples on the GPU
    U = rg.random((n, n_reps))
    X = theta * cp.tan(cp.pi * (U - 0.5))

    # Compute both columnwise quartiles.
    q = cp.quantile(X, [0.25, 0.75], axis=0)
    theta_hat = (q[1] - q[0]) / 2

    # Transfer the estimates back to CPU memory
    return cp.asnumpy(theta_hat)

and timed_cauchy_driver.py

import time
import numpy as np
from cauchy_gpu_worker import worker

# simulation parameters
seed, n, n_reps, theta, n_runs = 1234, 100, 1000000, 5, 10

# Untimed warmup
worker(seed, n, n_reps, theta)

elapsed = []

for run in range(n_runs):

    start = time.perf_counter()

    worker(seed + run + 1, n, n_reps, theta)

    elapsed.append(time.perf_counter() - start)

q1, q3 = np.quantile(elapsed, [0.25, 0.75])

print("Median time:", np.median(elapsed))
print("IQR:", q3 - q1)

The first of these listings is the worker program that is similar to the Cauchy worker function from Chapter 4. It generates data from the uniform distribution and then uses the Cauchy quantile function to produce Cauchy data. It computes n_reps inter-quartile range statistics on samples of size n.

The second listing is a short driver program that times the worker. It performs one untimed warm-up followed by ten timed runs, each with \(n=100\), one million replications, and \(\theta = 5\). The values printed by the driver are the median execution time across ten runs and the interquartile range of those times.

CuPy’s random number interface relies on NVIDIA’s infrastructure that offers a Philox-family generator (see Appendix B) designed for large-scale parallel simulation work. We have explicitly requested that in our creation of a bit generator.

A typical terminal session that conducts 10 runs computing 1,000,000 inter-quartile ranges for samples of size 100 looks like

(gpu) C:\gpu_cauchy>python timed_cauchy_driver.py
Median time: 0.214879
IQR: 0.005460

Comparing this result with the CPU runtimes in Table 4.3 we see that the GPU completed the Cauchy IQR experiment in a median time of 0.215 seconds, compared with 0.683 seconds using eight CPU processes and 3.236 seconds using one CPU process. Thus, the GPU was approximately 3.18 times faster than the eight-process implementation and 15.06 times faster than one process. This is a favorable example for GPU computation: the problem involves large arrays and repeated vectorized operations with little branching. Smaller problems, computations requiring frequent CPU–GPU transfers, and models involving complicated conditional logic may obtain much less benefit.

8.3 Beyond Your Machine: Cloud Computing

The gentlest step up from a personal machine is to a cloud virtual machine. A provider such as AWS, Google Cloud, or Azure will rent you, by the hour, what is essentially a much larger version of your laptop: the same kind of operating system, the same R or Python, far more cores and memory. Because it is the same kind of machine, nothing in your code has to change. The program does not know or care that it now has 64 cores instead of 8; it simply finishes sooner.

It helps to see the whole loop once, at a sketch level. You start an instance of the machine size you chose. You connect to it, either over SSH for a plain terminal or through a browser using a server build of RStudio or a Jupyter or Posit environment if you prefer that workflow. You move your code and data over, with rsync or scp for a one-off, or by reading from cloud object storage such as S3 if the data is large or shared. You run exactly the same sim_template call you have been running and you copy the results back, often just a small file of summaries. Then you terminate the rental. That last step is not optional housekeeping; it is where the cost is decided. Cost is the decision that matters most, and a back-of-the-envelope number makes it concrete. Cloud machines are priced at roughly a few cents per core-hour. A 64-core machine for a three-hour run is then somewhere around ten dollars, which is trivial next to the days of your time it saves. The genuine money sink is not the run; it is a powerful machine left switched on and idle for a week because someone forgot about it. Treat automatic shutdown at the end of the job, and a billing alert on the account, as part of the work rather than an afterthought.

For batch work that can be restarted, spot or preemptible instances cost a small fraction of the standard rate, with the single catch that the provider can reclaim them mid-run. That is exactly the case a reproducible simulation handles gracefully: seed it, have each core write its partial results as it finishes a chunk, and a reclaimed machine costs you a restart, not the run.

If you would rather avoid managing a raw machine at all, the hosted notebook and batch services the providers offer trade a little control and some markup for much less setup.

A few practicalities round this out. Pick the instance to your bottleneck, many cores if you are compute-bound, lots of RAM if you are memory-bound, rather than the biggest box on the menu. Moving large datasets in and out takes real time and can incur data-transfer charges, so for data-heavy work weigh that against the compute saved. And remember that the data now lives on someone else’s computer, which can be disqualifying for sensitive or regulated data regardless of cost. The cloud makes the most sense for occasional large or urgent runs, where buying and maintaining a big machine of your own would be overkill and you may need the capacity right now.

8.4 Beyond One Machine: Cluster Computing

A computing cluster is many separate machines, called nodes, wired together on a fast network and shared among many users. Each node is itself a full computer with many cores. What makes a cluster different from a big cloud box is how you use it: you do not run programs on it interactively. You write a short job script that says what you need and submit it to a scheduler, most commonly Slurm. The scheduler puts the job in a queue and starts it when room becomes available. You do not sit and watch; you collect the results when it has run.

The job script is unremarkable, which is the point. It is a few lines that ask for resources. So many cores, so much memory, a time limit, followed by the ordinary command you would have typed yourself, for instance running your R or Python script. You hand that file to the scheduler with one submit command and check on it with one status command. Nothing about your simulation code changes; it is wrapped, not rewritten.

The conceptual jump is from many cores on one machine to many machines each with many cores, and it has two layers. Inside one node, the parallelism is exactly what this book already taught: split the replications across that node’s cores with sim_template. Across nodes is a second layer, and for embarrassingly parallel work it is simpler than people fear. A job array, the scheduler running the same script many times with only an index changed, maps a 500-configuration sweep onto 500 independent tasks with no new parallel code at all; each task is one configuration, and inside each task you still use the ordinary multicore parallelism from the earlier chapters. That is the breadth problem from the first section, solved with the recipe you already have.

The heavier machinery for splitting a single tightly-coupled computation across nodes, message passing of the MPI kind, is real but rarely necessary for the independent-replication work this book is about. It is enough to know it exists.

The encouraging part is that R and Python drive clusters without anyone rewriting the hot loops in C, so the ideas here carry over directly and the friction is operational rather than conceptual. You need a little familiarity with the scheduler, the queueing etiquette of a shared system, and the manners of a shared filesystem, such as not pounding the login node and asking for resources you will not actually use in order to keep the queue fair for all users.

The trade against the cloud is straightforward. A cluster, if your institution runs one, is often the cheapest very large resource available to you and is frequently free at the point of use. But you wait in a queue and you do not control the hardware. The cloud costs money and is available the instant you need it. Both run the same code, so the choice is about access, urgency, and budget, not about rewriting anything.

8.5 Beyond the Assumptions

Everything in this book has leaned on one assumption: that the replications are independent of each other. That independence is the entire reason the work splits cleanly across processes. It is worth ending this chapter by looking at where that assumption holds beautifully and where it does not, because the way to handle the latter instance is a useful generalization of everything that came before.

The happy case is worth naming first, because it is so common in data science. The bootstrap, which Chapter 5 used to put confidence intervals on a change point estimator, is the poster child of embarrassingly parallel work: every resample is drawn and analyzed with no reference to any other. So, a 10,000-resample bootstrap is just sim_template with the resampling in the worker part of the simulation algorithm and nothing new to learn. Cross-validation folds, permutation tests, and parameter sweeps have the same shape. If your problem is one of these, you are not in “beyond” territory at all; you simply point the Chapter 2 recipe at it.

Markov chain models, including hidden Markov models with large state spaces, are the first genuinely different case. Here each step depends on the one before it, so a single chain is inherently sequential and cannot be sliced across cores. The independence has to be found somewhere else. Often it is at a higher level: run many independent chains at once, one per process, which is the original recipe from Chapter 2 applied to whole chains instead of single replications. Sometimes it is at a lower level: each step is a matrix-vector product over the state space, and that linear algebra is itself highly parallel, which is exactly the kind of work a GPU accelerates. Large hidden Markov models in genomics and speech, and big state-space time-series models, are where this matters in practice.

Markov chain Monte Carlo, used throughout modern Bayesian work, has the same character. One chain is sequential by nature because it must burn in and mix. So, you cannot simply cut it into more palatable pieces. But, you can run several chains from different starting points in parallel. This has the benefit of producing the between-chain convergence diagnostics you need; you can use schemes such as parallel tempering that are designed around multiple cooperating chains; and you can parallelize the expensive likelihood or gradient evaluation inside each iteration. The agent-based and discrete-event models, of which the epidemic in Chapter 6 is an example, sit in between. Separate runs are independent and parallelize easily, but within a run the agents may have to interact, which is why that chapter had to allow the processes to communicate.

The pattern is the recurring one from Chapter 1: when a problem resists parallelizing, the independence has not vanished, it has just moved. When the obvious level is sequential, look one level up, at many chains, resamples, or runs, or one level down, at the vectorized math inside a single step. These directions are beyond our scope here. But the questions to carry into them never changes: which parts of this are independent, and can that part be done at the same time?

8.6 Chapter Summary

In this chapter we looked past the single laptop. We began with how to tell that you have genuinely outgrown your machine, rather than merely written slow code, and what the cheapest next step is in each case. From there we surveyed the options. A GPU pays off for large, highly repetitive numerical work, though not for the branching, sequential logic of a model like the epidemic in Section 6.3. A cloud instance gives you more cores or memory for a while, and the sim_template code runs on it unchanged. A computing cluster goes further still, coordinating many machines through a job scheduler when one is no longer enough. We also stepped beyond the kinds of problems the rest of the book has assumed, to Markov chains, hidden Markov models, Markov chain Monte Carlo, and agent-based and discrete-event simulations, where the independence that makes parallelism easy is partly hidden and has to be sought out.

None of these is a recipe to lift straight from the page; each is a direction to explore when the need arises, and the surrounding technology changes quickly. What does not change is the question: which parts of the work are independent, and can those parts be done at the simultaneously? Carry that question with you, and the specific tool, whichever it turns out to be, becomes a detail.

Chapter 9 closes the book with a brief look back and a few thoughts on where the tools, and the competition, are headed. Let us end.