Deep Learning - Why 100 GB of NumPy Data Slowed Down Our ML Training

1. The Problem

Suppose we’re training a 3D LiDAR object recognition model to detect cars, pedestrians, and cyclists. We have three training configurations: centred crops, shifted crops, and temporal crops.

We launched three training jobs simultaneously, expecting the GPU to handle them efficiently. After 69 minutes:

Our machine has 62 GB of RAM, and two training datasets occupy approximately 51 GB each. Initially, we suspected the SSD. However, the NVMe was reading at approximately 680 MB/s.

The actual problem was memory pressure causing Linux to swap program memory onto a slow mechanical hard drive. To understand why, let’s start with the data format.

2. Why Does Each Dataset Occupy 51 GB?

Each training sample is a crop of a 3D point cloud. Each point contains four float32 values:

where x, y, z are 3D coordinates and I is intensity. Different objects have different numbers of points:

Our original storage layout uses one .npy file per recording, containing a structured array with one row per crop (we use a solid-state LiDAR, so its point cloud is structured). Each row reserves space for 4,096 points, regardless of how many real points exist:

1
2
3
Crop A: 650 real points + 3,446 zeros
Crop B: 1,400 real points + 2,696 zeros
Crop C: 3,200 real points + 896 zeros

This is called padding. With 779,000 crops, four values per point, and four bytes per value:

We have two such datasets, totalling approximately 102 GB.

Importantly, .npy itself does not require 4,096 points. This is a constraint of our fixed-size structured array layout.

3. How Does NumPy Memory Mapping Work?

Normally, loading a NumPy array reads the whole array into memory:

1
data = np.load("recording.npy")

Instead, we use memory mapping (mmap):

1
data = np.load("recording.npy", mmap_mode="r")

Rather than loading all 51 GB immediately, Linux maps the file into the process’s virtual address space. When we access a crop with crop = data[1000], Linux:

1. Checks whether the required file pages are already in RAM.
2. If not, reads them from the SSD.
3. Makes those pages available to the process.
4. Keeps them in RAM for possible reuse.

The cached file data is called the page cache.

Why is page cache useful?

Training repeatedly reads the same data across epochs. If a crop remains in page cache, subsequent reads avoid SSD access.

However, page cache consumes physical RAM. Our machine was observed with approximately 57 GB of page cache out of 62 GB of total RAM. Two 51 GB datasets cannot both remain entirely cached. That’s where memory pressure begins.

4. What Is the Difference Between Page Cache and Swap?

Linux manages two important categories of memory:

Page cache does not overflow into swap. When RAM becomes scarce, Linux can reclaim clean cached file pages because their contents already exist on disk. But program memory that has no file backing cannot simply be discarded; eligible pages may instead be written to swap.

1
2
3
4
5
RAM
├── Page cache: LiDAR crops
├── Python program memory
├── DataLoader buffers
└── Other system memory

When the training processes continuously load large datasets, Linux must choose which memory to reclaim. In our case, some program memory was moved to swap on a mechanical hard drive.

5. Why Did Training Stall? Memory Thrashing

The observed sequence was:

1. Multiple jobs read large .npy datasets.
2. Linux fills much of RAM with page cache.
3. Memory pressure causes some program memory to be swapped out.
4. Training processes need that memory again.
5. Linux must read it back from a slow hard drive.
6. The cycle repeats.

This is called memory thrashing: the machine spends excessive time moving memory instead of computing. Our measurements showed:

The temporal process was particularly affected. It had approximately 1.27 GB swapped out and was blocked waiting for memory-related I/O.

Why is HDD swap particularly slow?

Mechanical hard drives are poor at random reads. Suppose one small random read takes 10 milliseconds, and each read retrieves a 4 KiB page:

This is an illustrative serial random-read calculation, not the measured HDD throughput. Even though the NVMe dataset drive was fast, the training processes could be blocked by much slower swap accesses.

Immediate solutions

• Reduce concurrency: run the temporal training job separately.
• Move swap to NVMe: avoid slow mechanical HDD swap I/O.
• Lower swappiness: for example, sudo sysctl vm.swappiness=10 encourages reclaiming file cache relative to swapping anonymous memory.
• Disable swap where appropriate: avoids swap stalls, but accepts a greater risk of out-of-memory termination.

These changes address memory management. We can also reduce the dataset itself.

6. Why Are We Wasting Almost Half the Storage?

Recall that every crop reserves 4,096 points, but our average crop contains only approximately 2,177 real points. The average padding fraction is:

Nearly half of the point storage is unnecessary padding. The most important observation is that our neural network never uses those padded points. The existing loader already does something like:

1
2
3
4
5
count = crop["point_count"]

xy = crop["xy"][:count]
height = crop["height"][:count]
intensity = crop["intensity"][:count]

It discards padding and rasterizes the real points into a fixed-size grid, such as (3, 48, 48). The CNN receives this grid, not the original 4,096-point array. Therefore, we can eliminate padding from disk without changing the model input.

7. Solution: Ragged Arrays with Two .npy Files

A ragged array stores variable-length samples without padding. Instead of storing every crop in a fixed-size block, we concatenate all real points into one continuous array, using two files per recording:

Example

Suppose three crops contain different numbers of points:

The points file contains:

1
2
Index:   0 1 2 3 4 5 6 7 8 9
Points:  A B C D E F G H I J

To retrieve the car crop:

1
2
3
4
5
6
7
8
points = np.load("recording.points.npy", mmap_mode="r")
metadata = np.load("recording.npy", mmap_mode="r")

crop = metadata[1]
start = int(crop["point_start"])
count = int(crop["point_count"])

car_points = points[start:start + count]   # points D through H

No padding is required. Both files remain memory-mappable, so Linux reads only the required file pages rather than loading the whole dataset.

8. How Much Space Do We Save?

Previously, every crop stored 4,096 points. With ragged storage, we store approximately 2,177 real points per crop on average. At 16 bytes per point:

Adding approximately 0.5 GB for metadata gives:

That’s approximately a 46% reduction in storage per dataset, and less data for Linux to read and cache. However, two 27.6 GB datasets still consume substantial cache space on a 62 GB machine, so swapping and memory contention remain possible.

9. Why Two Files Instead of One?

A normal .npy file stores one NumPy array. Our metadata and concatenated points are different arrays. There are alternatives:

We chose two .npy files because they’re simple and preserve our memory-mapped loading workflow. We also avoid creating one file per crop, which would mean approximately 779,000 files per dataset.

10. What Changes in the Training Pipeline?

The network architecture doesn’t need to change:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Original pipeline                 New pipeline

3D point cloud                    3D point cloud
      ↓                                 ↓
Extract crop                      Extract crop
      ↓                                 ↓
Pad to 4,096 points               Store real points + offsets
      ↓                                 ↓
Store in .npy                     Load real points using mmap
      ↓                                 ↓
Load and discard padding          Rasterize into fixed-size grid
      ↓                                 ↓
Rasterize into fixed-size grid    Neural network
      ↓
Neural network

The 4,096-point limit can remain as a maximum at crop creation time; crops exceeding that size are subsampled. Shorter crops no longer need padding at any stage.

Implementation requires updating the crop builder, dataset loader, and viewer, followed by testing that old and new representations produce equivalent model inputs.

• Previous
  Deep Learning - Contrastive Learning