Vectorized Batch Dispatch

Definition #

Vectorized Batch Dispatch means grouping many similar events together and processing them in parallel instead of handling them one at a time.

Its role is to make the system faster when many events arrive together.

The Problem That Led to It #

Imagine our simulation receives:

10,000 routine vehicle events at the same time.

Without batching:

Event 1 → Surrogate Event 2 → Surrogate Event 3 → Surrogate ... Event 10,000 → Surrogate

The system repeatedly calls the model.

That's inefficient.

What Problem It Solves #

Instead, the system groups similar events:

10,000 similar events ↓ Batch together ↓ Surrogate Model ↓ Process them in parallel

This makes better use of GPU/CPU memory and parallel computation.

Easy Wording #

Instead of making the computer handle 10,000 similar jobs one by one, give it the 10,000 jobs together so it can process them in parallel.

Layman Example #

Imagine a restaurant with 100 identical orders.

Instead of:

Cook → Order 1 → finish → Order 2 → finish...

the kitchen prepares the common ingredients for many orders together.

Much more efficient.

Technical Example #

Suppose MoM receives:

Event 1 → classify vehicle Event 2 → classify vehicle Event 3 → classify vehicle ... Event 1000 → classify vehicle

MoM determines they're all suitable for the surrogate.

Instead of:

Event → Surrogate Event → Surrogate Event → Surrogate

it creates:

[Event1, Event2, Event3 ... Event1000] ↓ Surrogate ↓ [Result1, Result2 ... Result1000]

The events are processed in parallel.

Limitation #

Batching works best when events are similar enough to process together.

If every event requires completely different processing, batching becomes less useful.

Solution #

We've now discussed how to make model inference efficient.

But our simulation also has an enormous 1-billion-node network graph.

How do we store that huge graph efficiently?

That's where Compressed Sparse Row (CSR) comes in.