Prompt Caching

Definition #

Prompt caching is a technique for reusing work/results from previous requests instead of processing the same request again.

In our optimization system:

New request → check cache → if matching result exists → reuse it

The goal is to save computation and reduce latency.

The Problem That Led to It #

Imagine our simulation repeatedly asks the same question:

“What should happen when a traffic light turns red?”

Sending that same request to an LLM every time wastes computation.

If we've already processed it, why calculate it again?

What Problem It Solves #

Prompt caching stores previous results and uses them when a future request matches.

Your source mentions two approaches:

Exact SHA-256 keys → identify exactly matching prompts.

FAISS-based semantic matching → find previously processed prompts that are meaningfully similar.

So:

Request ↓ Cache? ┌─┴─────────┐ YES NO ↓ ↓ Reuse LLM/model result ↓ Store result

Easy Wording #

Prompt caching means “we've already solved this, so reuse the previous result.”

Layman Example #

A teacher repeatedly gets asked:

“What is 2 + 2?”

Instead of solving it every time, they already have the answer written on a card.

Technical Example #

Request A:

traffic_light=red, vehicle_waiting=true

→ LLM processes it

→ result is cached.

Later:

Request B: same prompt

→ cache finds the existing result

→ returns it without calling the LLM again.

Limitation #

Exact caching doesn't help when the request is slightly different.

That's why semantic matching can be useful: it can identify requests that are different in wording but similar in meaning.

Solution #

Caching handles repeated work.

But what about many new routine events arriving simultaneously?

That's where vectorized batch dispatch comes in.