← All posts

Who Gets the GPU?

How our AI workloads share two machines.

Painting a creature, generating a video and running a research model can all ask for the same two machines. Each job has its own idea of how much memory it needs and how long it wants to stay. The machines get no larger because another agent has a good idea.

One coordinator, two shared machines
One coordinator, two shared machines Creative requests and the research application use Engine. Engine controls workers on two GX10s over gRPC, keeps the queue, and returns results. The machines reuse model files locally or from their peer before requesting files from the model store. The research model is an alternative use of both machines together. WORKSTATIONTHE TWO GX10s Requests gRPC Dispatch / status Peer Results Creativerequests Researchapplication Engine coordinatorQueue · placement · results Saved jobs GX10-1Model workers GX10-2Model workers Or: research modelUses both machines together Model store One coordinator, two shared machines Creative requests and the research application use Engine. Engine controls workers on two GX10s over gRPC, keeps the queue, and returns results. The machines reuse model files locally or from their peer before requesting files from the model store. The research model is an alternative use of both machines together. gRPC Creativerequests Researchapplication Engine coordinatorQueue · placement Saved jobs GX10-1Workers GX10-2Workers Or: research modelUses both machines Model store
Control / requestsFile movementSaved state
Engine owns placement and the queue. Model files and completed results travel separately from control messages.

We use a pair of GX10s, compact NVIDIA GPU computers, for that shared work. Studio makes images, textures, video and audio. Pinecone runs research using a large language model across the pair. I want agents to be able to request what they need without each one becoming a part-time machine operator.

Engine is the internal service that owns that job. It accepts requests, finds a safe place to run them, manages loaded models and returns verified results. The interesting design choices are about what it can reuse, when it can interrupt something, and what counts as finished.

Keep the useful models warm

Loading a model costs time. If the right worker is already loaded and available, a new request can use it. An idle model can stay in memory while there is room, which is useful when several texture jobs arrive between other kinds of work.

Reuse first. Make room when needed.
Reuse first. Make room when needed. A queued request reuses a compatible worker if one exists. Otherwise, load another worker when it fits in memory, or drain an idleable worker at a safe job boundary. If capacity cannot be freed safely, the request waits. Yes No No Yes Yes No Queued work Reusableworker? FitsRAM? Safe toreclaim? Reuse worker Load + run Wait in queue Reuse first. Make room when needed. A queued request reuses a compatible worker if one exists. Otherwise, load another worker when it fits in memory, or drain an idleable worker at a safe job boundary. If capacity cannot be freed safely, the request waits. Yes No No Yes Yes No Queued work Reuse? Fits RAM? Safe toreclaim? Reuseworker Load + run Wait
Control / requests
A request that cannot fit stays queued. Reclaiming memory waits for a safe job boundary.

Compatibility includes the weights, runtime and numerical precision. Two workers reading the same files do not automatically share their memory. Engine budgets the whole running worker, including working memory, adds a ten-percent margin to its rough estimate, and leaves a separate allowance for the operating system.

What fits in memory?
What fits in memory? Rough whole-worker estimates including ten percent margin: the Klein image model uses 26.95 GiB, the LTX video model 102.41 GiB, and both total 129.36 GiB before operating-system memory. They do not fit together on one machine. Memory, including a 10% margin Klein · image model 26.95 GiB LTX · video model 102.41 GiB Together 129.36 GiB What fits in memory? Rough whole-worker estimates including ten percent margin: the Klein image model uses 26.95 GiB, the LTX video model 102.41 GiB, and both total 129.36 GiB before operating-system memory. They do not fit together on one machine. Memory, including a 10% margin Klein · image model 26.95 GiB LTX · video model 102.41 GiB Together 129.36 GiB
Rough estimates, including the 10% margin: together these workers need 129.36 GiB before the operating system. This combination does not fit one machine.

Several different media workers can run together when they fit. If they do not, Engine chooses what to drain and unload at a safe job boundary. Older unserved work gets priority so a stream of familiar jobs cannot keep everything else waiting indefinitely. A valid request that cannot fit safely stays queued; its prompt or image size does not become invalid just because the machines are busy.

A second cache for the SSDs

Each GX10 has only 1 TB of local storage, so we also need to manage which models stay on disk. Engine keeps their files nearby for reuse and clears unused ones when space runs short. When a model is needed again, it can copy the files from the other machine or fetch them from the central store.

Research uses the quiet periods

The research model needs both machines, so it runs during gaps in creative work. When an image, video or audio job arrives, Engine stops the research model and gives that job priority.

Research fills the gaps
Research fills the gaps Research uses both machines until creative work arrives. Creative jobs take priority. After twenty minutes without waiting or running creative jobs, the research model can load again. A new creative job interrupts loading. Jobs finish 20 minutes clear Ready New job Research running Creative job arrives Creative jobs run No creative work Research loads again Research fills the gaps Research uses both machines until creative work arrives. Creative jobs take priority. After twenty minutes without waiting or running creative jobs, the research model can load again. A new creative job interrupts loading. 20 minutes clear Research running Creative job arrives Creative jobs run No creative work Research loads again
Control / requestsNew job interrupts loading
Creative work takes priority, including while the research model is loading.

After twenty minutes with no creative jobs waiting or running, Engine can load the research model again. New creative work takes priority even while that model is loading.

Our own local multimedia server

Engine brings our local media tools together into a server we can use across projects. Whether we’re making creatures for Rowan, generating video or developing sound, our agents can request the work through the same service. It manages the models, queue and shared machines, giving us a common foundation for the things we want to create.
Who Gets the GPU?
0:00
0:00