Grok-1's Mixture of Experts: Active Parameters Are Not Stored Parameters

Grok-1's release describes 314 billion parameters and 25% of the weights active for a token. That percentage is easy to misread as a storage discount. It describes sparse computation, while loading and placing the weights remains a separate problem.

The distinction is small enough to demonstrate without downloading Grok-1. This note uses an eight-expert NumPy layer that selects two experts per token. The executable experiment counts the arrays that remain stored, changes an unselected expert, and then changes the router so that expert becomes relevant. No Grok checkpoint, GPU inference or training is claimed here.

What the release actually describes

The original Grok-1 release announcement identifies a 314B mixture-of-experts base model, trained from scratch, with 25% of its weights active for a given token. The released checkpoint is a base model rather than a chat-finetuned assistant. “Open weights” should not be read as a claim that this example reproduces its training data or training process.

The official repository describes eight experts with two selected and provides JAX inference code. Its hardware needs are not established by this notebook. There is no universal “one 80GB GPU” conclusion to extract from the active fraction: storage format, distribution across devices, offload, runtime buffers and sequence state all affect a real deployment.

A useful first reading therefore separates three questions. How many parameters exist? Which computations are selected for one token? Where are the arrays and runtime state stored while a batch is processed? The release's active fraction answers only part of the second question.

Route two tokens through eight experts

Our toy has four input features, four output features and eight linear expert matrices. Each expert contains sixteen scalar weights. The router is a separate four-by-eight matrix. There is no attention layer, shared transformer block, expert capacity mechanism, distributed communication or checkpoint loader.

For each token, the router produces eight scores. The code chooses the two largest, applies softmax only to those selected scores, and combines the two expert outputs with normalized weights. This selected-score normalization defines this toy's behavior; it is not presented as a transcription of every detail in Grok's implementation.

The routing core is small enough to inspect:

scores = x @ router
selected = np.argsort(-scores, axis=1, kind="stable")[:, :top_k]
logits = np.take_along_axis(scores, selected, axis=1)
weights = np.exp(logits - logits.max(axis=1, keepdims=True))
weights /= weights.sum(axis=1, keepdims=True)
output = np.zeros((len(x), experts.shape[2]))
for token, ids in enumerate(selected):
    for gate, expert_id in zip(weights[token], ids):
        output[token] += gate * (x[token] @ experts[expert_id])
return output, selected, weights

Stable sorting makes tied scores resolve by expert index, which keeps the fixture deterministic. A different tie policy is possible, but leaving the policy implicit would make exact expectations less useful. The implementation also rejects nonfinite inputs and incompatible shapes before doing matrix multiplication.

The code, tests, requirements and setup instructions run with Python 3.11.5 and NumPy 2.2.6. No external model or dataset is fetched. After installing the pinned dependency:

python3 -B -m unittest -v test_example.py

All five tests pass, including hand-computed mixture output, routing changes, ties and invalid inputs.

Count the stored weights separately

The first token selects experts 0 and 1; the second selects 6 and 7. The executable reports 128 stored expert parameters, 32 expert parameters selected per token, and 32 additional router parameters. Four distinct experts are selected across the two-token batch.

These counts come from the actual array sizes. They do not count each multiply-add, and they do not estimate Grok's latency. They also reveal why the denominator matters: “two of eight experts” is one quarter of this toy's expert weights, while the router still exists and runs. Applying the same fraction blindly to every part of a larger architecture loses shared components and other work.

The second test adds a large value to expert 3. Neither token currently selects it, so the output remains exactly unchanged. The expert array still occupies the same number of bytes. Next, the test raises the routing score for expert 3, and the output changes. Deleting that expert merely because it was unused by one request would alter what later routing decisions can do.

This is the storage argument in executable form. Sparse selection lets a request skip some expert computation. It does not make the other weights disappear from a fully resident model.

A batch changes the deployment question

One token's selected set is not necessarily the next token's set. Even our two-token fixture touches half of the experts collectively. A larger workload may route across a wider set, and the selection changes across layers and generation steps. A system that moves expert weights on demand must account for transfers and scheduling rather than assuming the unselected fraction is permanently irrelevant.

The toy intentionally does not simulate that system. NumPy stores every expert in a local array and evaluates selected experts in a Python loop. Timing that loop would mainly measure this demonstration, not the inference design of Grok-1 or the merits of expert parallelism.

For a real feasibility study, start with the exact released checkpoint and implementation revision. Record weight representation, placement, offload policy, batch and sequence lengths, then measure memory and throughput in that environment. Keep total model storage and per-token active work as separate columns. If a proposed hardware budget depends on deleting six experts permanently, it is evaluating a modified model whose outputs and quality need new evidence.

Grok-1 is useful here because its headline numbers expose a common mistake. The local experiment establishes the narrower claim: selecting fewer experts reduces the expert computation performed for a token, while all eight expert arrays remain stored in this implementation. It says nothing about the model's benchmark standing or a particular machine's ability to serve the released weights.