Parameter Server
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A parameter server is a distributed training architecture: server nodes hold the model's parameters while worker nodes compute gradients and send updates.
What is a Parameter Server?
A parameter server is an architecture for distributed model training: dedicated server nodes hold the model’s parameters, while worker nodes each process a slice of the training data, compute gradients, and send them to the servers to update the shared model.
It answers a scaling problem. When a model or dataset outgrows one machine, some machine still has to hold the authoritative copy of the weights – the parameter server is that machine, or a shard of several.
How a Parameter Server Works
Training proceeds in a push-pull loop. Each worker pulls the current parameters, runs forward and backward passes on its own batch of data, and pushes the resulting gradients back; the servers apply the updates and the loop repeats.
Updates are synchronous, where servers wait for every worker before updating, or asynchronous, where each gradient is applied on arrival. Asynchronous training keeps fast workers busy but computes on slightly stale parameters, a trade-off between hardware efficiency and gradient quality.
Mu Li and colleagues described the design in a 2014 paper, and it became a standard pattern for large-scale training. For dense deep learning models, all-reduce communication (workers averaging gradients among themselves) has largely displaced it, but parameter servers remain common for models with huge sparse parts, such as the embedding tables in recommendation systems.
Example of a Parameter Server
A retailer trains a recommendation model whose embedding table for products and users runs to billions of parameters – too many for one machine’s memory. The table is sharded across several parameter servers.
Each worker processes a stream of click logs, pulls only the embeddings its batch touches, computes gradients, and pushes them back to the owning shard. No single machine ever holds the whole model, yet the training run converges as one.
Related AI terms: Ray · Pipeline · Model Training · Stochastic Gradient Descent · Weights · ZeroMQ
Did you like the Parameter Server gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About