Start with the workload
“A server for a 14B model” is not a complete requirement. The same model may serve one engineer, a document-search service or dozens of employees. Define the use case, peak concurrency, context length, acceptable quantization and target latency first.
GPU memory is a working budget
Model weights, runtime buffers and the KV cache share GPU memory. Longer context and more concurrent sessions increase the cache even when model weights stay unchanged. Quantization can reduce the footprint, but it should be validated on the company’s own tasks rather than accepted from a generic benchmark.
The rest of the server still matters
CPU capacity supports tokenization, APIs and adjacent services. System RAM holds the operating environment, containers, document processing and vector-search components. NVMe storage affects model loading, indexing and recovery. Production storage also needs a resilience and backup plan.
Validate topology and facilities
For multi-GPU systems, check PCIe lanes, CPU-root placement, GPU interconnects and proximity to network and NVMe devices. Confirm power, cooling and rack limits under continuous load. A chassis with enough physical slots is not automatically a suitable AI platform.
Finish with a pilot
Measure time to first token, throughput, concurrent sessions, GPU utilization and memory use with real prompts and document sizes. The pilot turns assumptions into a capacity model and shows whether to scale up one node or add another.