Projects

Things I’ve built in the open

gpu-fit

PythonvLLMMITv1.1

How many users can this LLM serve on this GPU, and with what command?

A parameters-times-bytes estimate is a good starting point; newer architectures need more detail. gpu-fit reads the model’s real architecture from Hugging Face and mirrors vLLM’s memory budget: weights, runtime overhead, KV cache, requests at full context and the longest single request. Then it prints the vllm serve command.

  • Understands grouped-query attention, hybrid linear-attention models, DeepSeek MLA, sliding windows, FP8/AWQ/GPTQ checkpoints, tensor parallelism and FP8 KV caches.
  • Measured on three GPUs across two generations (L40S, A10G, L4: Ada and Ampere): dense models within 0.5% of what vLLM reports at 32K. H100, A100 and vLLM 0.34 are next.
  • Shared GPUs: --reserve-gib suggests a --gpu-memory-utilization setting when other processes (OCR, embeddings, a second model) share the card.
  • Warns you before vLLM refuses to start a hybrid model with too many request slots.
  • No dependencies.
pip install gpu-fit
gpu-fit Qwen/Qwen3-32B --context 128k --concurrency 4

GitHub · PyPI · Browser calculator

gpu-guard in progressGo

A small health check for self-hosted LLM servers (vLLM, Ollama and other OpenAI-compatible APIs) that fails when a model’s speed drops below a threshold you calibrate (a common sign it has fallen back to the CPU) and can check Ollama’s GPU placement. Recovery steps are next on the roadmap.

go install github.com/KiranTallapragada/gpu-guard@latest
gpu-guard calibrate --url http://localhost:8000/v1

GitHub

My production work on trading systems and smart contracts belongs to the organisations I built it for, so open-source work like this is written from scratch for anyone to read and use.