Project Website | 📄 Paper (pending) | 🛠️ GitHub
This repository includes code and materials for the paper "SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops" (submitted to arXiv).
@misc{zhang2026siliconbench,
title = {{SiliconBench}: Speed, Memory, and Fidelity for {LLM} Serving on Unified-Memory Desktops},
author = {Ranran Haoran Zhang and Aysa Xuemo Fan and David Munh{\'a} Correia and Alex Cheema and Rui Zhang},
year = {2026},
note = {Submitted to arXiv}
}SiliconBench evaluates local LLM serving through three lenses: speed, memory, and fidelity. Throughput and latency measure performance under concurrent load; memory measurements show how much headroom remains for other applications; and a classification task checks for quality regressions against an NVIDIA reference.
The main audit covers nine Apple Silicon serving engines on chat and agent workloads, using Qwen3, Qwen3.5, and Gemma 4. A complementary NVIDIA DGX Spark track evaluates serving performance for three shared engine families. The benchmark page presents the latest recorded results and run details.
Open this repository in your coding agent and ask it to run the benchmark skill:
Read
.claude/skills/weekly-bench/SKILL.mdand run SiliconBench.
In Claude Code, invoke the skill directly with /weekly-bench.
The Apple Silicon audit includes:
The DGX Spark track covers llama.cpp, vLLM, and SGLang. The paper's multi-node study also includes EXO, alongside MLX LM and llama.cpp.
Original SiliconBench harness code and documentation are licensed under the MIT License. Third-party inference engines, model weights, and source datasets retain their upstream licenses and access conditions. This license does not relicense those assets or third-party text retained in prompts and outputs.