Running Tencent Hy3 (295B MoE, IQ1_M + MTP) on a MacBook Pro M3 Max 128GB
I am aware that the discussions on HF are low quality, but decided to make a post anyway. In case someone wants to run hy3 with similar hardware to mine.
Observations
- It runs. A 295B MoE model running on a laptop with usable output speed. That alone is remarkable.
- KV cache is everything. The first request took 2:38 to ingest 17K tokens. Every request after that matched at 0.99+ LCP similarity and reprocessed in 1-8 seconds. Without prefix caching, this would be unusable interactively.
- Parallelism kills throughput. Single slot: 10 t/s. Three concurrent slots: 3.4 t/s. For an agent workload where multiple tools fire simultaneously, this is a real bottleneck. Sequential requests are the sweet spot.
- MTP helps but modestly. 47% average acceptance with 2.5-token mean is a ~1.5x throughput boost over no MTP, but not the 2-3x you'd see with a well-calibrated MTP head. The model's gating is conservative.
- Context size vs. cache limit. At 55K context with 23K prompts, the KV state can exceed the default 8 GB cache, forcing a full reprocessing. Tuning --cache-ram is important for long-context workloads.
- IQ1_M quality. Despite 1.75 bpw, the model produced coherent, grammatically correct output throughout. Reasoning traces were structured and logical. The AngelSlim mixed-recipe quantization (critical weights in Q8_0/Q6_K, experts in IQ1_M) clearly preserves the important weight distributions.
Build notes
- llama.cpp tag b10045 (July 16, 2026) includes PR #25395 (hy_v3 + MTP support)
- Built with -DLLAMA_METAL=ON on macOS
- One patch required: exp_probs_b → exp_probs_b.bias tensor name in src/models/hy-v3.cpp to match AngelSlim's GGUF
Hardware
| Spec | Value |
|---|---|
| Model | MacBook Pro (Mac15,9) |
| Chip | Apple M3 Max |
| CPU | 16 cores (12P + 4E) |
| GPU | 40 cores, Metal 4 |
| Unified Memory | 128 GB |
| Model | Hy3-IQ1_M-mtp.gguf (85 GB) |
| Quantization | IQ1_M (1.75 bpw) with MTP |
| Architecture | hy_v3 (192 experts, 8 active, 81 layers) |
| Runtime | llama.cpp b10045 (Metal, PR #25395) |
setup
llama-server \
-m ~/models/Hy3-IQ1_M-mtp.gguf \
-c 55536 \
-ngl 99 \
-fa on \
--spec-type draft-mtp \
--temp 0.6 \
--port 8080
All 99 layers on Metal, flash attention on, MTP speculative decoding enabled, 4 parallel slots at 55K context each, unified KV cache.
Note: A patch was needed, AngelSlim's GGUF stores exp_probs_b.bias but PR #25395 expected exp_probs_b (no .bias suffix). Fixed in src/models/hy-v3.cpp, rebuilt, and the model loaded cleanly. Details here. (https://github.com/ggml-org/llama.cpp/pull/25395)
results
Prefill (prompt processing)
| Scenario | Tokens | Speed |
|---|---|---|
| Cold start (first request) | 17,289 | 109 t/s |
| Warm cache hit (LCP 0.99+) | 13-639 | 50-120 t/s |
| Full cache miss | 22,965 | 84 t/s |
KV cache via LCP (longest common prefix) similarity works well, once the initial system prompt is ingested, subsequent requests with the same prefix match at 0.98-0.99+ similarity and reprocess in seconds instead of minutes.
One request hit the 8 GB KV cache limit (8293 MiB > 8192 MiB limit), causing a full 23K-token reprocessing at 84 t/s. Bumping --cache-ram or reducing context would avoid this.
Generation (token decoding)
| Concurrency | Speed |
|---|---|
| 1 slot active | 9-10 t/s |
| 2 slots active | 7-8 t/s |
| 3 slots active | 3-5 t/s |
Generation is bandwidth-bound. The 295B MoE activates 8 of 192 experts per token, each decode step touches massive weight regions. With a single slot, 9-10 t/s is consistent. Three concurrent slots saturate Metal memory bandwidth and drop to 3.4 t/s.
MTP Speculative Decoding
| Metric | Value |
|---|---|
| Acceptance rate | 35-67%, avg ~47% |
| Mean accepted draft length | 2.1-2.7 tokens |
| Graphs reused | up to 1,183 per session |
MTP acceptance is notably lower than Qwen3.6-27B's 90%+ on the same machine. This aligns with the satgeze/Hy3-1M-GGUF (https://huggingface.co/satgeze/Hy3-1M-GGUF) findings that Hy3's MTP head is "inherently confidence-gated". The draft head only proposes when it's confident, so many draft steps produce zero accepted tokens. Adding --spec-draft-p-min 0.75 should push acceptance to ~88% (per their audit), though the raw t/s improvement may be modest given the short accepted lengths (mean 2.5).
Session totals
| Metric | Value |
|---|---|
| Runtime | 33 min 51 sec |
| Total tasks | ~24 API requests |
| Max concurrent slots | 3 of 4 |
| Largest prompt | 23K tokens |
| Final context size | 26K+ tokens |
| Total tokens processed | ~60K+ across all tasks |
I am aware that the discussions on HF are low quality, but decided to make a post anyway. In case someone wants to run hy3 with similar hardware to mine.
Observations
- It runs. A 295B MoE model running on a laptop with usable output speed. That alone is remarkable.
- KV cache is everything. The first request took 2:38 to ingest 17K tokens. Every request after that matched at 0.99+ LCP similarity and reprocessed in 1-8 seconds. Without prefix caching, this would be unusable interactively.
- Parallelism kills throughput. Single slot: 10 t/s. Three concurrent slots: 3.4 t/s. For an agent workload where multiple tools fire simultaneously, this is a real bottleneck. Sequential requests are the sweet spot.
- MTP helps but modestly. 47% average acceptance with 2.5-token mean is a ~1.5x throughput boost over no MTP, but not the 2-3x you'd see with a well-calibrated MTP head. The model's gating is conservative.
- Context size vs. cache limit. At 55K context with 23K prompts, the KV state can exceed the default 8 GB cache, forcing a full reprocessing. Tuning --cache-ram is important for long-context workloads.
- IQ1_M quality. Despite 1.75 bpw, the model produced coherent, grammatically correct output throughout. Reasoning traces were structured and logical. The AngelSlim mixed-recipe quantization (critical weights in Q8_0/Q6_K, experts in IQ1_M) clearly preserves the important weight distributions.
Build notes
- llama.cpp tag b10045 (July 16, 2026) includes PR #25395 (hy_v3 + MTP support)
- Built with -DLLAMA_METAL=ON on macOS
- One patch required: exp_probs_b → exp_probs_b.bias tensor name in src/models/hy-v3.cpp to match AngelSlim's GGUF
Hardware
Spec Value Model MacBook Pro (Mac15,9) Chip Apple M3 Max CPU 16 cores (12P + 4E) GPU 40 cores, Metal 4 Unified Memory 128 GB Model Hy3-IQ1_M-mtp.gguf (85 GB) Quantization IQ1_M (1.75 bpw) with MTP Architecture hy_v3 (192 experts, 8 active, 81 layers) Runtime llama.cpp b10045 (Metal, PR #25395) setup
llama-server \ -m ~/models/Hy3-IQ1_M-mtp.gguf \ -c 55536 \ -ngl 99 \ -fa on \ --spec-type draft-mtp \ --temp 0.6 \ --port 8080All 99 layers on Metal, flash attention on, MTP speculative decoding enabled, 4 parallel slots at 55K context each, unified KV cache.
Note: A patch was needed, AngelSlim's GGUF stores exp_probs_b.bias but PR #25395 expected exp_probs_b (no .bias suffix). Fixed in src/models/hy-v3.cpp, rebuilt, and the model loaded cleanly. Details here. (https://github.com/ggml-org/llama.cpp/pull/25395)results
Prefill (prompt processing)
Scenario Tokens Speed Cold start (first request) 17,289 109 t/s Warm cache hit (LCP 0.99+) 13-639 50-120 t/s Full cache miss 22,965 84 t/s KV cache via LCP (longest common prefix) similarity works well, once the initial system prompt is ingested, subsequent requests with the same prefix match at 0.98-0.99+ similarity and reprocess in seconds instead of minutes.
One request hit the 8 GB KV cache limit (8293 MiB > 8192 MiB limit), causing a full 23K-token reprocessing at 84 t/s. Bumping --cache-ram or reducing context would avoid this.Generation (token decoding)
Concurrency Speed 1 slot active 9-10 t/s 2 slots active 7-8 t/s 3 slots active 3-5 t/s Generation is bandwidth-bound. The 295B MoE activates 8 of 192 experts per token, each decode step touches massive weight regions. With a single slot, 9-10 t/s is consistent. Three concurrent slots saturate Metal memory bandwidth and drop to 3.4 t/s.
MTP Speculative Decoding
Metric Value Acceptance rate 35-67%, avg ~47% Mean accepted draft length 2.1-2.7 tokens Graphs reused up to 1,183 per session MTP acceptance is notably lower than Qwen3.6-27B's 90%+ on the same machine. This aligns with the satgeze/Hy3-1M-GGUF (https://huggingface.co/satgeze/Hy3-1M-GGUF) findings that Hy3's MTP head is "inherently confidence-gated". The draft head only proposes when it's confident, so many draft steps produce zero accepted tokens. Adding --spec-draft-p-min 0.75 should push acceptance to ~88% (per their audit), though the raw t/s improvement may be modest given the short accepted lengths (mean 2.5).
Session totals
Metric Value Runtime 33 min 51 sec Total tasks ~24 API requests Max concurrent slots 3 of 4 Largest prompt 23K tokens Final context size 26K+ tokens Total tokens processed ~60K+ across all tasks
Thanks for sharing these detailed results! The lower MTP acceptance rate may be due to the combined quantization error from both the main model weights and the MTP head. It can also be workload-dependent—for example, it may perform better on code-generation tasks. We’ll continue exploring other speculative decoding approaches, such as DFlash, going forward. We’ve also updated the GGUF files to align the exp_probs_b tensor naming with upstream llama.cpp. If you download the latest GGUF files, the source patch is no longer required.
Just wanted to show more examples that this runs really well. Was just trying to show this off to more people since its a very good model with very stable token distribution. Was running about 25tps on mac m3 ultra with mtp but probably because acceptence was low. https://www.reddit.com/r/LocalLLaMA/s/8xHXUWzWlG