A3000 Laptop 6GB: Recommendations very poor

[Sophist-UK]

Description

I have been pondering which LLM to run (when I have time) for some months, and watched as new models get announced and new ways of running them are announced (MTP) and new distilled versions are announced etc. and I have read a lot of Reddit posts for people wanting to do similar things with similar hardware. So I have a reasonable idea of what might be best for my hardware.

The output is below...

For some reason it failed to download some stuff and gave an error. But when I ran it again it didn't give an error but gave the exact same results.

A few things in the results that stood out:

• The specific use case(s) matter - but there is no way for me to state I want e.g. agentic coding
• Qwen3.6 27B dense rather than Qwen3.6 35B A3B MoE which would run much better with hybrid inference.
• No TPS estimates - which are absolutely essential for evaluating LLMs - 35Tps vs. 1Tps is a huge impact
• Q8 rather than Q5 or Q6? Really?
• No MTP evaluations
• No distil evaluations
• No data regarding which runner should be used with which params

Steps to Reproduce

Run whichllm.

Hardware Info

Leaderboard fetch failed: Client error '429 Too Many Requests' for url 'https://datasets-server.huggingface.co/rows?dataset=open-llm-leaderboard%2Fcontents&config=default&split=train&offset=3800&length=100'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429
AA Index fetch failed, will use fallback: __NEXT_DATA__ payload not found

╭──────────────────────────────────────────────────────────────────────────────────────────────── Hardware Info ────────────────────────────────────────────────────────────────────────────────────────────────╮
│ GPU 0: NVIDIA RTX A3000 Laptop GPU — 6.0 GB (CUDA 13.2) — BW: N/A                                                                                                                                             │
│ GPU 1: Intel(R) UHD Graphics — shared memory — BW: N/A                                                                                                                                                        │
│ CPU: Unknown CPU — 8 cores (AVX2)                                                                                                                                                                             │
│ RAM: 31.3 GB                                                                                                                                                                                                  │
│ Disk free: 224.8 GB                                                                                                                                                                                           │
│ OS: windows                                                                                                                                                                                                   │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

                                                 Recommended Models
┏━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┓
┃   # ┃ Model                                         ┃ Params ┃ Quant  ┃ Published  ┃ Downloads ┃ Score ┃ License  ┃
┡━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━┩
│   1 │ Qwen/Qwen3.6-27B                              │  27.8B │  Q8_0  │ 2026-04-21 │      5.2M │  56.7 │ apache-… │
├─────┼───────────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│   2 │ google/gemma-4-31B-it                         │  32.7B │  Q6_K  │ 2026-03-11 │     11.3M │  54.3 │ apache-… │
├─────┼───────────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│   3 │ google/gemma-4-26B-A4B-it                     │  26.5B │  Q8_0  │ 2026-03-11 │     11.5M │  47.5 │ apache-… │
│     │                                               │ (3.8B… │        │            │           │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│   4 │ Qwen/Qwen3-30B-A3B                            │  30.5B │  Q6_K  │ 2025-04-27 │      2.1M │  47.5 │ apache-… │
│     │                                               │ (3.0B… │        │            │           │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│   5 │ zai-org/GLM-4.7-Flash                         │  31.2B │  Q6_K  │ 2026-01-19 │      1.1M │  45.8 │ mit      │
│     │                                               │ (12.0… │        │            │           │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│   6 │ Qwen/QwQ-32B                                  │  32.8B │  Q6_K  │ 2025-03-05 │     62.5K │  45.4 │ apache-… │
├─────┼───────────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│   7 │ openai/gpt-oss-20b                            │  21.5B │  Q8_0  │ 2025-08-04 │      7.9M │  45.0 │ apache-… │
│     │                                               │ (3.6B… │        │            │           │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│   8 │ deepseek-ai/DeepSeek-R1-Distill-Qwen-32B      │  32.8B │  Q6_K  │ 2025-01-20 │    608.3K │  44.6 │ mit      │
├─────┼───────────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│   9 │ mistralai/Mistral-Small-3.2-24B-Instruct-2506 │  24.0B │  Q8_0  │ 2025-06-19 │    632.7K │  43.9 │ apache-… │
├─────┼───────────────────────────────────────────────┼────────┼────────┼────────────┼───────────┼───────┼──────────┤
│  10 │ Qwen/Qwen3-14B                                │  14.8B │  Q8_0  │ 2025-04-27 │      1.7M │  43.3 │ apache-… │
└─────┴───────────────────────────────────────────────┴────────┴────────┴────────────┴───────────┴───────┴──────────┘
  Top pick confidence: Low (direct benchmark, gap +2.3, partial offload)
  Benchmark reference: 2026-05 curated snapshot; live AA / LiveBench / Aider merged when reachable.
  Speed caution: Low-confidence speed estimates in top ranks: #1, #2, #3
  Warning #1 Qwen3.6-27B: ~81% of layers will be offloaded to CPU RAM
  Warning #2 gemma-4-31B-it: ~79% of layers will be offloaded to CPU RAM
  Warning #3 gemma-4-26B-A4B-it: ~78% of layers will be offloaded to CPU RAM

Python Version

3.14

Operating System

Windows 11

whichllm Version

0.5.7

[Andyyyy64]

Thanks for the detailed report. I am sorry the recommendation was this far off for your setup.

The Q8 suggestion on a 6GB A3000 does look wrong, especially for agentic coding where latency and usable context matter a lot.

Could you paste these outputs?

whichllm --version
whichllm hardware
whichllm --status --top 20 --json

--status should show tok/s estimates, but if that was easy to miss, the default output is probably hiding too much. I will use this as a regression case for the 6GB laptop GPU ranking.

[Sophist-UK]

C:\Users\Paul>whichllm --version
0.5.7

C:\Users\Paul>whichllm hardware

╭──────────────────────────────────────────────────────────────────────────────────────────────── Hardware Info ────────────────────────────────────────────────────────────────────────────────────────────────╮
│ GPU 0: NVIDIA RTX A3000 Laptop GPU — 6.0 GB (CUDA 13.2) — BW: N/A                                                                                                                                             │
│ GPU 1: Intel(R) UHD Graphics — shared memory — BW: N/A                                                                                                                                                        │
│ CPU: Unknown CPU — 8 cores (AVX2)                                                                                                                                                                             │
│ RAM: 31.3 GB                                                                                                                                                                                                  │
│ Disk free: 225.1 GB                                                                                                                                                                                           │
│ OS: windows                                                                                                                                                                                                   │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯


C:\Users\Paul>whichllm --status --top 20 --json
Leaderboard fetch failed: Client error '429 Too Many Requests' for url 'https://datasets-server.huggingface.co/rows?dataset=open-llm-leaderboard%2Fcontents&config=default&split=train&offset=4100&length=100'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429
AA Index fetch failed, will use fallback: __NEXT_DATA__ payload not found
{
  "hardware": {
    "gpus": [
      {
        "name": "NVIDIA RTX A3000 Laptop GPU",
        "vendor": "nvidia",
        "vram_bytes": 6442450944,
        "memory_bandwidth_gbps": null,
        "shared_memory": false
      },
      {
        "name": "Intel(R) UHD Graphics",
        "vendor": "intel",
        "vram_bytes": 0,
        "memory_bandwidth_gbps": null,
        "shared_memory": true
      }
    ],
    "cpu": "Unknown CPU",
    "cpu_cores": 8,
    "ram_bytes": 33587777536,
    "os": "windows"
  },
  "models": [
    {
      "rank": 1,
      "model_id": "Qwen/Qwen3.6-27B",
      "parameter_count": 27781427952,
      "published_at": "2026-04-21T07:50:43.000Z",
      "downloads": 5377567,
      "quant_type": "Q8_0",
      "file_size_bytes": 29517767199,
      "vram_required_bytes": 33198114575,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 56.68,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~81% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 2,
      "model_id": "google/gemma-4-31B-it",
      "parameter_count": 32682372656,
      "published_at": "2026-03-11T18:22:36.000Z",
      "downloads": 11271936,
      "quant_type": "Q6_K",
      "file_size_bytes": 26554427783,
      "vram_required_bytes": 30698796917,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 54.34,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~79% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 3,
      "model_id": "google/gemma-4-26B-A4B-it",
      "parameter_count": 26544131376,
      "published_at": "2026-03-11T21:25:57.000Z",
      "downloads": 11696495,
      "quant_type": "Q8_0",
      "file_size_bytes": 28203139587,
      "vram_required_bytes": 29780276559,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 47.51,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~78% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 4,
      "model_id": "Qwen/Qwen3-30B-A3B",
      "parameter_count": 30532122624,
      "published_at": "2025-04-27T03:43:05.000Z",
      "downloads": 2133276,
      "quant_type": "Q6_K",
      "file_size_bytes": 24807349632,
      "vram_required_bytes": 26273510400,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 47.48,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~75% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 5,
      "model_id": "zai-org/GLM-4.7-Flash",
      "parameter_count": 31221488576,
      "published_at": "2026-01-19T06:28:10.000Z",
      "downloads": 1107961,
      "quant_type": "Q6_K",
      "file_size_bytes": 25367459468,
      "vram_required_bytes": 28082102540,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 45.75,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~77% of layers will be offloaded to CPU RAM"
      ],
      "license": "mit"
    },
    {
      "rank": 6,
      "model_id": "Qwen/QwQ-32B",
      "parameter_count": 32763876352,
      "published_at": "2025-03-05T14:16:59.000Z",
      "downloads": 61530,
      "quant_type": "Q6_K",
      "file_size_bytes": 26620649536,
      "vram_required_bytes": 30772735445,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 45.39,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~79% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 7,
      "model_id": "openai/gpt-oss-20b",
      "parameter_count": 21511953984,
      "published_at": "2025-08-04T22:33:29.000Z",
      "downloads": 7877081,
      "quant_type": "Q8_0",
      "file_size_bytes": 22856451108,
      "vram_required_bytes": 24405844029,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 45.01,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~74% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 8,
      "model_id": "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B",
      "parameter_count": 32763876352,
      "published_at": "2025-01-20T09:19:00.000Z",
      "downloads": 595825,
      "quant_type": "Q6_K",
      "file_size_bytes": 26620649536,
      "vram_required_bytes": 30772735445,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 44.65,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~79% of layers will be offloaded to CPU RAM"
      ],
      "license": "mit"
    },
    {
      "rank": 9,
      "model_id": "mistralai/Mistral-Small-3.2-24B-Instruct-2506",
      "parameter_count": 24011361280,
      "published_at": "2025-06-19T17:41:13.000Z",
      "downloads": 625168,
      "quant_type": "Q8_0",
      "file_size_bytes": 25512071360,
      "vram_required_bytes": 28835468582,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 43.94,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~78% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 10,
      "model_id": "Qwen/Qwen3-14B",
      "parameter_count": 14768307200,
      "published_at": "2025-04-27T03:42:45.000Z",
      "downloads": 1812505,
      "quant_type": "Q8_0",
      "file_size_bytes": 15691326400,
      "vram_required_bytes": 18139590670,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 43.32,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~64% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 11,
      "model_id": "microsoft/phi-4",
      "parameter_count": 14659507200,
      "published_at": "2024-12-11T11:47:29.000Z",
      "downloads": 891487,
      "quant_type": "Q8_0",
      "file_size_bytes": 15575726400,
      "vram_required_bytes": 18013689479,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 41.75,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~64% of layers will be offloaded to CPU RAM"
      ],
      "license": "mit"
    },
    {
      "rank": 12,
      "model_id": "Qwen/Qwen3-32B",
      "parameter_count": 32762123264,
      "published_at": "2025-04-27T03:52:59.000Z",
      "downloads": 4660504,
      "quant_type": "Q6_K",
      "file_size_bytes": 26619225152,
      "vram_required_bytes": 30771145079,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 41.61,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~79% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 13,
      "model_id": "google/gemma-3-27b-it",
      "parameter_count": 27432406640,
      "published_at": "2025-03-01T19:10:19.000Z",
      "downloads": 1443220,
      "quant_type": "Q8_0",
      "file_size_bytes": 29146932055,
      "vram_required_bytes": 32794234071,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 41.31,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~80% of layers will be offloaded to CPU RAM"
      ],
      "license": "gemma"
    },
    {
      "rank": 14,
      "model_id": "ibm-granite/granite-4.0-h-small",
      "parameter_count": 32207337984,
      "published_at": "2025-09-16T19:50:49.000Z",
      "downloads": 579921,
      "quant_type": "Q6_K",
      "file_size_bytes": 26168462112,
      "vram_required_bytes": 30267854932,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 35.29,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~79% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 15,
      "model_id": "google/gemma-3-12b-it",
      "parameter_count": 12187325040,
      "published_at": "2025-03-01T19:11:34.000Z",
      "downloads": 2955855,
      "quant_type": "Q8_0",
      "file_size_bytes": 12949032855,
      "vram_required_bytes": 15152929569,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 33.17,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~57% of layers will be offloaded to CPU RAM"
      ],
      "license": "gemma"
    },
    {
      "rank": 16,
      "model_id": "deepseek-ai/DeepSeek-R1-Distill-Qwen-14B",
      "parameter_count": 14770033664,
      "published_at": "2025-01-20T09:18:27.000Z",
      "downloads": 556565,
      "quant_type": "Q8_0",
      "file_size_bytes": 15693160768,
      "vram_required_bytes": 18141588500,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 32.24,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~64% of layers will be offloaded to CPU RAM"
      ],
      "license": "mit"
    },
    {
      "rank": 17,
      "model_id": "batiai/Qwen3.6-35B-A3B-GGUF",
      "parameter_count": 34660610688,
      "published_at": "2026-04-16T18:35:23.000Z",
      "downloads": 17564,
      "quant_type": "Q6_K",
      "file_size_bytes": 28161746184,
      "vram_required_bytes": 29627906952,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 34.32,
      "benchmark_status": "estimated",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~78% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 18,
      "model_id": "Qwen/Qwen2.5-14B-Instruct",
      "parameter_count": 14770033664,
      "published_at": "2024-09-16T11:56:10.000Z",
      "downloads": 2222936,
      "quant_type": "Q8_0",
      "file_size_bytes": 15693160768,
      "vram_required_bytes": 18141588500,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 29.25,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~64% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    },
    {
      "rank": 19,
      "model_id": "mistralai/Codestral-22B-v0.1",
      "parameter_count": 22247282688,
      "published_at": "2024-05-29T11:28:50.000Z",
      "downloads": 14143,
      "quant_type": "Q8_0",
      "file_size_bytes": 23637737856,
      "vram_required_bytes": 26794112004,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 28.5,
      "benchmark_status": "direct",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~76% of layers will be offloaded to CPU RAM"
      ],
      "license": "other"
    },
    {
      "rank": 20,
      "model_id": "AtomicChat/gemma-4-31B-it-assistant-GGUF",
      "parameter_count": 31000000000,
      "published_at": "2026-05-07T08:56:23.000Z",
      "downloads": 19448,
      "quant_type": "Q5_K_M",
      "file_size_bytes": 21312500000,
      "vram_required_bytes": 25297581984,
      "estimated_tok_per_sec": 0.0,
      "speed_confidence": "low",
      "speed_range_tok_per_sec": null,
      "speed_notes": [
        "Speed is estimated from memory bandwidth, quantization, backend, and fit type.",
        "No usable bandwidth estimate was available for this setup."
      ],
      "quality_score": 32.01,
      "benchmark_status": "estimated",
      "fit_type": "partial_offload",
      "can_run": true,
      "warnings": [
        "~75% of layers will be offloaded to CPU RAM"
      ],
      "license": "apache-2.0"
    }
  ]
}

[Sophist-UK]

C:\Users\Paul>whichllm --status

╭──────────────────────────────────────────────────────────────────────────────────────────────── Hardware Info ────────────────────────────────────────────────────────────────────────────────────────────────╮
│ GPU 0: NVIDIA RTX A3000 Laptop GPU — 6.0 GB (CUDA 13.2) — BW: N/A                                                                                                                                             │
│ GPU 1: Intel(R) UHD Graphics — shared memory — BW: N/A                                                                                                                                                        │
│ CPU: Unknown CPU — 8 cores (AVX2)                                                                                                                                                                             │
│ RAM: 31.3 GB                                                                                                                                                                                                  │
│ Disk free: 225.1 GB                                                                                                                                                                                           │
│ OS: windows                                                                                                                                                                                                   │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

                                                     Recommended Models
┏━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━┓
┃   # ┃ Model                                         ┃ Params ┃ Quant  ┃     VRAM ┃    Speed ┃   Fit   ┃ Score ┃ License  ┃
┡━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━┩
│   1 │ Qwen/Qwen3.6-27B                              │  27.8B │  Q8_0  │  30.9 GB │      0.0 │ Partial │  56.7 │ apache-… │
│     │                                               │        │        │          │  tok/s ? │         │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼──────────┼──────────┼─────────┼───────┼──────────┤
│   2 │ google/gemma-4-31B-it                         │  32.7B │  Q6_K  │  28.6 GB │      0.0 │ Partial │  54.3 │ apache-… │
│     │                                               │        │        │          │  tok/s ? │         │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼──────────┼──────────┼─────────┼───────┼──────────┤
│   3 │ google/gemma-4-26B-A4B-it                     │  26.5B │  Q8_0  │  27.7 GB │      0.0 │ Partial │  47.5 │ apache-… │
│     │                                               │ (3.8B… │        │          │  tok/s ? │         │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼──────────┼──────────┼─────────┼───────┼──────────┤
│   4 │ Qwen/Qwen3-30B-A3B                            │  30.5B │  Q6_K  │  24.5 GB │      0.0 │ Partial │  47.5 │ apache-… │
│     │                                               │ (3.0B… │        │          │  tok/s ? │         │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼──────────┼──────────┼─────────┼───────┼──────────┤
│   5 │ zai-org/GLM-4.7-Flash                         │  31.2B │  Q6_K  │  26.2 GB │      0.0 │ Partial │  45.8 │ mit      │
│     │                                               │ (12.0… │        │          │  tok/s ? │         │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼──────────┼──────────┼─────────┼───────┼──────────┤
│   6 │ Qwen/QwQ-32B                                  │  32.8B │  Q6_K  │  28.7 GB │      0.0 │ Partial │  45.4 │ apache-… │
│     │                                               │        │        │          │  tok/s ? │         │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼──────────┼──────────┼─────────┼───────┼──────────┤
│   7 │ openai/gpt-oss-20b                            │  21.5B │  Q8_0  │  22.7 GB │      0.0 │ Partial │  45.0 │ apache-… │
│     │                                               │ (3.6B… │        │          │  tok/s ? │         │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼──────────┼──────────┼─────────┼───────┼──────────┤
│   8 │ deepseek-ai/DeepSeek-R1-Distill-Qwen-32B      │  32.8B │  Q6_K  │  28.7 GB │      0.0 │ Partial │  44.6 │ mit      │
│     │                                               │        │        │          │  tok/s ? │         │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼──────────┼──────────┼─────────┼───────┼──────────┤
│   9 │ mistralai/Mistral-Small-3.2-24B-Instruct-2506 │  24.0B │  Q8_0  │  26.9 GB │      0.0 │ Partial │  43.9 │ apache-… │
│     │                                               │        │        │          │  tok/s ? │         │       │          │
├─────┼───────────────────────────────────────────────┼────────┼────────┼──────────┼──────────┼─────────┼───────┼──────────┤
│  10 │ Qwen/Qwen3-14B                                │  14.8B │  Q8_0  │  16.9 GB │      0.0 │ Partial │  43.3 │ apache-… │
│     │                                               │        │        │          │  tok/s ? │         │       │          │
└─────┴───────────────────────────────────────────────┴────────┴────────┴──────────┴──────────┴─────────┴───────┴──────────┘
  Speed:  ? = low-confidence/backend-sensitive tok/s
  Top pick confidence: Low (direct benchmark, gap +2.3, partial offload)
  Benchmark reference: 2026-05 curated snapshot; live AA / LiveBench / Aider merged when reachable.
  Speed caution: Low-confidence speed estimates in top ranks: #1, #2, #3
  Warning #1 Qwen3.6-27B: ~81% of layers will be offloaded to CPU RAM
  Warning #2 gemma-4-31B-it: ~79% of layers will be offloaded to CPU RAM
  Warning #3 gemma-4-26B-A4B-it: ~78% of layers will be offloaded to CPU RAM

[Andyyyy64]

Thanks, this is enough to reproduce the bad path.

The main failure is clearer now. The A3000 Laptop GPU is detected, but bandwidth is N/A, so every speed estimate collapses to 0.0 tok/s ?. After that, the ranker is still putting huge partial-offload models at the top, even when 75-81% of the model would be offloaded to CPU RAM. That is not a useful recommendation for a 6GB laptop GPU.

There are a few bugs here, not just one:

• NVIDIA RTX A3000 Laptop GPU needs a bandwidth/catalog entry instead of N/A
• Windows CPU detection is still too weak here (Unknown CPU)
• partial offload needs a much stronger penalty when most layers are offloaded
• models with 0.0 tok/s or unknown bandwidth should not be top-ranked
• the default output makes the problem too easy to miss; --status shows it, but the normal table hides the speed failure

The Hugging Face 429 is separate. It is noisy and should be handled better, but it is not the main reason these recommendations are bad.

This gives me a concrete regression case for the 6GB laptop GPU path. I’ll work from this.

[Andyyyy64]

I split the concrete follow-ups out so this does not turn into one giant issue:

• Add bandwidth entry for NVIDIA RTX A3000 Laptop GPU #78: A3000 Laptop GPU bandwidth/catalog entry
• Do not top-rank unknown-speed heavy partial-offload models #79: ranking penalty for unknown-speed heavy partial offload
• Improve Windows CPU name detection fallback #80: Windows CPU name detection fallback

I linked all three as sub-issues of this one.

[Sophist-UK]

From sysinfo:

Processor	Intel(R) Xeon(R) W-11955M CPU @ 2.60GHz, 2611 Mhz, 8 Core(s), 16 Logical Processor(s)

I think it may actually be an i9 11th gen, since this is a laptop rather than a Xeon.

In terms of the weaknesses you identified:

a. A3B MoE layers can be offloaded to CPU with far less impact than full layers. Your algorithm needs to understand different types of layer and their impact and on how to prioritise which to offload first.

b. As I said there are literally hundreds of variants on HF for each base model like Qwen3.,6 35B A3B - different quants, different types of quants, MTP or not (or similar), flash attention, retrained and / or distilled etc. etc. etc. And this is a rapidly moving target with new things coming along all the time and different runners supporting them at different times etc. I am not sure just how much data there is on these variants and their performance characteristics. So perhaps you need to stick to simple quants to narrow it down to a single model and basic quant, and then simply list the variants so someone can do their own research to narrow it down.

And judging only by the number of posts on Reddit, it seems like 6GB is a pretty popular hardware point - if you are buying hardware for AI then you would probably go for something bigger (with hindsight I could have got a 16GB A5000 version of the same laptop, perhaps with an i7 instead of an i9, for about the same money), but a lot of people are asking whatllm?(TM) for the hardware they already have,

[Andyyyy64]

Thanks, this is very useful.

The sysinfo line gives me a concrete CPU fallback case. Intel(R) Xeon(R) W-11955M CPU @ 2.60GHz is enough for #80. I will not try to rewrite it to i9; it may be a mobile workstation Xeon SKU, but either way whichllm should not show Unknown CPU.

I agree on the MoE point. The current partial-offload logic is too blunt. It mostly sees "how much of the model is offloaded", while A3B/MoE models care about what is offloaded: active experts, inactive experts, router/KV/cache/runtime choices do not hurt equally. I do not want the first fix to pretend it can model runner-specific layer placement, but #79 should at least stop unknown-speed / heavy-offload MoE picks from floating to the top.

On variants, I think you are right about the safer UX. For tight hardware like 6GB, whichllm should probably recommend a conservative base model + simple quant first, then show nearby variants/siblings as "look at these too", rather than pretending it has enough data to pick the perfect MTP/flash-attention/retrained variant.

And yes, 6GB should not be treated as an edge case. A lot of people are asking exactly the whatllm question for the hardware they already own. I will keep this issue as the 6GB laptop GPU regression case.

[Sophist-UK]

I really don't understand what the difficulty is 😉 with being able to give a perfect answer. After all there are only 868,638,944,834 different hardware combinations ... oops, sorry, by the time I had typed this it was 868,639,689,164 combinations and that is also now out of date.