通过 vllm 部署 Qwen3.8-27B-FP8 模型
下载模型
Qwen3.8-27B-FP8 模型文件大小 30.89GB
使用 modelscope 下载模型
https://www.modelscope.cn/models/Qwen/Qwen3.8-27B-FP8
modelscope download --model Qwen/Qwen3.8-27B-FP8 README.md --local_dir /data/models/Qwen3.8-27B-FP8
启动 Qwen3.8-27B-FP8
https://recipes.vllm.ai/Qwen/Qwen3.8-27B?variant=fp8
CUDA_VISIBLE_DEVICES=0,1 vllm serve \
/data/models/Qwen3.8-27B-FP8/ \
--host 0.0.0.0 \
--port 10304 \
--tensor-parallel-size 2 \
--gpu-memory-utilization 0.9 \
--max-model-len 131072 \
--enforce-eager \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--served-model-name Qwen3.8-27B-FP8
–enable-auto-tool-choice –tool-call-parser qwen3_coder 参数用于支持 tool call
查看显存占用情况
$ nvidia-smi
Mon Aug 17 16:11:57 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.84 Driver Version: 595.84 CUDA Version: 13.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4090 D Off | 00000000:01:00.0 On | Off |
| 57% 61C P2 202W / 425W | 22351MiB / 24564MiB | 85% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 4090 D Off | 00000000:08:00.0 Off | Off |
| 34% 55C P2 190W / 425W | 21980MiB / 24564MiB | 99% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 2080 G /usr/lib/xorg/Xorg 228MiB |
| 0 N/A N/A 2702 G /usr/bin/gnome-shell 25MiB |
| 0 N/A N/A 61076 C VLLM::Worker_TP0 21956MiB |
| 1 N/A N/A 2080 G /usr/lib/xorg/Xorg 4MiB |
| 1 N/A N/A 61077 C VLLM::Worker_TP1 21956MiB |
+-----------------------------------------------------------------------------------------+
在 claude code 中使用
~/.claude/setting.json
{
"effortLevel": "xhigh",
"enabledPlugins": {
"gopls-lsp@claude-plugins-official": true
},
"env": {
"ANTHROPIC_API_KEY": "dummy",
"ANTHROPIC_BASE_URL": "http://192.168.110.28:10304",
"ANTHROPIC_DEFAULT_FABLE_MODEL": "Qwen3.8-27B-FP8",
"ANTHROPIC_DEFAULT_FABLE_MODEL_NAME": "Qwen3.8-27B-FP8",
"ANTHROPIC_DEFAULT_HAIKU_MODEL": "Qwen3.8-27B-FP8",
"ANTHROPIC_DEFAULT_HAIKU_MODEL_NAME": "Qwen3.8-27B-FP8",
"ANTHROPIC_DEFAULT_OPUS_MODEL": "Qwen3.8-27B-FP8",
"ANTHROPIC_DEFAULT_OPUS_MODEL_NAME": "Qwen3.8-27B-FP8",
"ANTHROPIC_DEFAULT_SONNET_MODEL": "Qwen3.8-27B-FP8",
"ANTHROPIC_DEFAULT_SONNET_MODEL_NAME": "Qwen3.8-27B-FP8",
"API_TIMEOUT_MS": "3000000",
"CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1"
},
"model": "opus"
}
使用 Qwen3.8-27B-DFlash2 投机解码草稿模型
下载模型
modelscope download --model z-lab/Qwen3.8-27B-DFlash2 --local_dir ./Qwen3.8-27B-DFlash2
安装 vllm
conda create -n qwen38 python=3.11
conda activate qwen38
git clone https://github.com/vllm-project/vllm.git ./vllm-pr-52816
git fetch origin pull/52816/head:pr-52816
git checkout pr-52816
pip install -e . -i https://mirrors.aliyun.com/pypi/simple/
CUDA_VISIBLE_DEVICES=0,1 vllm serve \
/data/models/Qwen3.8-27B-FP8/ \
--host 0.0.0.0 \
--port 10304 \
--tensor-parallel-size 2 \
--gpu-memory-utilization 0.8 \
--max-model-len 32768 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--served-model-name Qwen3.8-27B-FP8 \
--speculative-config '{"method":"dflash","model":"/data/models/Qwen3.8-27B-DFlash2/","num_speculative_tokens":7}' \
--api-key xxx