VLLm: Unterschied zwischen den Versionen
Zur Navigation springen
Zur Suche springen
Die Seite wurde neu angelegt: „=== Beschreibung === === Download === Normal (ROCm) <syntaxhighlight lang="bash" line="1"> docker pull rocm/vllm-dev:nightly </syntaxhighlight> gfx906 <syntaxhighlight lang="bash" line="1"> docker pull nalanzeyu/vllm-gfx906 </syntaxhighlight> === Ausführen === <syntaxhighlight lang="bash" line="1"> docker run -it --rm --shm-size=8g --device=/dev/kfd --device=/dev/dri \ --group-add video -p 8086:8000 \ -v /mnt/share/models:/models \ nalanzey…“ |
|||
| Zeile 17: | Zeile 17: | ||
-v /mnt/share/models:/models \ | -v /mnt/share/models:/models \ | ||
nalanzeyu/vllm-gfx906 \ | nalanzeyu/vllm-gfx906 \ | ||
vllm serve /models/Qwen3-Coder-30B-A3B-Instruct-AWQ-4bit --max-model-len 30000 --enable-auto-tool-choice --tool-call-parser hermes | vllm serve /models/Qwen3-Coder-30B-A3B-Instruct-AWQ-4bit --served-model-name Homelab --max-model-len 30000 --enable-auto-tool-choice --tool-call-parser hermes | ||
</syntaxhighlight> | </syntaxhighlight> | ||
=== Test === | === Test === | ||
=== Bekannte Probleme === | === Bekannte Probleme === | ||
Aktuelle Version vom 13. November 2025, 01:16 Uhr
Beschreibung
Download
Normal (ROCm)
docker pull rocm/vllm-dev:nightlygfx906
docker pull nalanzeyu/vllm-gfx906Ausführen
docker run -it --rm --shm-size=8g --device=/dev/kfd --device=/dev/dri \
--group-add video -p 8086:8000 \
-v /mnt/share/models:/models \
nalanzeyu/vllm-gfx906 \
vllm serve /models/Qwen3-Coder-30B-A3B-Instruct-AWQ-4bit --served-model-name Homelab --max-model-len 30000 --enable-auto-tool-choice --tool-call-parser hermes