VLLm - Versionsgeschichte

Hendrik am 21. März 2026 um 00:40 Uhr

2026-03-21T00:40:24Z

@@ Zeile 13: / Zeile 13: @@
 === Ausführen ===
-Variante 1:<syntaxhighlight lang="bash" line="1">
+==== Variante 1 ====
 docker run -it --rm --shm-size=8g --device=/dev/kfd --device=/dev/dri \
      --group-add video -p 8086:8000 \
@@ Zeile 22: / Zeile 23: @@
 sudo docker run -it --rm --network=host --group-add=video --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --device /dev/kfd --device /dev/dri -v /home/hendrik/.lmstudio/models/:/app/models -e HF_HOME="/app/models" -e HF_TOKEN="<TOKEN>" -e NCCL_P2P_DISABLE=1 -e VLLM_CUSTOM_OPS=all -e VLLM_ROCM_USE_AITER=0 -e SAFETENSORS_FAST_GPU=1 -e PYTORCH_TUNABLEOP_ENABLED=1 rocm/vllm-dev:nightly
-</syntaxhighlight>Für gfx1201:<syntaxhighlight lang="bash">
+</syntaxhighlight>
 sudo docker run -it --rm --network=host \
 --group-add=video --ipc=host --cap-add=SYS_PTRACE \
@@ Zeile 48: / Zeile 50: @@
 vllm bench serve --num-prompts 1 --dataset-name=random --input-len 512 --output-len 128 --model Qwen/Qwen3-4B-Instruct-2507-FP8
 </syntaxhighlight>
 === Test ===
 === Bekannte Probleme ===

Hendrik: /* Ausführen */

2026-02-09T20:02:15Z

Ausführen

← Nächstältere Version		Version vom 9. Februar 2026, 22:02 Uhr
Zeile 20:		Zeile 20:
	vllm serve /models/Qwen3-Coder-30B-A3B-Instruct-AWQ-4bit --served-model-name Homelab --max-model-len 30000 --enable-auto-tool-choice --tool-call-parser hermes		vllm serve /models/Qwen3-Coder-30B-A3B-Instruct-AWQ-4bit --served-model-name Homelab --max-model-len 30000 --enable-auto-tool-choice --tool-call-parser hermes
	</syntaxhighlight>Variante 2, getestet 18.12.2025:<syntaxhighlight lang="bash">		</syntaxhighlight>Variante 2, getestet 18.12.2025:<syntaxhighlight lang="bash">
	sudo docker run -it --rm --network=host \		sudo docker run -it --rm --network=host --group-add=video --ipc=host --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --device /dev/kfd --device /dev/dri -v /home/hendrik/.lmstudio/models/:/app/models -e HF_HOME="/app/models" -e HF_TOKEN="<TOKEN>" -e NCCL_P2P_DISABLE=1 -e VLLM_CUSTOM_OPS=all -e VLLM_ROCM_USE_AITER=0 -e SAFETENSORS_FAST_GPU=1 -e PYTORCH_TUNABLEOP_ENABLED=1 rocm/vllm-dev:nightly
	--group-add=video --ipc=host --cap-add=SYS_PTRACE \
	--security-opt seccomp=unconfined --device /dev/kfd \
	--device /dev/dri \
	-v /home/hendrik/.lmstudio/models/:/app/models \
	-e HF_HOME="/app/models" \
	-e HF_TOKEN="<TOKEN>" \
	-e NCCL_P2P_DISABLE=1 \
	-e VLLM_CUSTOM_OPS=all \
	-e VLLM_ROCM_USE_AITER=0 \
	-e SAFETENSORS_FAST_GPU=1 \
	-e PYTORCH_TUNABLEOP_ENABLED=1
	rocm/vllm-dev:nightly
	</syntaxhighlight>Für gfx1201:<syntaxhighlight lang="bash">		</syntaxhighlight>Für gfx1201:<syntaxhighlight lang="bash">
	sudo docker run -it --rm --network=host \		sudo docker run -it --rm --network=host \

Hendrik: /* Download */

2025-12-18T22:06:11Z

Download

← Nächstältere Version		Version vom 19. Dezember 2025, 00:06 Uhr
Zeile 57:		Zeile 57:
	Benchmark:		Benchmark:
	<syntaxhighlight lang="bash">		<syntaxhighlight lang="bash">
	vllm bench serve --dataset-name=random --model Qwen/Qwen3-4B-Instruct-2507-FP8		vllm bench serve --num-prompts 1 --dataset-name=random --input-len 512 --output-len 128 --model Qwen/Qwen3-4B-Instruct-2507-FP8
	</syntaxhighlight>		</syntaxhighlight>

Hendrik: /* Ausführen */

2025-12-18T22:02:58Z

Ausführen

← Nächstältere Version		Version vom 19. Dezember 2025, 00:02 Uhr
Zeile 54:		Zeile 54:
	</syntaxhighlight>Mit:<syntaxhighlight lang="bash">		</syntaxhighlight>Mit:<syntaxhighlight lang="bash">
	vllm serve Qwen/Qwen3-VL-8B-Thinking --served-model-name Homelab --tp 2 --max_model_len 4096 --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3		vllm serve Qwen/Qwen3-VL-8B-Thinking --served-model-name Homelab --tp 2 --max_model_len 4096 --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3
	</syntaxhighlight>Benchmark:		</syntaxhighlight>
			Benchmark:
			<syntaxhighlight lang="bash">
			vllm bench serve --dataset-name=random --model Qwen/Qwen3-4B-Instruct-2507-FP8
			</syntaxhighlight>

	=== Test ===		=== Test ===

Hendrik: /* Ausführen */

2025-12-18T20:12:30Z

Ausführen

@@ Zeile 33: / Zeile 33: @@
 -e PYTORCH_TUNABLEOP_ENABLED=1
 rocm/vllm-dev:nightly
-</syntaxhighlight>Ohne Tensor Parallism:<syntaxhighlight lang="bash">
+</syntaxhighlight>Für gfx1201:<syntaxhighlight lang="bash">
 vllm serve Qwen/Qwen3-VL-8B-Thinking --served-model-name Homelab --max_model_len 4096 --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3
 </syntaxhighlight>Mit:<syntaxhighlight lang="bash">

Hendrik: /* Ausführen */

2025-12-18T19:47:20Z

Ausführen

← Nächstältere Version		Version vom 18. Dezember 2025, 21:47 Uhr
Zeile 34:		Zeile 34:
	rocm/vllm-dev:nightly		rocm/vllm-dev:nightly
	</syntaxhighlight>Ohne Tensor Parallism:<syntaxhighlight lang="bash">		</syntaxhighlight>Ohne Tensor Parallism:<syntaxhighlight lang="bash">
	vllm serve Qwen/Qwen3-VL-8B-Thinking --max_model_len 4096 --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3		vllm serve Qwen/Qwen3-VL-8B-Thinking --served-model-name Homelab --max_model_len 4096 --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3
	</syntaxhighlight>Mit:<syntaxhighlight lang="bash">		</syntaxhighlight>Mit:<syntaxhighlight lang="bash">
	vllm serve Qwen/Qwen3-VL-8B-Thinking --tp 2 --max_model_len 4096 --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3		vllm serve Qwen/Qwen3-VL-8B-Thinking --served-model-name Homelab --tp 2 --max_model_len 4096 --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3
	</syntaxhighlight>Benchmark:		</syntaxhighlight>Benchmark:

Hendrik: /* Ausführen */

2025-12-18T18:51:43Z

Ausführen

← Nächstältere Version		Version vom 18. Dezember 2025, 20:51 Uhr
Zeile 37:		Zeile 37:
	</syntaxhighlight>Mit:<syntaxhighlight lang="bash">		</syntaxhighlight>Mit:<syntaxhighlight lang="bash">
	vllm serve Qwen/Qwen3-VL-8B-Thinking --tp 2 --max_model_len 4096 --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3		vllm serve Qwen/Qwen3-VL-8B-Thinking --tp 2 --max_model_len 4096 --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3
	</syntaxhighlight>		</syntaxhighlight>Benchmark:

	=== Test ===		=== Test ===

Hendrik am 18. Dezember 2025 um 18:45 Uhr

2025-12-18T18:45:34Z

@@ Zeile 1: / Zeile 1: @@
 === Beschreibung ===
 === Download ===
@@ Zeile 12: / Zeile 13: @@
 === Ausführen ===
-<syntaxhighlight lang="bash" line="1">
+Variante 1:<syntaxhighlight lang="bash" line="1">
 docker run -it --rm --shm-size=8g --device=/dev/kfd --device=/dev/dri \
      --group-add video -p 8086:8000 \
@@ Zeile 18: / Zeile 19: @@
      nalanzeyu/vllm-gfx906 \
      vllm serve /models/Qwen3-Coder-30B-A3B-Instruct-AWQ-4bit --served-model-name Homelab --max-model-len 30000 --enable-auto-tool-choice --tool-call-parser hermes
 </syntaxhighlight>

Hendrik: /* Ausführen */

2025-11-13T00:16:48Z

Ausführen

← Nächstältere Version		Version vom 13. November 2025, 02:16 Uhr
Zeile 17:		Zeile 17:
	-v /mnt/share/models:/models \		-v /mnt/share/models:/models \
	nalanzeyu/vllm-gfx906 \		nalanzeyu/vllm-gfx906 \
	vllm serve /models/Qwen3-Coder-30B-A3B-Instruct-AWQ-4bit --max-model-len 30000 --enable-auto-tool-choice --tool-call-parser hermes ~~--served-model-name Homelab~~		vllm serve /models/Qwen3-Coder-30B-A3B-Instruct-AWQ-4bit --served-model-name Homelab --max-model-len 30000 --enable-auto-tool-choice --tool-call-parser hermes
	</syntaxhighlight>		</syntaxhighlight>

	=== Test ===		=== Test ===
	=== Bekannte Probleme ===		=== Bekannte Probleme ===

Hendrik: Die Seite wurde neu angelegt: „=== Beschreibung === === Download === Normal (ROCm) docker pull rocm/vllm-dev:nightly gfx906 docker pull nalanzeyu/vllm-gfx906 === Ausführen === docker run -it --rm --shm-size=8g --device=/dev/kfd --device=/dev/dri \ --group-add video -p 8086:8000 \ -v /mnt/share/models:/models \ nalanzey…“

2025-11-12T23:32:17Z

Die Seite wurde neu angelegt: „=== Beschreibung === === Download === Normal (ROCm) <syntaxhighlight lang="bash" line="1"> docker pull rocm/vllm-dev:nightly </syntaxhighlight> gfx906 <syntaxhighlight lang="bash" line="1"> docker pull nalanzeyu/vllm-gfx906 </syntaxhighlight> === Ausführen === <syntaxhighlight lang="bash" line="1"> docker run -it --rm --shm-size=8g --device=/dev/kfd --device=/dev/dri \ --group-add video -p 8086:8000 \ -v /mnt/share/models:/models \ nalanzey…“

Neue Seite

=== Beschreibung ===

=== Download ===
Normal (ROCm)
<syntaxhighlight lang="bash" line="1">
docker pull rocm/vllm-dev:nightly
</syntaxhighlight>
gfx906
<syntaxhighlight lang="bash" line="1">
docker pull nalanzeyu/vllm-gfx906
</syntaxhighlight>

=== Ausführen ===
<syntaxhighlight lang="bash" line="1">
docker run -it --rm --shm-size=8g --device=/dev/kfd --device=/dev/dri \
--group-add video -p 8086:8000 \
-v /mnt/share/models:/models \
nalanzeyu/vllm-gfx906 \
vllm serve /models/Qwen3-Coder-30B-A3B-Instruct-AWQ-4bit --max-model-len 30000 --enable-auto-tool-choice --tool-call-parser hermes --served-model-name Homelab
</syntaxhighlight>
=== Test ===
=== Bekannte Probleme ===
=== Nützliche Links ===
* [https://github.com/ggml-org/llama.cpp Offizielles llama.cpp Repository]
* [https://github.com/ggml-org/llama.cpp/blob/master/examples/server/README.md llama-server Dokumentation]
* [https://github.com/ggml-org/llama.cpp/discussions llama.cpp Discussions]
* [https://huggingface.co/models?library=gguf GGUF Modelle auf Hugging Face]
* [https://docs.rocm.com/ ROCm Dokumentation]
* [https://vulkan.lunarg.com/doc/sdk Vulkan SDK Dokumentation]