主题

srebro2026-09-04

Qwen2.5-VL-7B-Instruct-AWQ 部署指南 ​

本文介绍两种部署方式：二进制（vLLM）与 Docker Compose。

一、二进制版本 ​

1.1 系统与 NVIDIA 驱动 ​

系统选择 Ubuntu 22.04.5 LTS。先做系统更新并安装 NVIDIA 驱动，重启后确认显卡信息：

bash
apt update -y; apt upgrade -y
apt install --no-install-recommends nvidia-headless-535-server nvidia-utils-535-server -y
reboot
nvidia-smi

NOTE

若有板载核显导致 nvidia-smi 不可用：编辑 /etc/default/grub，在 GRUB_CMDLINE_LINUX_DEFAULT="" 内添加 pci=realloc=off，执行 update-grub 后重启。

1.2 安装 CUDA ​

WARNING

已安装 NVIDIA 驱动时，安装 CUDA 不要再勾选/安装驱动。

bash
wget https://developer.download.nvidia.com/compute/cuda/12.4.1/local_installers/cuda_12.4.1_550.54.15_linux.run
sudo sh cuda_12.1.1_530.30.02_linux.run \
  --toolkit \
  --silent \
  --override

导入环境变量并验证：

bash
echo 'export PATH=/usr/local/cuda-12.1/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.1/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

# 验证
nvcc --version

1.3 安装指定版本 Python ​

bash
sudo apt install -y python3.11 python3.11-venv python3.11-dev

# 创建独立环境
python3.11 -m venv ~/vllm-env
source ~/vllm-env/bin/activate
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
pip install --upgrade pip

1.4 安装 Torch ​

NOTE

须在 venv 中安装。

bash
# 指定 CUDA 12.1 版本的 wheel
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 \
  --index-url https://download.pytorch.org/whl/cu121

# 验证
python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"

1.5 安装 vLLM ​

WARNING

transformers 务必限制版本 <5。

bash
pip install vllm==0.7.3 "transformers<5.0.0"

# 验证
python -c "import vllm; print(vllm.__version__)"
python -c "import torch; import vllm; print(f'CUDA available: {torch.cuda.is_available()}'); print(f'vLLM version: {vllm.__version__}')"

1.6 下载模型 ​

NOTE

确认已安装 git-lfs：apt install git-lfs。

bash
mkdir -p /home/application/vllm/models
git clone https://www.modelscope.cn/Qwen/Qwen2.5-VL-7B-Instruct-AWQ.git /home/application/models/Qwen2.5-VL-7B-Instruct-AWQ

1.7 启动服务 ​

bash
nohup python -m vllm.entrypoints.openai.api_server \
    --model /home/application/models/Qwen2.5-VL-7B-Instruct-AWQ \
    --quantization awq \
    --dtype float16 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.9 \
    --served-model-name Qwen2.5-VL-7B-Instruct-AWQ \
    --trust-remote-code \
    --max-num-seqs 1 \
    --limit-mm-per-prompt "image=1" \
    > /var/log/vllm.log 2>&1 &

二、Docker 版本 ​

2.1 安装 Docker 与 Docker Compose ​

先安装好 Docker 与 Docker Compose，过程参考：

2.2 安装 NVIDIA 驱动 ​

bash
apt install --no-install-recommends nvidia-headless-535-server nvidia-utils-535-server
reboot
nvidia-smi

2.3 安装 nvidia-container-toolkit ​

bash
sudo apt-get update && sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

apt update

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.19.0-1
sudo apt-get install -y \
    nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

sudo nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

# 验证
docker info | grep -i runtime

预期输出示例：

bash
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc

2.4 下载模型 ​

NOTE

确认已安装 git-lfs：apt install git-lfs。

bash
mkdir -p /home/application/vllm/models
git clone https://www.modelscope.cn/Qwen/Qwen2.5-VL-7B-Instruct-AWQ.git /home/application/vllm/models/Qwen2.5-VL-7B-Instruct-AWQ

2.5 使用 Docker Compose 启动 ​

bash
docker pull swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/vllm/vllm-openai:v0.7.3

cat << 'EOF' | sudo tee /home/application/vllm/docker-compose.yml > /dev/null
services:
  vllm-qwen25-vl-7b:
    container_name: vllm-qwen25-vl-7b
    image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/vllm/vllm-openai:v0.7.3
    restart: unless-stopped
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=0        # 改为实际 GPU 编号
      - VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
      - TZ=Asia/Shanghai
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - ./models:/models
    ports:
      - "8000:8000"
    ipc: host
    command: >
      --model /models/Qwen2.5-VL-7B-Instruct-AWQ
      --served-model-name Qwen2.5-VL-7B-Instruct-AWQ
      --quantization awq
      --dtype float16
      --max-model-len 4096
      --gpu-memory-utilization 0.9
      --trust-remote-code
      --max-num-seqs 1
      --limit-mm-per-prompt image=1
      --disable-log-requests
EOF

docker-compose -f /home/application/vllm/docker-compose.yml up -d

最近更新