vLLM: Efficient Serving with Scalable Performance

A guide to serving multimodal models like LLaVA on a CPU with vLLM

6 min readSep 14, 2024

Co-Author:

Introduction:

vLLM is an open-source library that revolutionizes Large Language Model (LLM) inference and serving. It addresses the challenges of efficient LLM deployment and scaling, making it possible to run these models on a variety of hardware configurations, including CPUs. vLLM introduces innovative techniques like PagedAttention and continuous batching to optimize memory usage and enhance throughput. With its OpenAI-compatible API and seamless integration with the Hugging Face ecosystem, vLLM offers a flexible and powerful solution for AI practitioners and researchers. Whether you’re working with LLMs or multimodal systems like LLaVA, vLLM provides the tools to push the boundaries of what’s possible in AI inference.

Key Features:

PagedAttention: This is vLLM’s core innovation. It manages attention keys and values like an operating system manages memory pages, significantly reducing memory usage and enhancing throughput.
Continuous Batching: vLLM can dynamically batch requests, even those arriving at different times, to maximize GPU utilization.
Quantization: Supports various quantization methods including GPTQ, AWQ, INT4, INT8, and FP8 to reduce model size and increase inference speed
Optimized Execution: Utilizes CUDA/HIP graph for fast model execution and includes optimized CUDA kernels with FlashAttention and FlashInfer integration.
Tensor Parallelism: Allows distribution of model layers across multiple GPUs for handling larger models and distributed inference.
OpenAI-compatible API: Provides an API that’s compatible with OpenAI’s, making it easy to integrate into existing systems.
Hugging Face Integration: Seamlessly works with models from the Hugging Face ecosystem.

Advantages of vLLM:

High Performance: Achieves state-of-the-art serving throughput.
Memory Efficiency: Optimizes memory usage, allowing larger models to fit on given hardware.
Flexibility: Supports various model architectures and deployment scenarios.
Easy Integration: OpenAI-compatible API makes it straightforward to use in existing applications.
Open Source: Allows for community contributions and customizations.

Setting Up LLaVA for CPU Inference with vLLM:

LLaVA (Large Language and Vision Assistant) is a multimodal model that combines text and image understanding. While it is typically run on GPUs due to its complexity, vLLM enables us to leverage its capabilities on a CPU. This can be particularly useful for development or in environments where GPU access is limited or unavailable.

In this article, we’ll specifically focus on the llava-hf/llava-1.5-7b-hf model. This model is relatively compact compared to other large-scale multimodal models, making it well-suited for CPU-based inference. Its balanced performance and resource requirements make it an excellent candidate for running on CPUs, allowing us to harness its text and image processing capabilities without needing a GPU.

Now, let’s explore the specific steps for serving LLaVA on a CPU using vLLM:

Step 1: Download and Extract vLLM

First, we need to download and extract the vLLM package. Run the following commands:

wget https://github.com/vllm-project/vllm/archive/refs/tags/v0.5.3.tar.gz
tar -xzvf v0.5.3.tar.gz
rm v0.5.3.tar.gz  
cd vllm-0.5.3

Step 2: Prepare the Dockerfile

Create a new Dockerfile with the following content:

vi Dockerfile.custom

# This vLLM Dockerfile is used to construct an image that can build and run vLLM on x86 CPU platform  with data types FP32 and BF16.
FROM ubuntu:22.04 AS cpu-test-1

RUN apt-get update -y \
    && apt-get install -y git wget vim numactl gcc-12 g++-12 python3 python3-pip libtcmalloc-minimal4 \
    && update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 10 --slave /usr/bin/g++ g++ /usr/bin/g++-12

# intel-openmp provides additional performance improvement vs. openmp
# tcmalloc provides better memory allocation efficiency, e.g., holding memory in caches to speed up access of commonly-used objects.
RUN pip install intel-openmp

ENV LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libtcmalloc_minimal.so.4:/usr/local/lib/libiomp5.so:$LD_PRELOAD"

RUN pip install https://intel-extension-for-pytorch.s3.amazonaws.com/ipex_dev/cpu/intel_extension_for_pytorch-2.3.100%2Bgit0eb3473-cp310-cp310-linux_x86_64.whl

RUN pip install --upgrade pip \
    && pip install wheel packaging ninja "setuptools>=49.4.0" numpy

FROM cpu-test-1 AS build

COPY ./ /workspace/vllm

WORKDIR /workspace/vllm

RUN pip install -v -r requirements-cpu.txt --extra-index-url https://download.pytorch.org/whl/cpu

# Support for building with non-AVX512 vLLM: docker build --build-arg VLLM_CPU_DISABLE_AVX512="true" ...
ARG VLLM_CPU_DISABLE_AVX512="false"
ENV VLLM_CPU_DISABLE_AVX512=${VLLM_CPU_DISABLE_AVX512}

# Set environment variables for build
ENV VLLM_CPU_KVCACHE_SPACE=40 
ENV VLLM_IMAGE_FETCH_TIMEOUT=300
ENV VLLM_ENGINE_ITERATION_TIMEOUT_S=300

RUN VLLM_TARGET_DEVICE=cpu python3 setup.py install

WORKDIR /workspace/vllm

# Expose port 8000
EXPOSE 8000

# By default, `vllm serve` command with placeholder model and parameters
ENTRYPOINT ["vllm"]
CMD ["serve", "llava-hf/llava-1.5-7b-hf", "--chat-template", "./examples/template_llava.jinja"]

This Dockerfile is used to construct an image that can build and run vLLM on x86 CPU platform with data types FP32 and BF16.

After creating the Dockerfile, rename it:

mv Dockerfile Dockerfile.temp
mv Dockerfile.custom Dockerfile

Step 3: Build the Image

Run the following command to build the Image via Podman:

podman build -t llava-vllm-cpu .

Step 4: Run the Container

Start the container with the following command:

podman run --rm --ipc=host -p 8000:8000 llava-vllm-cpu

Step 5: Test using curl command

Once the server is running, we can query the LLaVA model using curl:

curl -X POST "http://localhost:8000/v1/chat/completions" \
-H "Content-Type: application/json" \
-d '{
  "model": "llava-hf/llava-1.5-7b-hf",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What is the content of this image?"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://easy-peasy.ai/cdn-cgi/image/quality=80,format=auto,width=700/https://fdczvxmwwjwpwbeeqcth.supabase.co/storage/v1/object/public/images/966117e6-2158-4095-8416-b31e9df72aba/ce4319d0-5e00-40da-b1f8-67db09a009f3.png"
          }
        }
      ]
    }
  ]
}'

OpenAI-Compatible Server:

vLLM also provides an HTTP server that implements OpenAI’s Completions and Chat API. This allows for easy integration with existing systems and workflows. For vision-language models like LLaVA, vLLM’s HTTP server is compatible with the OpenAI Vision API. Here’s an example of how to launch the llava-hf/llava-1.5-7b-hf model with vLLM’s OpenAI-compatible API server:

Import Required Modules

import io
import base64
import requests
from PIL import Image
import matplotlib.pyplot as plt
from openai import OpenAI as OpenAI_vLLM

Initialize vLLM API server

openai_api_key = "sample"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI_vLLM(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

Visualize the Image

image_path =  "demo_image.jpg"
img = plt.imread(image_path)
plt.imshow(img)
plt.axis('off')  
plt.show()

Define the Prompt and the Image URL

image_url = "https://easy-peasy.ai/cdn-cgi/image/quality=80,format=auto,width=700/https://fdczvxmwwjwpwbeeqcth.supabase.co/storage/v1/object/public/images/966117e6-2158-4095-8416-b31e9df72aba/ce4319d0-5e00-40da-b1f8-67db09a009f3.png"

response = client.chat.completions.create(
    model="llava-hf/llava-1.5-7b-hf",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What is the content of given Image?"},
            {
                "type": "image_url",
                "image_url": {
                    "url": image_url,
                },
            },
        ],
    }],
    stream=False
)

print(response.choices[0].message.content.strip())

Encode an Image retrieved from a remote url to base64 format

We can also use the base64 encoded image with our prompt instructions. Keep in mind that the size of the encoded image can affect the answer output quality as well as the processing performance. For optimal results, ensure the image is within supported size limits and formats.

# Use base64 encoded image 
def encode_image_base64_from_url(image_url: str) -> str:
    """Encode an image retrieved from a remote url to base64 format."""
    with requests.get(image_url) as response:
        response.raise_for_status()
        result = base64.b64encode(response.content).decode('utf-8')
    return result


def resize_base64_image(base64_string, size=(100, 100)):
    """
    Resize an image encoded as a Base64 string.

    Args:
    base64_string (str): Base64 string of the original image.
    size (tuple): Desired size of the image as (width, height).

    Returns:
    str: Base64 string of the resized image.
    """
    # Decode the Base64 string
    img_data = base64.b64decode(base64_string)
    img = Image.open(io.BytesIO(img_data))
    # Resize the image
    resized_img = img.resize(size, Image.LANCZOS)
    # Save the resized image to a bytes buffer
    buffered = io.BytesIO()
    resized_img.save(buffered, format=img.format)
    # Encode the resized image to Base64
    return base64.b64encode(buffered.getvalue()).decode("utf-8")


base64_image = encode_image_base64_from_url(image_url)
base64_image = resize_base64_image(base64_image)

Use base64 encoded Image with Prompt Instructions

chat_response = client.chat.completions.create(
    model="llava-hf/llava-1.5-7b-hf",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is the content of given Image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{base64_image}"
                    },
                },
            ],
        },
    ],
    max_tokens=1000,
    stream=True
)

for chunk in chat_response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

This setup ensures that you can utilize LLaVA’s multimodal capabilities through a familiar API interface, simplifying integration into our projects or workflows.

Conclusion:

vLLM provides an efficient way to run large language models, including multimodal models like LLaVA, on various hardware configurations. By leveraging vLLM’s optimizations and Docker for easy deployment, we can run these sophisticated models even on CPU environments, making AI more accessible for development and testing purposes. While CPU inference may not match the speed of GPU-based solutions, it offers a viable option for scenarios where GPU resources are limited or unavailable.

Looking ahead, vLLM’s development will continue to introduce new features and optimizations, further democratizing access to powerful Generative AI models.