Deploy Ollama and Open-WebUI on OpenShift

6 min readDec 3, 2024

In the rapidly evolving world of Generative AI, Large Language Models (LLMs) and Vision Language Models (VLMs) have become indispensable tools for a variety of applications. However, deploying and interacting with these models can be daunting, especially for those less comfortable with command-line interfaces. This is where Ollama and Open WebUI shine.

Ollama offers a wide range of high-performing open-source models like Llama3, Granite, CodeLlama, LLaVa and Mistral, which can be easily accessed and tested. When paired with Open WebUI, a user-friendly and extensible interface, it becomes easier to interact with these models, even in offline or self-hosted environments, while maintaining control over data and ensuring a secure, customizable environment.

Key Features of Open WebUI

Effortless Setup: Simplify the installation process with Docker or Kubernetes, providing a smooth experience with support for both :ollama and :cuda tagged images.
Ollama/OpenAI API Integration: Easily integrate OpenAI-compatible APIs alongside Ollama models, and customize the API URL to link with other platforms like LMStudio, GroqCloud, Mistral, and more.
Granular Permissions and User Groups: Enhance security by allowing administrators to define user roles and permissions, ensuring a secure and personalized experience for all users.
Local RAG Integration: Leverage the power of Retrieval Augmented Generation (RAG) for enhanced interactions, integrating documents and web search results directly into the chat.
Image Generation Integration: Incorporate dynamic visual content into conversations with seamless integration of image generation capabilities like AUTOMATIC1111, ComfyUI, or OpenAI’s DALL-E.

Steps to Deploy Ollama and Open WebUI

Let’s dive into how you can deploy Ollama and Open WebUI on RedHat OpenShift. Before starting, ensure you have:

Access to an OpenShift cluster with admin privileges (non-production).
OpenShift CLI (oc) installed locally.

If you have a GPU-enabled cluster, Ollama models will run more efficiently. However, CPU-only clusters work reasonably well for smaller models.

1. Log in to OpenShift Console

Open the OpenShift Console and log in using cluster admin credentials. Run the following command in your terminal to authenticate.

oc login --token=<YOUR_TOKEN> --server=<YOUR_SERVER>

2. Create a new Project

Create a dedicated project for Ollama and Open WebUI:

oc new-project open-webui

3. Set Up Persistent Storage

Create a file named pvc.yaml with the following content to define the Persistent Volume Claims for Ollama and Open WebUI:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: webui-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

This YAML file defines two PVCs:

ollama-data for Ollama storage with 30Gi.
webui-data for Open WebUI storage with 30Gi.

Once the pvc.yaml file is created, run the following command to apply the PVC configuration in your OpenShift environment:

oc apply -f pvc.yaml

To verify that the PVCs have been created successfully, you can use the following command:

oc get pvc

This will show the status of your PVCs, and you should see both ollama-data and webui-data listed with their respective statuses. Once applied, these PVCs will be available for use by your pods or deployments to store data persistently.

4. Deploy Ollama

Create a file named ollama-deployment.yaml with the following content to define the Ollama deployment and service in your OpenShift cluster:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports:
        - containerPort: 11434
        volumeMounts:
        - name: ollama-data
          mountPath: /.ollama 
        tty: true
      volumes:
      - name: ollama-data
        persistentVolumeClaim:
          claimName: ollama-data
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
spec:
  ports:
  - protocol: TCP
    port: 11434
    targetPort: 11434
  selector:
    app: ollama

This file includes:

Deployment Configuration: Deploys the Ollama container using the ollama/ollama:latest image and mounts the ollama-data PVC at the path /.ollama.
Service Configuration: Exposes the Ollama service on port 11434 using TCP.

Once the ollama-deployment.yaml file is created, deploy Ollama by running the following command:

oc apply -f ollama-deployment.yaml

To verify the status of your deployment and services, use the following commands:

Check the status of Pods:

oc get pods

This command will show the status of the pods running in your cluster. Look for a pod with the name ollama and ensure its status is Running.

Check the status of Services:

oc get services

This command will display the services running in your OpenShift cluster. You should see the ollama service listed with port 11434 exposed.

5. Port Forward to access Ollama locally

Set up port forwarding to access the Ollama service locally. This allows you to interact with the service running in OpenShift from your local machine:

oc port-forward svc/ollama 11434:11434

Verify that Ollama is running by sending a test request to localhost:11434:

curl localhost:11434

This will confirm that Ollama is accessible on your local machine and functioning as expected.

6. Pull the Models

Open an interactive bash terminal inside the specified pod to execute commands related to model management. Replace <pod-name> with the actual name of the Ollama pod (e.g., ollama-95cb23b13d-qwerty):

oc exec -ti <pod-name> -- bash

Pull LLMs (Large Language Models) like granite-code:3b, mistral:7b and VLMs (Vision Language Models) like llama3.2-vision:11b. Execute the following commands to pull the models:

ollama pull llama3.2:3b
ollama pull granite-code:3b
ollama pull mistral:7b

ollama pull llava:7b
ollama pull llama3.2-vision:11b

List all the models you’ve pulled to ensure they’re available:

ollama list

This will ensure that you have successfully pulled the models and they are ready to be used.

7. Deploy Open WebUI

Create the deployment configuration file — openwebui-deployment.yaml for Open WebUI.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: open-webui
spec:
  replicas: 1
  selector:
    matchLabels:
      app: open-webui
  template:
    metadata:
      labels:
        app: open-webui
    spec:
      containers:
      - name: open-webui
        image: ghcr.io/open-webui/open-webui:main
        ports:
        - containerPort: 8080
        env:
        - name: OLLAMA_BASE_URL
          value: "http://ollama:11434"
        - name: WEBUI_SECRET_KEY
          value: "your-secret-key"            
        volumeMounts:
        - name: webui-data
          mountPath: /app/backend/data
      volumes:
      - name: webui-data
        persistentVolumeClaim:
          claimName: webui-data
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: open-webui
spec:
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 8080
  selector:
    app: open-webui

Apply the deployment configuration to deploy Open WebUI within your OpenShift cluster:

oc apply -f openwebui-deployment.yaml

This step will deploy the Open WebUI, setting it up to interact with your models deployed in Ollama.

8. Setup external access for Open WebUI

Create an OpenShift route to expose the WebUI service so it can be accessed externally:

oc create route edge --service open-webui

Retrieve the route URL to access the Open WebUI from your browser:

oc get routes.route.openshift.io open-webui -o json | jq -r '.spec.host' | sed 's/^/https:\/\//'

Access the Open WebUI in your browser by using the retrieved URL, which will allow you to interact with your deployed models via a graphical interface.

When you first access the WebUI, you’ll be directed to a login page. Simply sign up with any credentials to begin. The first user to sign up will be granted administrator privileges, giving them full control over the WebUI, including managing other users and configuring settings.

Now, let’s perform inference by selecting a model and querying it through the Open WebUI.

Conclusion

This deployment guide highlights the simplicity of integrating advanced LLM and VLM capabilities into your OpenShift environment. With Ollama for managing models and Open WebUI for user interaction, you’ll have a robust setup ready for AI-powered applications.

With Ollama for model management and Open WebUI for user interaction, you gain a secure, flexible, and efficient setup tailored for AI-powered applications.

So why wait? Spin up your own instance and unlock the power of LLMs today! 🚀