Skip to content

LLMs Runtime

To run LLMs on Dalek nodes, you can use the Open WebUI frontend on your machine locally and connect directly to a node running the backend of your choice (Ollama, vLLM, llama.cpp...) on one of Dalek's nodes.

Tip

For a daily usage, consider using LLoby as backend for your AI applications. Here are some key advantages in using LLoby:

  • Flexibility: OpenAI compatibility for supporting a wide range of AI tools.
  • Automated Resource Management: Node allocation is handled transparently and automatically, simplifying deployment.
  • Optimized Performance: Service is faster, especially when models are already loaded by other users.
  • Efficient Resource Sharing: Inference resources are shared across users, maximizing efficiency and minimizing computational waste.

Install and Run the OpenWeb-UI Frontend

From the quick-start page, there are multiple installation methods. We follow here the one for docker. Pull the image from the docker repositories:

docker pull ghcr.io/open-webui/open-webui:main

Run it on your local machine, making it available on localhost:3000: 

sudo docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway \
                -e WEBUI_AUTH=False \
                -v open-webui:/app/backend/data \
                --name open-webui ghcr.io/open-webui/open-webui:main

Info

The important options here are that we launch openwebui in single user mode (we don't want to handle multiple accounts on a local machine) and the --add-host=host.docker.internal:host-gateway option will redirect localhost of you main OS to host.docker.internal inside the docker container. This will simplify the connection to the node(s) later.

Ollama

Run Ollama on a Node

Multiple runtimes (also called backends) are available but Open WebUI handles out-of-the-box Ollama and makes getting models running easy.

You need to allocate exclusively the node to be able to connect with SSH on it. This can be done by connecting to the front then issuing either an srun or sbatch:

srun -w [NODE_NAME] --exclusive --interactive --pty bash

Then, on the node, start ollama by running: 

module load ollama
ollama serve

Info

This makes Ollama listen to web requests on the node, on port 11434.

Meeting the Ends

You then need to perform port forwarding to make your local and node 11434 ports communicate. On an allocated node, run the following: 

ssh -J front.dalek.lip6 -N -f [USER_DALEK]@[NODE_NAME] -L 0.0.0.0:11434:localhost:11434

Info

This commands connects to the node by first performing a jump on the front. It then performs port forwarding to bind your port 11434 with the node's localhost:11434. The binding is done on all your local interfaces. This command runs in the background without launching a shell so if no error are printed after issuing it, it is working properly.

[USER_DALEK] is your Dalek login and [NODE_NAME] is the name of the node you want to connect to (typically something like az4-n4090-2 or az4-a7900-0).

To check if it works, you can browse http://localhost:11434/ and it should display Ollama is running.

Last step is to navigate to Open WebUI Admin Settings (or localhost:3000 page, User > Settings > Administration Settings > Connections > Ollama API (or ) and make sure that the Ollama API connection is set to http://host.docker.internal:11434.

That's all folks!

Installation Notes

Models are installed and shared by all the users. The list of the available models is given in the Ollama Models section. So, by default, you cannot add or update these models (unless if you are in the ai-models group, which is not automatic, see the Technical Details about ai-models Group section for more information about it). If you need specific models or specific versions, you can do unset OLLAMA_MODELS after the module load ollama in order for Ollama to default to your home repository as the default model library. As the disk quota available on the NFS is limited, you need to set Ollama default models path to the scratch and download your models there.

For example, you can do the following to store the models on the scratch: 

module load ollama
mkdir -p /scratch/$USER/ollama/
OLLAMA_MODELS=/scratch/$USER/ollama/ ollama serve

Tips

Sometimes, large models can take a long time to load into memory. This is because NFS is not as fast as a local disk. If you need to reuse a specific model (or a subset of models) frequently, it may be a good idea to copy it to the scratch workspace.

Danger

Please keep in mind that models are heavy. When you store them on the NFS or the scratch, keep an eye on what you really use and clean occasionally.

Backend APIs

Since Ollama v15.0.0, Dalek provides multiple modules to help users to target specific backends:

  • ollama/x.y.z-cpu: execute only on CPU.
  • ollama/x.y.z-cuda: run with CUDA (Nvidia GPUs).
  • ollama/x.y.z-rocm: run with ROCm (AMD GPUs).
  • ollama/x.y.z-vulkan: run Vulkan (support many different GPUs).
  • ollama/x.y.z-zauto: automatic version, let Ollama decide how to run.

Even though different modules are used for a given version, the same Ollama executable binary is always used. The only difference lies in the definition of certain specific environment variables used to configure Ollama. For all the modules, the following environment variables are set:

  • OLLAMA_KEEP_ALIVE="-1": keep models in RAM and VRAM indefinitely as long as Ollama serves.
  • OLLAMA_HOST="0.0.0.0": serve any incoming IP addresses.
  • OLLAMA_MODELS="/mnt/data/ai-models/ollama/": as explained in the Installation Notes section, by default Ollama will search for preinstalled models in the /mnt/data/ai-models/ollama/ shared folder on the NFS.

The following subsections describe the environment variables depending on the proposed modules.

ollama/x.y.z-cpu

The following environment variable is set:

  • OLLAMA_LLM_LIBRARY="cpu": force Ollama to run models on CPU (and this to avoid GPU).

ollama/x.y.z-cuda

The following environment variable is set:

  • OLLAMA_LLM_LIBRARY="cuda_v13": force Ollama to run models with CUDA v13 (OLLAMA_LLM_LIBRARY="cuda_v12" is also possible)

ollama/x.y.z-rocm

The following environment variables are set:

  • OLLAMA_LLM_LIBRARY="rocm": force Ollama to run models with ROCm (\(\approx\) HIP).
  • HSA_OVERRIDE_GFX_VERSION=11.5.1: only on the az5-a890m partition. If not set, the Radeon 890M iGPU is not supported by ROCm and the models execute on CPU.

ollama/x.y.z-vulkan

The following environment variables are set:

  • OLLAMA_VULKAN="1": force Ollama to use Vulkan backend.
  • GGML_VK_VISIBLE_DEVICES="[x]": specify the GPU id to use. [x] is a placeholder and depending on the partition its default value is:
    • az4-n4090: 1 (select the GeForce RTX 4090 dGPU and NOT the Radeon 610M iGPU).
    • az4-a7900: 1 (select the Radeon RX 7900 XTX dGPU and NOT the Radeon 610M iGPU).
    • iml-ia770: 1 (select the Arc A770 eGPU and NOT the Arc Mobile iGPU)
    • az5-a890m: 0 (select the Radeon 890M iGPU).

Note

Users can override GGML_VK_VISIBLE_DEVICES environment variable to target the GPU they want.

Warning

Vulkan does not appear to work on the Intel GPUs of the iml-ia770 partition (Intel Arc A770 eGPU and Intel Arc Mobile iGPU).

ollama/0.9.3-ipex-llm-2.3

This is a specific version provided by Intel that combines with IPEX-LLM to run Ollama on Intel GPUs like the ones available in the iml-ia770 partition (Arc A770 eGPU and Arc Mobile iGPU). Before to load this module, it is required to source the OneAPI script as follows: 

source /opt/intel/oneapi/setvars.sh

Then you can load the ollama/0.9.3-ipex-llm-2.3 module and the following environment variables are set:

  • ONEAPI_DEVICE_SELECTOR="level_zero:0": select the GPU to run models as follow
    • level_zero:0: Intel Arc A770 eGPU (by default),
    • level_zero:1: Intel Arc Mobile iGPU.
  • OLLAMA_NUM_GPU=999: to make sure all layers of your model are running on Intel GPU, otherwise, some layers may run on CPU.
  • no_proxy=localhost,127.0.0.1
  • ZES_ENABLE_SYSMAN="1"
  • SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS="1"

OLLAMA_NUM_GPU, no_proxy, ZES_ENABLE_SYSMAN and SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS variables are recommended and detailed in the IPEX-LLM documentation.

Warning

Ollama v0.9.3 is quite outdated now (released Jun 25, 2025) and some models will not run with this version. Please refer to the "Comments" column in the Ollama Models table to check if the model can run with this version.

To determine whether IPEX-LLM is incompatible with a given model, either it will be indicated that the model is not compatible with Ollama v0.9.3+IPEX-LLM, or the required version of Ollama will be higher than v0.9.3. If none of the above reasons are mentioned, Ollama v0.9.3+IPEX-LLM should work fine on Intel GPUs.

Warning

For now, even though this version of Ollama appears to run on the Arc Mobile iGPU, we haven't been able to generate tokens consistently, and the implementation seems to have some bugs.

This contradicts the IPEX-LLM documentation, which states that the Intel Core Ultra processors and Intel Arc A-Series GPU are supported. Even on the main Readme of the project, Intel Core Ultra iGPU is mentioned to work...

Danger

IPEX-LLM is no longer supported by Intel, and the project was archived in early 2026... To the best of our knowledge, Intel has not yet announced any alternative to support Ollama on its GPUs.

There is an interesting discussion about IPEX-LLM alternatives on reddit.