LLMs Runtime

To run LLMs on Dalek nodes, you can use the Open WebUI frontend on your machine locally and connect directly to a node running the backend of your choice (Ollama, llama.cpp, vLLM, ...) on one of Dalek's nodes.

Tip

For a daily usage, consider using LLoby as backend for your AI applications. Here are some key advantages in using LLoby:

• Flexibility: OpenAI compatibility for supporting a wide range of AI tools.
• Automated Resource Management: Node allocation is handled transparently and automatically, simplifying deployment.
• Optimized Performance: Service is faster, especially when models are already loaded by other users.
• Efficient Resource Sharing: Inference resources are shared across users, maximizing efficiency and minimizing computational waste.

Open WebUI

From the quick-start page, there are multiple installation methods. We follow here the one for docker. Pull the image from the docker repositories:

docker pull ghcr.io/open-webui/open-webui:main

Run it on your local machine, making it available on localhost:3000:

sudo docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway \
                -e WEBUI_AUTH=False \
                -v open-webui:/app/backend/data \
                --name open-webui ghcr.io/open-webui/open-webui:main

Info

The important options here are that we launch Open WebUI in single user mode (we don't want to handle multiple accounts on a local machine) and the --add-host=host.docker.internal:host-gateway option will redirect localhost of you main OS to host.docker.internal inside the docker container. This will simplify the connection to the node(s) later.

Ollama

Running Ollama on a Node

Multiple runtimes (also called backends) are available but Open WebUI handles Ollama out-of-the-box and makes getting models running easy.

You need to allocate exclusively the node to be able to connect with SSH on it. This can be done by connecting to the front then issuing either an srun or sbatch:

srun -w [NODE_NAME] --exclusive --interactive --pty bash

Then, on the node, start ollama by running:

module load ollama
ollama serve

Info

This makes Ollama listen to web requests on the node, on port 11434.

Port Forwarding

You then need to perform port forwarding to make your local and node 11434 ports communicate. On an allocated node, run the following:

ssh -J front.dalek.lip6 -N -f [USER_DALEK]@[NODE_NAME] -L 0.0.0.0:11434:localhost:11434

Info

This commands connects to the node by first performing a jump on the front. It then performs port forwarding to bind your port 11434 with the node's localhost:11434. The binding is done on all your local interfaces. This command runs in the background without launching a shell so if no error are printed after issuing it, it is working properly.

[USER_DALEK] is your Dalek login and [NODE_NAME] is the name of the node you want to connect to (typically something like az4-n4090-2 or az4-a7900-0).

To check if it works, you can browse http://localhost:11434/ and it should display Ollama is running.

Last step is to navigate to Open WebUI Admin Settings (or localhost:3000 page, User > Settings > Administration Settings > Connections > Ollama API (or ) and make sure that the Ollama API connection is set to http://host.docker.internal:11434.

That's all folks!

Installation Notes

Models are installed and shared by all the users. The list of the available models is given in the Ollama Models section. So, by default, you cannot add or update these models (unless if you are in the ai-models group, which is not automatic, see the Technical Details about ai-models Group section for more information about it). If you need specific models or specific versions, you can do unset OLLAMA_MODELS after the module load ollama in order for Ollama to default to your home repository as the default model library. As the disk quota available on the NFS is limited, you need to set Ollama default models path to the scratch and download your models there.

For example, you can do the following to store the models on the scratch:

module load ollama
mkdir -p /scratch/$USER/ollama/
OLLAMA_MODELS=/scratch/$USER/ollama/ ollama serve

Tips

Sometimes, large models can take a long time to load into memory. This is because NFS is not as fast as a local disk. If you need to reuse a specific model (or a subset of models) frequently, it may be a good idea to copy it to the scratch workspace.

Danger

Please keep in mind that models are heavy. When you store them on the NFS or the scratch, keep an eye on what you really use and clean occasionally.

Backend APIs

Since Ollama v15.0.0, Dalek provides multiple modules to help users to target specific backends:

• ollama/x.y.z-cpu: execute only on CPU.
• ollama/x.y.z-cuda: run with CUDA (Nvidia GPUs).
• ollama/x.y.z-rocm: run with ROCm (AMD GPUs).
• ollama/x.y.z-vulkan: run Vulkan (support many different GPUs).
• ollama/x.y.z-zauto: automatic version, let Ollama decide how to run.

Even though different modules are used for a given version, the same Ollama executable binary is always used. The only difference lies in the definition of certain specific environment variables used to configure Ollama. For all the modules, the following environment variables are set:

• OLLAMA_KEEP_ALIVE="-1": keep models in RAM and VRAM indefinitely as long as Ollama serves.
• OLLAMA_HOST="0.0.0.0": serve any incoming IP addresses.
• OLLAMA_MODELS="/mnt/data/ai-models/ollama/": as explained in the Installation Notes section, by default Ollama will search for preinstalled models in the /mnt/data/ai-models/ollama/ shared folder on the NFS.

The following subsections describe the environment variables depending on the proposed modules.

ollama/x.y.z-cpu

The following environment variable is set:

• OLLAMA_LLM_LIBRARY="cpu": force Ollama to run models on CPU (and this to avoid GPU).

ollama/x.y.z-cuda

The following environment variable is set:

• OLLAMA_LLM_LIBRARY="cuda_v13": force Ollama to run models with CUDA v13 (OLLAMA_LLM_LIBRARY="cuda_v12" is also possible)

ollama/x.y.z-rocm

The following environment variables are set:

• OLLAMA_LLM_LIBRARY="rocm": force Ollama to run models with ROCm (\(\approx\) HIP).
• HSA_OVERRIDE_GFX_VERSION=11.5.1: only on the az5-a890m partition. If not set, the Radeon 890M iGPU is not supported by ROCm and the models execute on CPU.

ollama/x.y.z-vulkan

The following environment variables are set:

• OLLAMA_VULKAN="1": force Ollama to use Vulkan backend.
• GGML_VK_VISIBLE_DEVICES="[x]": specify the GPU id to use. [x] is a placeholder and depending on the partition its default value is:
  • az4-n4090: 1 (select the GeForce RTX 4090 dGPU and NOT the Radeon 610M iGPU).
  • az4-a7900: 1 (select the Radeon RX 7900 XTX dGPU and NOT the Radeon 610M iGPU).
  • iml-ia770: 1 (select the Arc A770 eGPU and NOT the Arc Mobile iGPU)
  • az5-a890m: 0 (select the Radeon 890M iGPU).

Note

Users can override GGML_VK_VISIBLE_DEVICES environment variable to target the GPU they want.

Warning

Vulkan does not appear to work on the Intel GPUs of the iml-ia770 partition (Intel Arc A770 eGPU and Intel Arc Mobile iGPU).

ollama/0.9.3-ipex-llm-2.3

This is a specific version provided by Intel that combines with IPEX-LLM to run Ollama on Intel GPUs like the ones available in the iml-ia770 partition (Arc A770 eGPU and Arc Mobile iGPU).

You can load the ollama/0.9.3-ipex-llm-2.3 module and the following environment variables are set:

• ONEAPI_DEVICE_SELECTOR="level_zero:0": select the GPU to run models as follow
  • level_zero:0: Intel Arc A770 eGPU (by default),
  • level_zero:1: Intel Arc Mobile iGPU.
• OLLAMA_NUM_GPU=999: to make sure all layers of your model are running on Intel GPU, otherwise, some layers may run on CPU.
• no_proxy=localhost,127.0.0.1
• ZES_ENABLE_SYSMAN="1"
• SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS="1"

OLLAMA_NUM_GPU, no_proxy, ZES_ENABLE_SYSMAN and SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS variables are recommended and detailed in the IPEX-LLM documentation.

Warning

Ollama v0.9.3 is quite outdated now (released Jun 25, 2025) and some models will not run with this version. Please refer to the "Comments" column in the Ollama Models table to check if the model can run with this version.

To determine whether IPEX-LLM is incompatible with a given model, either it will be indicated that the model is not compatible with Ollama v0.9.3+IPEX-LLM, or the required version of Ollama will be higher than v0.9.3. If none of the above reasons are mentioned, Ollama v0.9.3+IPEX-LLM should work fine on Intel GPUs.

Warning

For now, even though this version of Ollama appears to run on the Arc Mobile iGPU, we haven't been able to generate tokens consistently, and the implementation seems to have some bugs.

This contradicts the IPEX-LLM documentation, which states that the Intel Core Ultra processors and Intel Arc A-Series GPU are supported. Even on the main Readme of the project, Intel Core Ultra iGPU is mentioned to work...

Danger

IPEX-LLM is no longer supported by Intel, and the project was archived in early 2026... To the best of our knowledge, Intel has not yet announced any alternative to support Ollama on its GPUs.

There is an interesting discussion about IPEX-LLM alternatives on reddit.

Llama.cpp

Llama.cpp is used internally by Ollama and can also be run as a standalone application. The primary difference is that the llama-server web client requires a specific model to be loaded at startup. Consequently, you must restart the server each time you want to switch between different models. A key advantage of llama.cpp over Ollama is its support for cutting-edge builds and diverse backend optimizations (e.g., SYCL, OpenCL, and OpenVINO).

Running Llama.cpp Server on a Node

To connect via SSH, you must allocate an exclusive node. You can do this by connecting to the front node and running either srun or sbatch:

srun -w [NODE_NAME] --exclusive --interactive --pty bash

Once on the node, start the server by loading the appropriate module and running llama-server:

module load llama/b[BUILD]-[BACKEND]
llama-server --n-gpu-layers 100 --ctx-size 4096 --port 11434 --host 0.0.0.0 -m "$LLAMA_MODELS/[MODEL]"

Parameters:

• [NODE_NAME]: The name of the node (e.g., iml-ia770-1, az4-a7900-0, ...).
• [BUILD]: The specific llama.cpp build version.
• [BACKEND]: Choose the backend based on your target hardware (e.g., cpu, cuda, opencl, openvino, rocm, sycl-fp16, sycl-fp32, or vulkan). For more details, see the Backend APIs section.
• [MODEL]: Select a GGUF model from the available list (see GGUF Models section) or provide a path to a manually downloaded model (Note: models are large files and it is preferred to used the ones located at /mnt/data/ai-models).

Tip

If you have reserved the iml-ia770-1 node, for example, you can run:

module load llama/b9632-sycl-fp16
llama-server --n-gpu-layers 100 --ctx-size 4096 --port 11434 --host 0.0.0.0 -m "$LLAMA_MODELS/gemma4:12b"

This command configures llama.cpp to listen for web requests on the node at port 11434, put all the model layers on GPU (Intel Arc A770 eGPU here) with a maximum context size of 4096 tokens. Finally, it runs an Gemma 4 model with 12 billions of parameters.

Port Forwarding

Next, you must set up port forwarding to allow your local machine to communicate with the node's port 11434. From an allocated node, run the following command:

ssh -J front.dalek.lip6 -N -f [USER_DALEK]@[NODE_NAME] -L 0.0.0.0:11434:localhost:11434

Info

This command establishes a tunnel by first jumping through the front node. It then binds your local port 11434 to the node's localhost:11434 across all local interfaces.

Because the -f flag is used, the command runs in the background without opening a shell. If no error is displayed, the tunnel is active.

Parameters:

• [USER_DALEK]: Your Dalek login.
• [NODE_NAME]: The name of the node (e.g., iml-ia770-1, az4-a7900-0, ...).

Verification:

To verify the connection, visit http://localhost:11434/. You should see the llama-ui web interface. You can now begin testing the model.

Open WebUI Configuration:

To connect Open WebUI to the llama.cpp endpoint, navigate to Admin Panel > Settings > Connections > OpenAI API - Manage OpenAI API Connections and add the appropriate URL:

• Docker instance: http://host.docker.internal:11434/v1
• Python venv: http://localhost:11434/v1

Backend APIs

CPU

To load this backend:

module load llama/b[BUILD]-cpu

This backend targets the CPU. While it generally offers lower performance compared to GPU or NPU acceleration, it provides the highest level of portability across different systems.

CUDA

To load this backend:

module load llama/b[BUILD]-cuda

This backend is exclusive to NVIDIA GPUs (available on the az4-n4090 partition on Dalek). It is designed to provide the highest performance for these architectures.

Official documentation: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#cuda

ROCm

To load this backend:

module load llama/b[BUILD]-rocm

This backend is exclusive to AMD GPUs (available on the az4-a7900 partition on Dalek) and is optimized for maximum performance on these devices and relies on HIP kernels.

Official documentation: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#hip

Tip

Important environment variables:

• HSA_OVERRIDE_GFX_VERSION=<VERSION>: for iGPUs that are not officially supported by ROCm, fake that it is a dGPU:
  • Radeon 610M: <VERSION>=10.3.0
  • Radeon 780M: <VERSION>=11.0.2
  • Radeon 890M: <VERSION>=11.5.1

This environment variable is automatically set by the amd-rocm module depending on the targeted node.

OpenCL

To load this backend:

module load llama/b[BUILD]-opencl

OpenCL is designed for portability across various CPUs and GPUs. According to the official llama.cpp documentation, this backend is specifically optimized for Adreno GPUs.

Official documentation: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENCL.md

Tip

Important environment variables:

• GGML_OPENCL_DEVICE=<GPUID>: select between multiple targets

OpenVINO

To load this backend:

module load llama/b[BUILD]-openvino

This backend is tailored for Intel hardware (CPUs, GPUs, and NPUs). It is currently the only backend in llama.cpp capable of targeting the NPUs on the iml-ia770 partition. Please note that this is a "work in progress" and is not yet fully operational in our environment.

Official documentation: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/OPENVINO.md

SYCL

To load this backend:

module load llama/b[BUILD]-sycl-fp16

or:

module load llama/b[BUILD]-sycl-fp32

SYCL is a portable backend compatible with various CPUs and GPUs. The llama.cpp developers note that it is optimized for Intel GPUs and Intel Level Zero. This is the recommended method for utilizing these resources (see iml-ia770, nas, and demo nodes on Dalek).

Note on Precision: The difference between sycl-fp16 and sycl-fp32 lies in the precision used during the prefill phase. While sycl-fp16 offers faster speeds, it provides lower numerical accuracy compared to sycl-fp32.

Official documentation: https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/SYCL.md

Tip

Important environment variables:

• ONEAPI_DEVICE_SELECTOR="level_zero:<GPUID>": with Level Zero API, select the GPU(s) to run models
  • level_zero:0: GPU id 0,
  • level_zero:1: GPU id 1.
  • level_zero:0;level_zero:1: GPU ids 0 and 1 together.
• ZES_ENABLE_SYSMAN="1": Support to get free memory of GPU by sycl::aspect::ext_intel_free_memory. Recommended to use when --split-mode = layer
• SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS="1"

ZES_ENABLE_SYSMAN and SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS variables are recommended and detailed in the IPEX-LLM documentation.

Vulkan

To load this backend:

module load llama/b[BUILD]-vulkan

Originally designed for graphics stacks, Vulkan has gained significant traction in the AI ecosystem. It is compatible with a wide range of modern GPUs and serves as an excellent fallback solution when primary backends (such as CUDA, HIP, or SYCL) are unavailable.

Official documentation: https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#vulkan

Tip

Important environment variables:

• GGML_VK_VISIBLE_DEVICES="<GPUID>": specify the GPU id to use.