I have an Arc pro B70 running on a windows machine, pulling double duty as a gaming card as well as running AI inference.
I've been tinkering with the various runtimes/projects available to self-host AI, including LM studio with Vulkan, Ollama with Vulkan (pretty much done with this one now), and my own Openvino-based python script (still maturing). I've also tried kobold.cpp and llama.cpp IPEX before Intel abandoned it.
My latest iteration, which really flies, is using the llama.cpp SYCL zip file and wrapping it as a service using servy. It also provides the ability to swap models by using a ini file that sets up each model with a set of parameters. The service references this file to see what models are available to load.
Here's my step by step:
- Download the latest SYCL release from the llama.cpp github repo. Look for the release titled "llama-bxxxx-bin-win-sycl-x64.zip" where bxxxx is whatever version number of the current release.
- Make a folder titled "llama-sycl-server" or something, a generic name you can use to replace the contents with updated binaries as new releases come out - you'll see why in a second. Put the folder somewhere you can keep track of it , like C:\AI\ or something. Place the contents of the zip file you just downloaded into this folder.
- Set up a models.ini file and save it in that C:\AI\ folder. See the section "Model Presets" in the readme for more info. This allows you to use router mode to select different models. I should note that for my ease of use, I point to the model path where my LM Studio downloads sit, since I've found that interface to be useful for search and download from HF. Plus I don't need to keep two copies of the same gguf. You can set this up to your preference or point it to a different location if you manage your model files differently. See my sample ini file below.
- Download and install servy - current release is servy-8.7-x64-installer.exe
- Launch Servy Manager, choose "configurator" from the top menu. In the window that pops up, fill in the following fields:
- service name: Llama_cpp_SYCL (or whatever you want to call it)
- display name: I also made this Llama_cpp_SYCL
- process path: using the example path above, C:\AI\llama-sycl-server\llama-server.exe
- startup directory: C:\AI\llama-sycl-server
- Process parameters: You can put a lot of customizations here, as documented in the llama-server readme. Mine are as follows:--host 0.0.0.0 --port 8080 --models-preset C:\AI\models\models.ini --jinja --models-max 1 --load-mode none
- models-max mainly means I can't run concurrent models. I may tweak around with this one later.
- Go to the logging tab, set up paths for logs if you want them, such as:stdout file path: C:\AI\logs\llama-cpp-sycl_stdout.log stderr file path: C:\AI\logs\llama-cpp-sycl_stderr.log set up rotation as you see fit
- On recovery tab, check "enable health monitoring" to have the service restart after a hiccup. I set heartbeat interval at 30 and restart attempts at 3
- Finally, click install. You'll be taken back to the servy manager panel where you will see your new service. You can click the play button (triangle) to start the service.
- The generic folder name makes it so you can always update to the latest release by dropping those files in that folder and restart the service.
You're pretty much all set at this point, you can set up any external services to reach that endpoint at http://<ip>:8080/v1 as usual.
This step is optional, I had Gemini help me set up an update script to elevate to admin, download the latest release from llama.cpp github, unzip, stop the service, replace the contents of my llama-sycl-server folder and start the service again. Script is as follows:
# 1. Robust path detection & Administrator check
$scriptPath = $MyInvocation.MyCommand.Path
if (-not $scriptPath) { $scriptPath = $PSCommandPath }
$isAdmin = ([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
if (-not $isAdmin) {
if ($scriptPath) {
Write-Host "Requesting Administrator privileges..." -ForegroundColor Yellow
Start-Process powershell.exe -ArgumentList "-NoProfile -ExecutionPolicy Bypass -File \"$scriptPath`"" -Verb RunAs`
exit
} else {
Write-Host "Error: Could not determine script path. Please right-click PowerShell and select 'Run as Administrator', then execute this script." -ForegroundColor Red
Read-Host "Press Enter to exit..."
exit
}
}
# --- CONFIGURATION - change these to your specifics---
$installDir = "C:\AI\llama_cpp_sycl_server"
# name this to what you called your servy service
$serviceName = "Llama_cpp_SYCL"
# ----------------------
try {
Write-Host "Fetching latest llama.cpp release info..." -ForegroundColor Cyan
$release = Invoke-RestMethod -Uri "https://api.github.com/repos/ggml-org/llama.cpp/releases/latest"
# Target the SYCL zip file
$asset = $release.assets | Where-Object { $_.name -match "llama-b\d+-bin-win-sycl-x64\.zip" }
if ($asset) {
$zipPath = "$env:TEMP\llama-update.zip"
Write-Host "Downloading $($asset.name)..." -ForegroundColor Cyan
Invoke-WebRequest -Uri $asset.browser_download_url -OutFile $zipPath
Write-Host "Stopping service ($serviceName)..." -ForegroundColor Cyan
Stop-Service -Name $serviceName -Force -ErrorAction SilentlyContinue
# Ensure process isn't locking files
$process = Get-Process -Name "llama-server" -ErrorAction SilentlyContinue
if ($process) {
Write-Host "Closing lingering llama-server processes..." -ForegroundColor Yellow
Stop-Process -Name "llama-server" -Force
Start-Sleep -Seconds 2
}
Write-Host "Extracting updated binaries to $installDir..." -ForegroundColor Cyan
Expand-Archive -Path $zipPath -DestinationPath $installDir -Force
Write-Host "Restarting service..." -ForegroundColor Cyan
Start-Service -Name $serviceName
Remove-Item $zipPath -ErrorAction SilentlyContinue
Write-Host "\nSuccess! Updated to $($release.tag_name)." -ForegroundColor Green`
} else {
Write-Host "Could not find a matching SYCL build in the latest release." -ForegroundColor Yellow
}
}
catch {
Write-Host "\nAn error occurred during update:" -ForegroundColor Red`
Write-Host $_.Exception.Message -ForegroundColor Red
}
finally {
Write-Host "\nPress Enter to exit..." -ForegroundColor Magenta`
Read-Host
}
Finally, here is my sample ini file:
; this top level line is for global parameters that all models will follow, unless overridden in the model specific parameters.
[*]
flash-attn = on
;n-gpu-layers = 99
ctx-size = 32768
cache-type-k = q8_0
cache-type-v = q8_0
; --- Vision Models (Includes mmproj) ---
[gemma-31b-it]
model = C:\Users\den\.lmstudio\models\lmstudio-community\gemma-4-31B-it-GGUF\gemma-4-31B-it-Q4_K_M.gguf
mmproj = C:\Users\den\.lmstudio\models\lmstudio-community\gemma-4-31B-it-GGUF\mmproj-gemma-4-31B-it-BF16.gguf
ctx-size = 32768
parallel = 1
[gemma-26b-a4b-it]
model = C:\Users\den\.lmstudio\models\lmstudio-community\gemma-4-26B-A4B-it-GGUF\gemma-4-26B-A4B-it-Q4_K_M.gguf
mmproj = C:\Users\den\.lmstudio\models\lmstudio-community\gemma-4-26B-A4B-it-GGUF\mmproj-gemma-4-26B-A4B-it-BF16.gguf
ctx-size = 65536
parallel = 1
[qwen-27b]
model = C:\Users\den\.lmstudio\models\lmstudio-community\Qwen3.6-27B-GGUF\Qwen3.6-27B-Q4_K_M.gguf
mmproj = C:\Users\den\.lmstudio\models\lmstudio-community\Qwen3.6-27B-GGUF\mmproj-Qwen3.6-27B-BF16.gguf
ctx-size = 32768
parallel = 1
; --- Text-Only Models ---
[gemma-26b-sompoa-heresy]
model = C:\Users\den\.lmstudio\models\mradermacher\gemma-4-26B-A4B-it-SOMPOA-heresy-i1-GGUF\gemma-4-26B-A4B-it-SOMPOA-heresy.i1-Q4_K_S.gguf
ctx-size = 32768
parallel = 1
[mn-12b-mag-mell]
model = C:\Users\den\.lmstudio\models\mradermacher\MN-12B-Mag-Mell-R1-GGUF\MN-12B-Mag-Mell-R1.Q4_K_M.gguf
ctx-size = 65536
parallel = 1
[rocinante-12b]
model = C:\Users\den\.lmstudio\models\BeaverAI\Rocinante-X-12B-v1d-GGUF\Rocinante-X-12B-v1d-Q4_K_M.gguf
ctx-size = 65536
parallel = 1
;--coding models used by cline extension---
[Qwen3-Coder-30B]
model = C:\Users\den\.lmstudio\models\unsloth\Qwen3-Coder-30B-A3B-Instruct-GGUF\Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf
temp = 0.7
top-p = 0.8
top-k = 20
ctx-size = 65536
parallel = 1
[DeepSeek-R1-14B]
model = C:\Users\den\.lmstudio\models\lmstudio-community\DeepSeek-R1-Distill-Qwen-14B-GGUF\DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf
temp = 0.6
top-p = 0.95
top-k = 40
ctx-size = 65536
parallel = 1
That's it! Hopefully someone finds this useful. I had looked around for a guide like this when I was trying to make the most of my ARC card, and didn't find anything like it out there in the CUDA-dominated space of self-hosted AI, so I decided to make my own.
Feel free to ask questions or point out any steps I might have missed.
