rkllama.cpp: llama.cpp tuned for the RK3576 NPU

Why I built it

Getting llama.cpp built with NPU support on the RK3576 took too many manual steps. This wraps them into one script so a fresh board is ready in minutes.

Architecture

How it fits together

Setup

Build script

Build

llama.cpp
RKNPU2 support

Run

GGUF models
Q8_0 and F16
NPU, with CPU fallback for MoE

Documentation

From the README

Open on GitHub

RK3576 optimized build setup for llama.cpp with RKNPU2 acceleration support.

Based on:

invisiofficial/rk-llama.cpp
ggml / llama.cpp

Features

RK3576 NPU acceleration
Automatic build script
Simplified setup
GGUF model support
Q8_0 / F16 support
MoE CPU fallback support

Requirements

RK3576 board
Linux
RKNPU2 drivers installed
cmake
git
gcc / g++

Install dependencies:

sudo apt update
sudo apt install -y git cmake build-essential

Quick Start

Clone repository:

git clone https://github.com/anand34577/rkllama.cpp-rk3576.git
cd rkllama.cpp-rk3576

Run build script:

chmod +x build_rk3576.sh
./build_rk3576.sh

Running Models

Standard GGUF

RKNPU_DEVICE=RK3576 ./build/bin/llama-cli -m model.gguf

F16 Model

RKNPU_DEVICE=RK3576 ./build/bin/llama-cli -m model-F16.gguf

MoE Models

RKNPU_DEVICE=RK3576 ./build/bin/llama-cli \
  -m moe-model.gguf \
  --cpu-moe

Cleaning Build Files

rm -rf build

Credits

This project is based on:

rkllama.cpp