Speculative summarization

In this example, we will use the summarization speculation preset to generate a summary of the input text.

Python
Swift
TypeScript
Rust

Create a new project

uv init demo && cd demo

Install dependencies

uv add uzu

Paste into main.py

Run the snippet

uv run main.py

Create a new SwiftUI project

Add the SDK

Add this package through SPM:

https://github.com/trymirai/uzu.git

Paste the snippet

Add the snippet call

var body: some View {
    VStack {
        Text("On-device AI")
    }
    .onAppear() {
        Task {
            try await runExampleName()
        }
    }
}

Make a new project folder

mkdir demo && cd demo

Initialize with pnpm

pnpm init

Install dependencies

pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu

Initialize a tsconfig.json

{
    "compilerOptions": {
        "target": "es2020",
        "module": "commonjs",
        "moduleResolution": "node",
        "strict": true,
        "esModuleInterop": true,
        "outDir": "dist",
        "types": [
            "node"
        ]
    },
    "include": [
        "*.ts"
    ]
}

Create main.ts

Run the snippet

pnpm ts-node main.ts

Create a new project

cargo new demo && cd demo

Install dependencies

cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full

Paste into src/main.rs

Run the snippet

cargo run --release

You will notice that the model’s run count is lower than the actual number of generated tokens due to speculative decoding, which significantly improves generation speed.

Overview

Quick start

Speculative summarization