# About
Source: https://docs.trymirai.com/index
Looking for an instant start? Check out the drop-in snippets:
Chat
Structured output
Cloud
Chat
Structured output
Cloud
Chat
Structured output
Cloud
Chat
Structured output
Cloud
[**Mirai**](https://trymirai.com/) lets you add high-performance AI right into your app with zero latency, full data privacy, and no inference costs. You don’t need an ML team or weeks of setup anymore. One developer can get it all running in minutes. To achieve this, we offer the following key products:
A Rust inference engine built to run AI with hardware specifics in mind.
A set of tools to optimize and convert models for on-device use.
A command-line tool to chat with models and serve them as a local API.
If you run into a problem and don’t find an answer here, feel free to reach out in our [Discord](https://discord.com/invite/trymirai) community. We can even debug the issue with you over voice chat, or you can [book a call](https://calendar.google.com/calendar/u/0/appointments/schedules/AcZssZ0RkTVFVCqRHJdidCc0aRAr4T44KeO58N01tMrNBkjbNFndBAaTf6rKdtTTEryPo8JB_UNQ8Lx6) with our engineers ❤️.
## FAQ
The [**uzu**](https://github.com/trymirai/uzu) and [**lalamo**](https://github.com/trymirai/lalamo) libraries are fully open source under the MIT license.
The [**uzu**](https://github.com/trymirai/uzu) and [**lalamo**](https://github.com/trymirai/lalamo) libraries are completly free to use.
Currently, only Apple Silicon (iOS/macOS) devices are supported.
The full list of supported models is available [here](https://trymirai.com/local-models).
# Chat
Source: https://docs.trymirai.com/quick-start/chat
In this example, we will download a model and get a reply to a specific list of messages.
```sh theme={null}
uv init demo && cd demo
```
```sh theme={null}
uv add uzu
```
```python theme={null}
import asyncio
from uzu import (
ChatConfig,
ChatMessage,
ChatReplyConfig,
ChatSessionStreamChunk,
Engine,
EngineConfig,
)
async def main() -> None:
engine_config = EngineConfig.create()
engine = await Engine.create(engine_config)
model = await engine.model("alibaba:qwen3.5:0.8b:mirai:mirai-m:4")
if model is None:
raise RuntimeError("Model not found")
async for update in (await engine.download(model)).iterator():
print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
print()
messages = [
ChatMessage.system().with_text("You are a helpful assistant"),
ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
]
session = await engine.chat(model, ChatConfig.create())
stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
message: ChatMessage | None = None
async for chunk in stream.iterator():
if isinstance(chunk, ChatSessionStreamChunk.Replies):
replies = chunk.replies
if replies:
reply = replies[0]
message = reply.message
print(f"Generated tokens: {reply.stats.tokens_count_output}")
elif isinstance(chunk, ChatSessionStreamChunk.Error):
print(f"Error: {chunk.error}")
if message is not None:
print(f"Reasoning: {message.reasoning}")
print(f"Text: {message.text}")
if __name__ == "__main__":
asyncio.run(main())
```
```sh theme={null}
uv run main.py
```
![]()
Add this package through SPM:
```sh theme={null}
https://github.com/trymirai/uzu.git
```
```swift theme={null}
import Foundation
import Uzu
public func runChat() async throws {
let engineConfig = EngineConfig.create()
let engine = try await Engine.create(config: engineConfig)
guard let model = try await engine.model(identifier: "alibaba:qwen3.5:0.8b:mirai:mirai-m:4") else {
return
}
for try await update in try await engine.download(model: model).iterator() {
print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
fflush(stdout)
}
print()
let messages = [
ChatMessage.system().withText(text: "You are a helpful assistant"),
ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
]
let session = try await engine.chat(model: model, config: .create())
let stream = await session.replyWithStream(input: messages, config: .create())
var message: ChatMessage? = nil
for try await update in stream.iterator() {
switch update {
case .replies(let replies):
let reply = replies.last
message = reply?.message
print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
case .error(let error):
print("Error: \(error)")
}
}
print("Reasoning: \(message?.reasoning() ?? "empty")")
print("Text: \(message?.text() ?? "empty")")
}
```
```swift theme={null}
var body: some View {
VStack {
Text("On-device AI")
}
.onAppear() {
Task {
try await runExampleName()
}
}
}
```
```sh theme={null}
mkdir demo && cd demo
```
```sh theme={null}
pnpm init
```
```sh theme={null}
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```
```json theme={null}
{
"compilerOptions": {
"target": "es2020",
"module": "commonjs",
"moduleResolution": "node",
"strict": true,
"esModuleInterop": true,
"outDir": "dist",
"types": [
"node"
]
},
"include": [
"*.ts"
]
}
```
```ts theme={null}
import {
ChatConfig,
ChatMessage,
ChatReplyConfig,
ChatSessionStreamChunkError,
ChatSessionStreamChunkReplies,
Engine,
EngineConfig
} from '@trymirai/uzu';
async function main() {
let engineConfig = EngineConfig.create();
let engine = await Engine.create(engineConfig);
let model = await engine.model('alibaba:qwen3.5:0.8b:mirai:mirai-m:4');
if (!model) {
throw new Error('Model not found');
}
for await (const update of await engine.download(model)) {
process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
}
console.log();
let messages = [
ChatMessage.system().withText('You are a helpful assistant'),
ChatMessage.user().withText('Tell me a short, funny story about a robot')
];
let session = await engine.chat(model, ChatConfig.create());
let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
let message: ChatMessage | undefined;
for await (const chunk of stream) {
if (chunk instanceof ChatSessionStreamChunkReplies) {
message = chunk.replies[0]?.message;
console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
} else if (chunk instanceof ChatSessionStreamChunkError) {
console.error('Error: ', chunk.error);
}
}
console.log('Reasoning: ', message?.reasoning);
console.log('Text: ', message?.text);
}
main().catch((error) => {
console.error(error);
});
```
```sh theme={null}
pnpm ts-node main.ts
```
```sh theme={null}
cargo new demo && cd demo
```
```sh theme={null}
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```
```rust theme={null}
use std::io::{self, Write};
use uzu::{
engine::{Engine, EngineConfig},
session::chat::ChatSessionStreamChunk,
types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};
#[tokio::main]
async fn main() -> Result<(), Box> {
let engine_config = EngineConfig::default();
let engine = Engine::new(engine_config).await?;
let model = engine.model("alibaba:qwen3.5:0.8b:mirai:mirai-m:4".to_string()).await?.ok_or("Model not found")?;
let downloader = engine.download(&model).await?;
while let Some(update) = downloader.next().await {
print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
io::stdout().flush()?;
}
println!();
let messages = vec![
ChatMessage::system().with_text("You are a helpful assistant".to_string()),
ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
];
let session = engine.chat(model, ChatConfig::default()).await?;
let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
let mut last_message: Option = None;
while let Some(chunk) = stream.next().await {
match chunk {
ChatSessionStreamChunk::Replies {
replies,
} => {
if let Some(reply) = replies.first() {
last_message = Some(reply.message.clone());
println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
}
},
ChatSessionStreamChunk::Error {
error,
} => {
println!("Error: {error}");
},
}
}
if let Some(message) = last_message {
println!("Reasoning: {}", message.reasoning().unwrap_or_default());
println!("Text: {}", message.text().unwrap_or_default());
}
Ok(())
}
```
```sh theme={null}
cargo run --release
```
Once loaded, the same `ChatSession` can be reused for multiple requests until you drop it. Each model may consume a significant amount of RAM, so it's important to keep only one session loaded at a time. For iOS apps, we recommend adding the [Increased Memory Capability](https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.developer.kernel.increased-memory-limit) entitlement to ensure your app can allocate the required memory.
# Cloud
Source: https://docs.trymirai.com/quick-start/chat-cloud
In this example, we will get a reply to a specific list of messages from a cloud model.
```sh theme={null}
uv init demo && cd demo
```
```sh theme={null}
uv add uzu
```
```python theme={null}
import asyncio
from uzu import ChatConfig, ChatMessage, ChatReplyConfig, Engine, EngineConfig, ReasoningEffort
async def main() -> None:
engine_config = EngineConfig.create().with_openai_api_key("OPENAI_API_KEY")
engine = await Engine.create(engine_config)
model = await engine.model("gpt-5")
if model is None:
raise RuntimeError("Model not found")
messages = [
ChatMessage.system().with_reasoning_effort(ReasoningEffort.Low),
ChatMessage.user().with_text("How LLMs work"),
]
session = await engine.chat(model, ChatConfig.create())
replies = await session.reply(messages, ChatReplyConfig.create())
if replies:
message = replies[0].message
print(f"Reasoning: {message.reasoning}")
print(f"Text: {message.text}")
if __name__ == "__main__":
asyncio.run(main())
```
```sh theme={null}
uv run main.py
```
![]()
Add this package through SPM:
```sh theme={null}
https://github.com/trymirai/uzu.git
```
```swift theme={null}
import Uzu
public func runChatCloud() async throws {
let engineConfig = EngineConfig.create().withOpenaiApiKey(openaiApiKey: "OPENAI_API_KEY")
let engine = try await Engine.create(config: engineConfig)
guard let model = try await engine.model(identifier: "gpt-5") else {
return
}
let messages = [
ChatMessage.system().withReasoningEffort(reasoningEffort: .low),
ChatMessage.user().withText(text: "How LLMs work")
]
let session = try await engine.chat(model: model, config: .create())
let reply = try await session.reply(input: messages, config: .create())
guard let message = reply.last?.message else {
return
}
print("Reasoning: \(message.reasoning() ?? "empty")")
print("Text: \(message.text() ?? "empty")")
}
```
```swift theme={null}
var body: some View {
VStack {
Text("On-device AI")
}
.onAppear() {
Task {
try await runExampleName()
}
}
}
```
```sh theme={null}
mkdir demo && cd demo
```
```sh theme={null}
pnpm init
```
```sh theme={null}
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```
```json theme={null}
{
"compilerOptions": {
"target": "es2020",
"module": "commonjs",
"moduleResolution": "node",
"strict": true,
"esModuleInterop": true,
"outDir": "dist",
"types": [
"node"
]
},
"include": [
"*.ts"
]
}
```
```ts theme={null}
import { ChatConfig, ChatMessage, ChatReplyConfig, Engine, EngineConfig, ReasoningEffort } from '@trymirai/uzu';
async function main() {
let engineConfig = EngineConfig.create().withOpenaiApiKey('OPENAI_API_KEY');
let engine = await Engine.create(engineConfig);
let model = await engine.model('gpt-5');
if (!model) {
throw new Error('Model not found');
}
let messages = [
ChatMessage.system().withReasoningEffort("Low" as ReasoningEffort),
ChatMessage.user().withText('How LLMs work')
];
let session = await engine.chat(model, ChatConfig.create());
let reply = await session.reply(messages, ChatReplyConfig.create());
let message = reply[0]?.message;
if (message) {
console.log('Reasoning: ', message.reasoning);
console.log('Text: ', message.text);
}
}
main().catch((error) => {
console.error(error);
});
```
```sh theme={null}
pnpm ts-node main.ts
```
```sh theme={null}
cargo new demo && cd demo
```
```sh theme={null}
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```
```rust theme={null}
use uzu::{
engine::{Engine, EngineConfig},
types::{
basic::ReasoningEffort,
session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
},
};
#[tokio::main]
async fn main() -> Result<(), Box> {
let engine_config = EngineConfig::default().with_openai_api_key("OPENAI_API_KEY".to_string());
let engine = Engine::new(engine_config).await?;
let model = engine.model("gpt-5".to_string()).await?.ok_or("Model not found")?;
let messages = vec![
ChatMessage::system().with_reasoning_effort(ReasoningEffort::Low),
ChatMessage::user().with_text("How LLMs work".to_string()),
];
let session = engine.chat(model, ChatConfig::default()).await?;
let replies = session.reply(messages, ChatReplyConfig::default()).await?;
if let Some(reply) = replies.first() {
println!("Reasoning: {}", reply.message.reasoning().unwrap_or_default());
println!("Text: {}", reply.message.text().unwrap_or_default());
}
Ok(())
}
```
```sh theme={null}
cargo run --release
```
# Structured output
Source: https://docs.trymirai.com/quick-start/chat-structured-output
Sometimes you want the generated output to be valid JSON with predefined fields. You can use `Grammar` to manually specify a JSON schema for the response you want to receive.
```sh theme={null}
uv init demo && cd demo
```
```sh theme={null}
uv add uzu
```
```python theme={null}
import asyncio
import json
from pydantic import BaseModel
from uzu import (
ChatConfig,
ChatMessage,
ChatReplyConfig,
Engine,
EngineConfig,
Grammar,
ReasoningEffort,
)
class Country(BaseModel):
name: str
capital: str
class CountryList(BaseModel):
countries: list[Country]
def structured_response(response: str | None, model_type: type[BaseModel]) -> BaseModel | None:
if not response:
return None
return model_type.model_validate_json(response)
async def main() -> None:
engine_config = EngineConfig.create()
engine = await Engine.create(engine_config)
model = await engine.model("alibaba:qwen3.5:0.8b:mirai:mirai-m:4")
if model is None:
raise RuntimeError("Model not found")
async for update in (await engine.download(model)).iterator():
print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
print()
schema_string = json.dumps(CountryList.model_json_schema())
messages = [
ChatMessage.system().with_reasoning_effort(ReasoningEffort.Disabled),
ChatMessage.user().with_text(
"Give me a JSON object containing a list of 3 countries, where each country has name and capital fields"
),
]
session = await engine.chat(model, ChatConfig.create())
replies = await session.reply(
messages,
ChatReplyConfig.create().with_grammar(Grammar.JsonSchema(schema_string)),
)
if replies:
countries = structured_response(replies[0].message.text, CountryList)
print(countries)
if __name__ == "__main__":
asyncio.run(main())
```
```sh theme={null}
uv run main.py
```
![]()
Add this package through SPM:
```sh theme={null}
https://github.com/trymirai/uzu.git
```
```swift theme={null}
import Foundation
import FoundationModels
import Uzu
@Generable()
struct Country: Codable {
let name: String
let capital: String
}
public func runChatStructuredOutput() async throws {
let engineConfig = EngineConfig.create()
let engine = try await Engine.create(config: engineConfig)
guard let model = try await engine.model(identifier: "alibaba:qwen3.5:0.8b:mirai:mirai-m:4") else {
return
}
for try await update in try await engine.download(model: model).iterator() {
print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
fflush(stdout)
}
print()
let messages = [
ChatMessage.system().withReasoningEffort(reasoningEffort: .disabled),
ChatMessage.user().withText(text: "Give me a JSON object containing a list of 3 countries, where each country has name and capital fields")
]
let session = try await engine.chat(model: model, config: .create())
let reply = try await session.reply(input: messages, config: .create().withGrammar(grammar: .fromType([Country].self)))
guard let message = reply.last?.message else {
return
}
guard let countries: [Country] = message.textDecoded() else {
return
}
print(countries)
}
```
```swift theme={null}
var body: some View {
VStack {
Text("On-device AI")
}
.onAppear() {
Task {
try await runExampleName()
}
}
}
```
```sh theme={null}
mkdir demo && cd demo
```
```sh theme={null}
pnpm init
```
```sh theme={null}
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```
```json theme={null}
{
"compilerOptions": {
"target": "es2020",
"module": "commonjs",
"moduleResolution": "node",
"strict": true,
"esModuleInterop": true,
"outDir": "dist",
"types": [
"node"
]
},
"include": [
"*.ts"
]
}
```
```ts theme={null}
import { ChatConfig, ChatMessage, ChatReplyConfig, Engine, EngineConfig, GrammarJsonSchema, ReasoningEffort } from '@trymirai/uzu';
import * as z from "zod";
const CountryType = z.object({
name: z.string(),
capital: z.string(),
});
const CountryListType = z.array(CountryType);
function structuredResponse(response: string | null | undefined, type: T): z.infer | undefined {
if (!response) {
return undefined;
}
const data = JSON.parse(response);
const result = type.parse(data);
return result;
}
async function main() {
let engineConfig = EngineConfig.create();
let engine = await Engine.create(engineConfig);
let model = await engine.model('alibaba:qwen3.5:0.8b:mirai:mirai-m:4');
if (!model) {
throw new Error('Model not found');
}
for await (const update of await engine.download(model)) {
process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
}
console.log();
let schema = z.toJSONSchema(CountryListType);
let schemaString = JSON.stringify(schema);
let messages = [
ChatMessage.system().withReasoningEffort("Disabled" as ReasoningEffort),
ChatMessage.user().withText('Give me a JSON object containing a list of 3 countries, where each country has name and capital fields')
];
let session = await engine.chat(model, ChatConfig.create());
let reply = await session.reply(messages, ChatReplyConfig.create().withGrammar(new GrammarJsonSchema(schemaString)));
let message = reply[0]?.message;
let countries = structuredResponse(message?.text, CountryListType);
console.log(countries);
}
main().catch((error) => {
console.error(error);
});
```
```sh theme={null}
pnpm ts-node main.ts
```
```sh theme={null}
cargo new demo && cd demo
```
```sh theme={null}
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```
```rust theme={null}
use std::io::{self, Write};
use schemars::{JsonSchema, schema_for};
use serde::{Deserialize, Serialize};
use uzu::{
engine::{Engine, EngineConfig},
types::{
basic::{Grammar, ReasoningEffort},
session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
},
};
#[derive(Debug, Serialize, Deserialize, JsonSchema)]
struct Country {
name: String,
capital: String,
}
#[derive(Debug, Serialize, Deserialize, JsonSchema)]
struct CountryList {
countries: Vec,
}
#[tokio::main]
async fn main() -> Result<(), Box> {
let engine_config = EngineConfig::default();
let engine = Engine::new(engine_config).await?;
let model = engine.model("alibaba:qwen3.5:0.8b:mirai:mirai-m:4".to_string()).await?.ok_or("Model not found")?;
let downloader = engine.download(&model).await?;
while let Some(update) = downloader.next().await {
print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
io::stdout().flush()?;
}
println!();
let schema_string = serde_json::to_string(&schema_for!(CountryList))?;
let messages = vec![
ChatMessage::system().with_reasoning_effort(ReasoningEffort::Disabled),
ChatMessage::user().with_text(
"Give me a JSON object containing a list of 3 countries, where each country has name and capital fields"
.to_string(),
),
];
let session = engine.chat(model, ChatConfig::default()).await?;
let chat_reply_config = ChatReplyConfig::default().with_grammar(Some(Grammar::JsonSchema {
schema: schema_string,
}));
let replies = session.reply(messages, chat_reply_config).await?;
if let Some(reply) = replies.first()
&& let Some(text) = reply.message.text()
{
let parsed: CountryList = serde_json::from_str(&text)?;
println!("{parsed:#?}");
}
Ok(())
}
```
```sh theme={null}
cargo run --release
```