Skip to main content

​
Overview

To run a model, we need to create an inference Session with the configuration we discussed in the previous step. Session is the core entity used to communicate with the model. After creation, the model weights are loaded into the device’s RAM and remain there until you deallocate the Session object. Once loaded, the same Session can be reused for multiple requests until you drop it.
Each model may consume a significant amount of RAM, so it’s important to keep only one session loaded at a time. For iOS apps, we recommend adding the Increased Memory Capability entitlement to ensure your app can allocate the required memory.
Don’t try to run a single Session instance from different threads, as it’s not thread-safe and doesn’t support batch requests.
After creating a Session, you need to build an input prompt. To do this, use the Input enum, which can have two possible values:

Input
messagesA list of role-based messages. Possible roles are: system, user, assistant.
textText to use as the prompt
The last step is simply calling the run method with the input and a max tokens limit. You can also optionally specify a sampling method if you want to override the one recommended by the model. This method accepts a callback with intermediate results and expects a boolean flag that indicates whether generation should continue.

​
Steps

Now we finally can run the model. Choose the use case that’s closest to your product.
  • Chat
  • Summarization
  • Classification
Simple flow for chat-based use cases
1

Create a session for the selected model

let session = try engine.chatSession(model, config: Config(preset: .general))
2

Build an input prompt

let messages = [
    Message(role: .system, content: "You are a helpful assistant."),
    Message(role: .user, content: "Tell me a short, funny story about a robot."),
]
let input: Input = .messages(messages: messages)
3

Run the session

let runConfig = RunConfig()
    .tokensLimit(1024)

let output = try session.run(
    input: input,
    config: runConfig
) { _ in
    return true
}
As a result, the session returns a Output object, which contains execution stats such as tokens per second or prefill time:
If you run into any problems with the flow described above, check the troubleshooting section.

Now that we’ve integrated AI into the app, you can also check out a detailed overview of how the uzu inference engine works under the hood.