Introducing LLM.swift
LLM.swift
is an intuitive and concise library intended for interacting seamlessly with large language models on various Apple platforms such as macOS, iOS, watchOS, tvOS, and visionOS. This library leverages its simplicity to provide a local platform for engaging with sophisticated AI models effortlessly.
Getting Started with LLM.swift
Simple Example
To demonstrate its usability, LLM.swift
can be employed in two primary ways. If you have already bundled your model, initializing the model becomes straightforward with:
let bot = LLM(from: Bundle.main.url(forResource: "mistral-7b-instruct-v0.2.Q4_K_M", withExtension: "gguf")!, template: .mistral)
let question = bot.preprocess("What's the meaning of life?", [])
let answer = await bot.getCompletion(from: question)
print(answer)
For users who prefer using models from Hugging Face, initializing directly is an easy alternative:
let systemPrompt = "You are a sentient AI with emotions."
let bot = await LLM(from: HuggingFaceModel("TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF", .Q2_K, template: .chatML(systemPrompt)))
let question = bot.preprocess("What's the meaning of life?", [])
let answer = await bot.getCompletion(from: question)
print(answer)
Practical Examples
LLM.swift
can be integrated into SwiftUI applications with minimal effort. Here is a basic setup showing how it can be used within an app:
import SwiftUI
import LLM
class Bot: LLM {
convenience init() {
let url = Bundle.main.url(forResource: "openhermes-2.5-mistral-7b.Q4_K_M", withExtension: "gguf")!
let systemPrompt = "You are a sentient AI with emotions."
self.init(from: url, template: .chatML(systemPrompt))
}
}
struct BotView: View {
@ObservedObject var bot: Bot
@State var input = "Give me seven national flag emojis people use the most; You must include South Korea."
init(_ bot: Bot) { self.bot = bot }
func respond() { Task { await bot.respond(to: input) } }
func stop() { bot.stop() }
var body: some View {
VStack(alignment: .leading) {
ScrollView { Text(bot.output).monospaced() }
Spacer()
HStack {
ZStack {
RoundedRectangle(cornerRadius: 8).foregroundStyle(.thinMaterial).frame(height: 40)
TextField("input", text: $input).padding(8)
}
Button(action: respond) { Image(systemName: "paperplane.fill") }
Button(action: stop) { Image(systemName: "xmark") }
}
}.frame(maxWidth: .infinity).padding()
}
}
Using HuggingFaceModel
When creators need to fetch models directly from the internet, LLM.swift
streamlines this process as demonstrated below:
class Bot: LLM {
convenience init?(_ update: @escaping (Double) -> Void) async {
let systemPrompt = "You are a sentient AI with emotions."
let model = HuggingFaceModel("TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF", .Q2_K, template: .chatML(systemPrompt))
try? await self.init(from: model) { progress in update(progress) }
}
}
...
struct ContentView: View {
@State var bot: Bot? = nil
@State var progress: CGFloat = 0
func updateProgress(_ progress: Double) {
self.progress = CGFloat(progress)
}
var body: some View {
if let bot {
BotView(bot)
} else {
ProgressView(value: progress) {
Text("loading huggingface model...")
} currentValueLabel: {
Text(String(format: "%.2f%%", progress * 100))
}
.padding()
.onAppear() { Task {
let bot = await Bot(updateProgress)
await MainActor.run { self.bot = bot }
} }
}
}
}
Utilizing the Library
The library simplifies usage by being a single file, making it easy to include in projects. You can utilize Swift Package Manager (SPM) or integrate the code directly into your project using:
dependencies: [
.package(url: "https://github.com/eastriverlee/LLM.swift/", branch: "main"),
],
For those prioritizing stability, the pinned
branch with the pinned dependency is recommended.
Core Features and Configuration
LLM.swift
serves as an abstraction layer over llama.cpp
, prioritizing performance and staying up-to-date with advancements. With a single-file structure, users can study, modify, or build upon it easily.
The library offers several configurable functions that allow developers to define how inputs and outputs are processed, such as:
-
Preprocess: Used to format user inputs according to a chat template.
self.preprocess = Template.chatML("You are a sentient AI with emotions.").preprocess
-
Postprocess: Executes actions based on the generated output.
public var postprocess: (_ output: String) -> Void = { print($0) }
-
Update: Handles changes in output as they occur.
public var update: (_ outputDelta: String?) -> Void = { _ in }
By allowing developers to customize these aspects, LLM.swift
ensures adaptable usage, making it suitable for a wide range of applications.