Visual Intelligence brings iOS 17's Visual Look Up capabilities to a new developer-facing API surface in iOS 27, letting apps pipe live camera frames or static images through on-device scene understanding to extract subjects, text, barcodes, and rich semantic labels without any cloud round-trip.
• On-device privacy: all inference runs locally via Apple's Neural Engine, so no user images are uploaded to a server
• Drop-in SwiftUI view modifier surfaces real-time results with minimal boilerplate, replacing a complex AVFoundation + Vision pipeline
• Unified API spans QR/barcode, text recognition, subject lift, and scene classification under one coherent request model
Attaches a Visual Intelligence analyzer to a live camera feed and displays the top scene label and any detected text in real time using the new VisualIntelligence SwiftUI integration.
import SwiftUI
import VisualIntelligence
@available(iOS 27, *)
struct LiveSceneAnalyzerView: View {
@State private var topLabel: String = "Scanning…"
@State private var detectedText: String = ""
@State private var analyzer = VisualIntelligenceAnalyzer()
var body: some View {
ZStack(alignment: .bottom) {
// Built-in camera preview wired to the analyzer
VisualIntelligenceCameraView(analyzer: analyzer)
.ignoresSafeArea()
VStack(alignment: .leading, spacing: 8) {
Label(topLabel, systemImage: "eye")
.font(.headline)
if !detectedText.isEmpty {
Label(detectedText, systemImage: "text.viewfinder")
.font(.subheadline)
.lineLimit(2)
}
}
.padding()
.background(.ultraThinMaterial, in: RoundedRectangle(cornerRadius: 16))
.padding()
}
.task {
// Stream analysis results as an AsyncSequence
do {
let stream = analyzer.results(
for: [
.sceneClassification(maximumCount: 3),
.textRecognition(languages: ["en"])
]
)
for try await result in stream {
await MainActor.run {
if let scene = result.sceneClassification?.first {
topLabel = "\(scene.label) (\(Int(scene.confidence * 100))%)"
}
if let textObs = result.textRecognition?.first {
detectedText = textObs.string
}
}
}
} catch {
await MainActor.run { topLabel = "Error: \(error.localizedDescription)" }
}
}
.onDisappear { analyzer.stop() }
}
}
#Preview {
if #available(iOS 27, *) {
LiveSceneAnalyzerView()
}
}Foundation Models is a new Apple framework introduced in iOS 27 that gives developers on-device access to the same Apple Intelligence language model powering system features, enabling text generation, structured output, and tool-calling entirely on-device without a network connection.
iOS 27 opens the Foundation Models framework to third-party LLM providers via a new public LanguageModel protocol, allowing anyone to integrate server-based or local models (e.g. Claude, Gemini, CoreAI, MLX) using the same API as Apple's on-device system model.
iOS 27 introduces GenerateIterativeSegmentationRequest in the Vision framework, letting users interactively isolate any object in an image by providing a point, bounding box, lasso, or scribble as a seed, then iteratively refine the resulting mask.
In-depth guide
iOS 27 On-Device AI & Apple Intelligence →VisualIntelligence framework is new in iOS 27 beta and API surface may shift before GM; always guard with #available(iOS 27, *). Live camera analysis requires explicit NSCameraUsageDescription in Info.plist. Results arrive on a background queue — marshal UI updates to MainActor.
Requires Apple Silicon or A17 Pro or later Neural Engine; some subject-lift features require iPhone 15 Pro or later
iOS 27 opens the Foundation Models framework to third-party LLM providers via a new public LanguageModel protocol, enabling anyone to integrate custom, server-based, or open-source models using the same Swift API as Apple's on-device system model.