Visual Intelligence brings iOS 17's Visual Look Up capabilities to a new developer-facing API surface in iOS 27, letting apps pipe live camera frames or static images through on-device scene understanding to extract subjects, text, barcodes, and rich semantic labels without any cloud round-trip.
• On-device privacy: all inference runs locally via Apple's Neural Engine, so no user images are uploaded to a server
• Drop-in SwiftUI view modifier surfaces real-time results with minimal boilerplate, replacing a complex AVFoundation + Vision pipeline
• Unified API spans QR/barcode, text recognition, subject lift, and scene classification under one coherent request model
Attaches a Visual Intelligence analyzer to a live camera feed and displays the top scene label and any detected text in real time using the new VisualIntelligence SwiftUI integration.
import SwiftUI
import VisualIntelligence
@available(iOS 27, *)
struct LiveSceneAnalyzerView: View {
@State private var topLabel: String = "Scanning…"
@State private var detectedText: String = ""
@State private var analyzer = VisualIntelligenceAnalyzer()
var body: some View {
ZStack(alignment: .bottom) {
// Built-in camera preview wired to the analyzer
VisualIntelligenceCameraView(analyzer: analyzer)
.ignoresSafeArea()
VStack(alignment: .leading, spacing: 8) {
Label(topLabel, systemImage: "eye")
.font(.headline)
if !detectedText.isEmpty {
Label(detectedText, systemImage: "text.viewfinder")
.font(.subheadline)
.lineLimit(2)
}
}
.padding()
.background(.ultraThinMaterial, in: RoundedRectangle(cornerRadius: 16))
.padding()
}
.task {
// Stream analysis results as an AsyncSequence
do {
let stream = analyzer.results(
for: [
.sceneClassification(maximumCount: 3),
.textRecognition(languages: ["en"])
]
)
for try await result in stream {
await MainActor.run {
if let scene = result.sceneClassification?.first {
topLabel = "\(scene.label) (\(Int(scene.confidence * 100))%)"
}
if let textObs = result.textRecognition?.first {
detectedText = textObs.string
}
}
}
} catch {
await MainActor.run { topLabel = "Error: \(error.localizedDescription)" }
}
}
.onDisappear { analyzer.stop() }
}
}
#Preview {
if #available(iOS 27, *) {
LiveSceneAnalyzerView()
}
}Foundation Models is a new Apple framework introduced in iOS 27 that gives developers on-device access to the same Apple Intelligence language model powering system features, enabling text generation, structured output, and tool-calling entirely on-device without a network connection.
App Schemas let developers describe their app's content and actions using pre-defined domain schemas (like the Calendar domain) so Siri can understand, search, and act on app data without custom NLP. Entities conforming to IndexedEntity are donated to Spotlight's semantic index, enabling natural-language queries over app content.
iOS 27 introduces GenerateIterativeSegmentationRequest in the Vision framework, letting users interactively isolate any object in an image by providing a point, bounding box, lasso, or scribble as a seed, then iteratively refine the resulting mask.
In-depth guide
iOS 27 On-Device AI & Apple Intelligence →VisualIntelligence framework is new in iOS 27 beta and API surface may shift before GM; always guard with #available(iOS 27, *). Live camera analysis requires explicit NSCameraUsageDescription in Info.plist. Results arrive on a background queue — marshal UI updates to MainActor.
Requires Apple Silicon or A17 Pro or later Neural Engine; some subject-lift features require iPhone 15 Pro or later
iOS 27 opens the Foundation Models framework to third-party LLM providers via a new public LanguageModel protocol, enabling anyone to integrate custom, server-based, or open-source models using the same Swift API as Apple's on-device system model.