Visual Intelligence now lets apps register as image search providers via App Intents, returning matched entities when users highlight and search images. Introduced in iOS 26 and now expanded to iPadOS and macOS in iOS 27, apps can surface rich, ranked results directly in the Visual Intelligence UI.
⢠Apps can surface relevant content (albums, products, events) directly inside the system Visual Intelligence results sheet without any user context-switching
⢠The same IntentValueQuery, entity, and OpenIntent code works across iOS, iPadOS, and macOS with no platform-specific rewrites required
⢠System store integrations (EventKit, Contacts, HealthKit) let Visual Intelligence automatically feed structured data into your app, turning the OS into a passive input source
Demonstrates registering an App Entity and IntentValueQuery so Visual Intelligence can search a local music catalog by image similarity using Vision feature prints, then open the matched album in-app.
import AppIntents
import VisualIntelligence
import Vision
import UIKit
import VideoToolbox
// MARK: - App Entity
struct AlbumEntity: AppEntity {
static var typeDisplayRepresentation: TypeDisplayRepresentation = "Album"
static var defaultQuery = AlbumEntityQuery()
var id: String
var name: String
var artistName: String
var thumbnailURL: URL?
var displayRepresentation: DisplayRepresentation {
DisplayRepresentation(
title: "\(name)",
subtitle: "\(artistName)",
image: thumbnailURL.map { .init(url: $0) }
)
}
}
// MARK: - Entity Query (standard App Intents lookup)
struct AlbumEntityQuery: EntityQuery {
func entities(for identifiers: [String]) async throws -> [AlbumEntity] {
MusicCatalog.shared.albums(for: identifiers)
}
}
// MARK: - Visual Intelligence Intent Value Query
struct AlbumImageSearchQuery: IntentValueQuery {
typealias Value = AlbumEntity
func values(for requirement: SemanticContentDescriptor) async throws -> [AlbumEntity] {
guard let pixelBuffer = requirement.pixelBuffer else { return [] }
// Convert CVPixelBuffer -> CGImage
var cgImage: CGImage?
VTCreateCGImageFromCVPixelBuffer(pixelBuffer, options: nil, imageOut: &cgImage)
guard let image = cgImage else { return [] }
// Generate a feature print for the captured image
let request = GenerateImageFeaturePrintRequest()
let results = try await request.perform(on: image)
guard let queryPrint = results.first?.featurePrint else { return [] }
// Compare against pre-computed catalog prints
let maxDistance: Float = 0.45
let matches = MusicCatalog.shared.allAlbums.compactMap { album -> (AlbumEntity, Float)? in
guard let catalogPrint = album.featurePrint else { return nil }
var distance: Float = 0
try? catalogPrint.computeDistance(&distance, to: queryPrint)
return distance < maxDistance ? (album.entity, distance) : nil
}
return matches
.sorted { $0.1 < $1.1 }
.prefix(4)
.map(\.0)
}
}
// MARK: - Open Intent (navigates to album on tap)
struct OpenAlbumIntent: OpenIntent {
static var title: LocalizedStringResource = "Open Album"
@Parameter(title: "Album") var target: AlbumEntity
@MainActor
func perform() async throws -> some IntentResult {
AppNavigator.shared.navigate(to: .album(id: target.id))
return .result()
}
}
// MARK: - Minimal catalog stub (replace with real data layer)
struct CatalogAlbum {
var entity: AlbumEntity
var featurePrint: VNFeaturePrintObservation?
}
class MusicCatalog {
static let shared = MusicCatalog()
var allAlbums: [CatalogAlbum] = []
func albums(for ids: [String]) -> [AlbumEntity] {
allAlbums.filter { ids.contains($0.entity.id) }.map(\.entity)
}
}Foundation Models is a new Apple framework introduced in iOS 27 that gives developers on-device access to the same Apple Intelligence language model powering system features, enabling text generation, structured output, and tool-calling entirely on-device without a network connection.
iOS 27 opens the Foundation Models framework to third-party LLM providers via a new public LanguageModel protocol, enabling anyone to integrate custom, server-based, or open-source models using the same Swift API as Apple's on-device system model.
App Schemas let developers describe their app's content and actions using pre-defined domain schemas (like the Calendar domain) so Siri can understand, search, and act on app data without custom NLP. Entities conforming to IndexedEntity are donated to Spotlight's semantic index, enabling natural-language queries over app content.
In-depth guide
iOS 27 On-Device AI & Apple Intelligence āOnly one IntentValueQuery accepting a SemanticContentDescriptor is allowed per app; use @UnionValue to return multiple entity types from a single query. Pre-compute Vision feature prints at catalog build time, not at query time, to stay within latency budgets. The OpenIntent.perform() runs as the app foregrounds ā keep navigation lightweight and defer heavy work until after the view appears.
Apple Intelligence-capable devices required; on macOS the captured pixel buffer can be significantly larger than on iPhone ā consider resizing before running CV models
Visual Intelligence brings iOS 17's Visual Look Up capabilities to a new developer-facing API surface in iOS 27, letting apps pipe live camera frames or static images through on-device scene understanding to extract subjects, text, barcodes, and rich semantic labels without any cloud round-trip.