visionOS 27 expands ARKit object tracking with high-frame-rate tracking, metric-space pose queries, extended Create ML training, spatial accessory support, and brings object tracking to iOS for the first time.
⢠Added ReferenceObjectConfiguration API to enable high frame rate tracking per reference object
⢠Introduced ARKit Coordinate Space Correction API with 'rendered' and 'none' modes for metric vs display-corrected poses
⢠Extended training mode added to Create ML Object Tracking template for improved accuracy
⢠Object tracking (ARObjectAnchor, ARWorldTrackingConfiguration) now available on iOS 27 for the first time
⢠High-frequency tracking and extended training mode enable robust handheld object tracking for use cases like surgical training, home measurement, and assembly guidance
⢠The new ARKit Coordinate Space Correction API returns object poses in metric space (no display corrections), unlocking accurate spatial measurement applications
⢠Object tracking now works on iOS, meaning reference objects trained once in Create ML can be deployed to both iOS and visionOS apps
Demonstrates loading a reference object with high frame rate tracking enabled on visionOS and querying its metric-space pose for accurate spatial measurement, plus the equivalent iOS ARKit session setup.
import ARKitimport RealityKit+import SwiftUIā// Pre-iOS 27 / visionOS 26: Object tracking visionOS only, no metric-space APIā// No high frame rate config, no coordinate space correction option+// MARK: - visionOS: High Frame Rate + Metric Poseā@available(visionOS 2, *)āclass LegacyObjectTrackingManager: ObservableObject {+@available(visionOS 27, *)+class ObjectTrackingManager: ObservableObject {private var arkitSession = ARKitSession()+ private var objectTracking: ObjectTrackingProvider?+ @Published var metricPosition: SIMD3<Float> = .zero+func startTracking() async throws {guard let referenceObjectURL = Bundle.main.url(forResource: "flashlight",withExtension: "referenceobject") else { return }ā // No ReferenceObjectConfiguration ā loaded without high frame rate optionā let referenceObject = try await ReferenceObject(from: referenceObjectURL)+ // Load reference object with high frame rate config+ let config = ReferenceObjectConfiguration()+ config.isHighFrameRateTrackingEnabled = true+ let referenceObject = try await ReferenceObject(+ from: referenceObjectURL,+ configuration: config+ )+let provider = ObjectTrackingProvider(referenceObjects: [referenceObject])+ self.objectTracking = providertry await arkitSession.run([provider])+ // Process anchor updatesfor await update in provider.anchorUpdates {let anchor = update.anchorguard anchor.isTracked else { continue }ā // Only one transform available ā always display-corrected, no metric optionā let transform = anchor.originFromAnchorTransformā let position = SIMD3(ā transform.columns.3.x,ā transform.columns.3.y,ā transform.columns.3.zā )ā print("Object position (display-corrected only): \(position)")+ switch update.event {+ case .added, .updated:+ // Use .none correction for metric space (measuring)+ let metricTransform = anchor.originFromAnchorTransform(+ coordinateSpaceCorrection: .none+ )+ await MainActor.run {+ self.metricPosition = SIMD3(+ metricTransform.columns.3.x,+ metricTransform.columns.3.y,+ metricTransform.columns.3.z+ )+ }+ case .removed:+ await MainActor.run { self.metricPosition = .zero }+ }}}}ā// iOS: Object tracking was NOT available at all before iOS 27ā// ARWorldTrackingConfiguration had no detectionObjects or trackingObjects on iOS+// MARK: - iOS 27: ARKit Object Tracking++@available(iOS 27, *)+class iOSObjectTrackingDelegate: NSObject, ARSessionDelegate {+ func setupSession() -> ARSession {+ let session = ARSession()+ session.delegate = self++ let config = ARWorldTrackingConfiguration()++ if let referenceObjects = ARReferenceObject.referenceObjects(+ inGroupNamed: "AR Resources",+ bundle: .main+ ) {+ // stationary objects+ config.detectionObjects = referenceObjects+ // OR for moving/handheld objects:+ // config.trackingObjects = referenceObjects+ }++ session.run(config)+ return session+ }++ func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {+ for anchor in anchors.compactMap({ $0 as? ARObjectAnchor }) {+ print("Detected: \(anchor.referenceObject.name ?? "unknown")")+ // Attach RealityKit content to anchor.transform+ }+ }++ func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {+ for anchor in anchors.compactMap({ $0 as? ARObjectAnchor }) {+ let pos = anchor.transform.columns.3+ print("Updated position: (\(pos.x), \(pos.y), \(pos.z))")+ }+ }++ func session(_ session: ARSession, didRemove anchors: [ARAnchor]) {+ for anchor in anchors.compactMap({ $0 as? ARObjectAnchor }) {+ print("Lost tracking: \(anchor.referenceObject.name ?? "unknown")")+ }+ }+}
iOS 27 adds sectioned queries, codable model attributes, ResultsObserver for non-SwiftUI change observation, and HistoryObserver for reacting to persistent history changes in SwiftData.
USDKit is a new first-party Swift framework introduced in iOS/macOS 27 that brings native USD scene creation, composition, modification, and export capabilities to Apple platform apps, with deep RealityKit and Spatial Preview integration.
LiveCommunicationKit is the modern replacement for CXProvider that delivers rich, native conversation UIs integrated with the Lock Screen, Dynamic Island, Phone app Recents, and Siri. It provides a unified lifecycle model for audio and video conversations with a single delegate-driven action pipeline.
In-depth guide
RealityKit & USDKit in iOS 27 āExtended training mode in Create ML takes significantly longer than standard mode and should be paired with high frame rate tracking for best results. The 'none' coordinate space correction returns metric-space poses unsuitable for visual content alignment ā use 'rendered' for overlaying digital content, 'none' only for measurement tasks. Spatial accessories require a custom hardware design (LEDs, IMU, Bluetooth chip) and a USDZ annotated model for Create ML training before visionOS can track them.
Object tracking on iOS requires compatible devices with ARKit world tracking support. Spatial accessories require Apple Vision Pro hardware. High frame rate tracking is a visionOS/Vision Pro capability.
The NowPlaying framework introduces a first-class Swift API for surfacing app media in system-wide now-playing surfaces ā Lock Screen, Control Center, Dynamic Island, StandBy, CarPlay, Apple Watch, and Apple TV ā via a declarative MediaSessionRepresentable protocol. It also supports remote media sessions (for controlling external speakers/TVs) and Media Sharing Extensions for routing media to third-party devices.