RealityKit & USDKit in iOS 27: Spatial Computing for iOS Developers

USDKit is a new first-party Swift framework introduced in iOS/macOS 27 that brings native USD scene creation, composition, modification, and export capabilities to Apple platform apps, with deep RealityKit and Spatial Preview integration.

iOS 27 introduces new SwiftUI drag and drop APIs including reorderable, reorderContainer, dragContainer, and configuration modifiers that enable reordering within and across collections, multi-item drag, and fine-grained control over how data is transferred during drag and drop operations.

PaperKit is Apple's full-featured canvas framework — previously internal-only — now publicly available in iOS/macOS/visionOS 27. It powers the drawing and markup experience in Notes, Preview, and Freeform, giving developers access to a complete pencil, shapes, images, and text canvas with a rich data model.

The Spatial Preview framework lets macOS apps share and synchronize 2D and 3D content to Apple Vision Pro in real time, enabling users to view spatial photos, immersive video, PDFs, and USD scenes at full scale using Mac Virtual Display or a device picker.

Reality Composer Pro 3 introduces Animation Graph, Behavior Trees, Script Graph, Navigation Mesh, and Compute Graph — a suite of visual, node-based editors that let you author character animation, autonomous AI routines, interactivity, pathfinding, and particle effects entirely within the editor without writing code.

visionOS 27 introduces Gaussian Splatting support in RealityKit, enabling developers to scan real-world objects and render them as photorealistic 3D Gaussian splats directly in their spatial experiences. This bridges the gap between physical reality and virtual content without requiring manual 3D modeling.

Reality Composer Pro 3 introduces a node-based visual scripting system called Script Graph that lets developers and designers build interactive RealityKit experiences — including full games — without writing any code. Features include drag gesture events, physics integration, prototyped subgraphs, custom events, and live preview directly on Vision Pro.

The FoveatedStreaming framework lets visionOS apps connect to an external PC or cloud endpoint to stream OpenXR content directly to Apple Vision Pro, using eye-tracking-based foveated video compression and the built-in NVIDIA CloudXR runtime for low-latency wireless delivery.

Xcode plugins can now extend Reality Composer Pro 3 by registering custom ECS components, systems, animation actions, and Script Graph nodes so artists can iterate on 3D content directly inside the editor without rebuilding the app.

iOS 27 introduces on-device AI-generated subtitles that automatically transcribe or translate audio/subtitles during video playback, plus a new subtitle style preview API that lets users audition caption styles without leaving the player.

iOS 27 adds soft shadow rendering to RealityKit spotlights and point lights via a configurable lightSize and quality, plus a SurroundingsLightComponent that lets virtual lights illuminate real-world surfaces using the scene understanding mesh.

Safari on visionOS now exposes a JavaScript immersive API that lets websites transport users into full 3D environments using the HTML model element, mirroring the Fullscreen API pattern with requestImmersive(). Developers can build inline 3D previews and seamless transitions into spatial environments directly from a webpage.

The Evaluations framework in Xcode 27 lets developers generate and validate synthetic test data at scale using the SampleGenerator API, then run robust evaluations against agentic workflows that involve tool calling — all from Swift code.

visionOS 27 introduces ClippingComponent and enhanced ManipulationComponent placement patterns in RealityKit, enabling multiple users in a SharePlay session to simultaneously manipulate, disassemble, and cross-section structured 3D assemblies in shared space.

Reality Composer Pro 3 is a standalone tool (no longer bundled inside Xcode) that introduces live on-device preview, GPU-based Compute Graphs, a Lightmapping system for baked indirect lighting, a prototype/instancing workflow, and an AI Assistant that generates 3D objects and materials on demand for visionOS experiences.

iOS 27 introduces APIs and standards for building live production tools for Apple Immersive Video, enabling developers to capture, transport, record, and play back 3D immersive video using ProRes over SMPTE 2110 with spatial audio and per-frame metadata. The format combines ProRes-encoded stereo video (2110-22), ASAF spatial audio (2110-30), and JSON metadata (2110-41) into a unified production pipeline.

RAW 9 is a major new decoder version for CIRAWFilter that uses a tiled CoreML model running on the Apple Neural Engine to combine demosaicing and denoising in a single pass, dramatically improving sharpness, color accuracy, and noise reduction for hundreds of supported camera models.

visionOS supports fully photo-realistic, spatially-aware 360° immersive environments with true depth, parallax, spatial audio, and motion — going far beyond static panoramas. Developers can now surface these environments on both native visionOS apps and the spatial web.

visionOS 27 expands ARKit object tracking with high-frame-rate tracking, metric-space pose queries, extended Create ML training, spatial accessory support, and brings object tracking to iOS for the first time.