iOS 27 On-Device AI & Apple Intelligence: Foundation Models and Beyond

Foundation Models is a new Apple framework introduced in iOS 27 that gives developers on-device access to the same Apple Intelligence language model powering system features, enabling text generation, structured output, and tool-calling entirely on-device without a network connection.

Visual Intelligence brings iOS 17's Visual Look Up capabilities to a new developer-facing API surface in iOS 27, letting apps pipe live camera frames or static images through on-device scene understanding to extract subjects, text, barcodes, and rich semantic labels without any cloud round-trip.

App Schemas let developers describe their app's content and actions using pre-defined domain schemas (like the Calendar domain) so Siri can understand, search, and act on app data without custom NLP. Entities conforming to IndexedEntity are donated to Spotlight's semantic index, enabling natural-language queries over app content.

iOS 27 introduces GenerateIterativeSegmentationRequest in the Vision framework, letting users interactively isolate any object in an image by providing a point, bounding box, lasso, or scribble as a seed, then iteratively refine the resulting mask.

Core AI is Apple's new on-device inference framework for iOS 27, giving developers direct access to the same high-performance ML execution engine that powers Apple Intelligence. It supports CPU, GPU, and Neural Engine across all Apple Silicon devices.

Visual Intelligence now lets apps register as image search providers via App Intents, returning matched entities when users highlight and search images. Introduced in iOS 26 and now expanded to iPadOS and macOS in iOS 27, apps can surface rich, ranked results directly in the Visual Intelligence UI.

Core AI is a new iOS 27 framework that lets developers bring their own on-device AI models (vision transformers, LLMs, etc.) directly into apps, using the familiar FoundationModels session API. Models run entirely on-device with no cloud dependency, no per-token cost, and no data leaving the device.

SpotlightSearchTool is a new Foundation Models tool that lets a language model directly query your app's Core Spotlight index, enabling conversational, context-aware search over your app's own donated content. Developers can drop it into any LanguageModelSession to get AI-generated answers grounded in app data.

App Schemas let developers map their app's entities and actions to predefined Siri-understandable structures, grouped into App Schema Domains (e.g. messages, photos, mail), so Siri can find content, answer questions, and execute actions in natural language without any custom NLU code.

Dynamic profiles let you switch language models, instructions, and tools within a single LanguageModelSession, enabling multi-agent orchestration patterns like baton-pass and phone-a-friend directly in your app. Combined with session properties and lifecycle modifiers, you can build context-aware, multi-stage AI workflows on-device and in the cloud.

A new Apple framework for measuring the quality and reliability of intelligent features powered by generative AI. It integrates with Swift Testing to let developers define datasets, metrics, and optimization targets that automatically assess probabilistic outputs from language models.

iOS 27 introduces security patterns and APIs for protecting agentic features built with Foundation Models and App Intents, including lifecycle event modifiers to inject deterministic security checkpoints into LLM-driven agent execution loops.

iOS 27 opens the Foundation Models framework to third-party LLM providers via a new public LanguageModel protocol, allowing anyone to integrate server-based or local models (e.g. Claude, Gemini, CoreAI, MLX) using the same API as Apple's on-device system model.

A new on-device framework that analyzes audio files for musical properties including beat/rhythm, key signature, structure (sections/segments/phrases), pace, instrument activity, and loudness — all without requiring ML or signal processing expertise.

iOS 27 opens the Foundation Models framework to third-party LLM providers via a new public LanguageModel protocol, enabling anyone to integrate custom, server-based, or open-source models using the same Swift API as Apple's on-device system model.

The Evaluations framework in iOS 27 lets developers iteratively improve AI-powered features by running structured evaluations, scoring outputs with a model judge, and measuring alignment between model and human ratings using Cohen's kappa coefficient. This hill-climbing workflow enables systematic prompt and feature quality improvement with confidence.

Core AI is Apple's new Python-based suite for converting, optimizing, and deploying PyTorch models on Apple Silicon. It covers the full lifecycle from torch.export through compression via coreai-opt to on-device inference, with a companion Core AI Debugger app for runtime inspection.

macOS 27 ships a pre-installed `fm` command-line tool and a pip-installable Python SDK that expose the on-device Apple Foundation Model outside of Swift apps, enabling shell scripts, automation pipelines, and Python-based ML evaluation workflows.

The Evaluations framework in Xcode 27 lets developers generate and validate synthetic test data at scale using the SampleGenerator API, then run robust evaluations against agentic workflows that involve tool calling — all from Swift code.

MLX-LM Server exposes a locally running language model via an OpenAI-compatible HTTP API on your Mac, enabling fully offline agentic AI workflows — tool calling, multi-step reasoning, and concurrent subagents — with no cloud dependency. New in macOS 26, MLX targets dedicated Neural Accelerators on M5 for up to 4× faster prompt processing.

iOS 27 introduces on-device AI-generated subtitles that automatically transcribe or translate audio/subtitles during video playback, plus a new subtitle style preview API that lets users audition caption styles without leaving the player.

MLX Swift is an open-source array computing framework for Apple platforms that lets you write mathematical code using n-dimensional arrays, with automatic GPU execution and automatic differentiation via function transformations like `grad`. It brings NumPy-style numerical computing to Swift with lazy evaluation and a clean, math-like API.

macOS 26.2 introduces RDMA over Thunderbolt 5 and the JACCL collective communication library, enabling MLX to shard large language model inference and fine-tuning across multiple Apple Silicon Macs in a cluster. Developers can now run models too large for a single machine or dramatically accelerate token generation by spreading computation across up to N nodes with a single CLI flag.

A new Instruments template for the Foundation Models framework lets you record, visualize, and debug on-device and server-based LLM sessions — showing sessions, requests, model inferences, tool calls, token counts, and latency breakdowns in a single timeline.

iOS 27 expands App Intents to deeply integrate with Siri's on-device language model, enabling natural-language invocation of app actions with richer parameter resolution and multi-step task chaining. Apps can now expose capabilities that Siri understands contextually without rigid phrase matching.

iOS 27 extends App Intents with new APIs for custom Siri dialog responses, interaction donations, OwnershipProvidingEntity for smart confirmations, and IndexedEntityQuery for semantic Spotlight indexing — enabling richer, more personal Siri and Apple Intelligence experiences.

TensorOps is a Metal Shading Language library that lets developers write optimized custom machine learning kernels—including matrix multiplication, convolution, and FlashAttention—with automatic hardware acceleration across all Apple Silicon GPU generations, including the new M5 neural accelerator. iOS/macOS 27 extends its quantized data type support to FP8, 2-bit integers, and MX scaling formats.

ImagePlayground.framework now generates high-quality, photorealistic images via Private Cloud Compute instead of on-device models, with a redesigned API that replaces the deprecated ImageCreator. Developers present a familiar sheet UI and receive a URL to the generated image.

RAW 9 is a major new decoder version for CIRAWFilter that uses a tiled CoreML model running on the Apple Neural Engine to combine demosaicing and denoising in a single pass, dramatically improving sharpness, color accuracy, and noise reduction for hundreds of supported camera models.