A new on-device framework that analyzes audio files for musical properties including beat/rhythm, key signature, structure (sections/segments/phrases), pace, instrument activity, and loudness — all without requiring ML or signal processing expertise.
• Enables rich music-driven experiences like beat-synced video editing, DJ apps, and reactive game audio without any server-side processing or ML knowledge
• All analysis runs entirely on-device, keeping user audio private and working offline
• Pre-computed results are fully Codable, so you can bundle analysis data with your app or export it for later use
Loads a local audio file, runs Music Understanding analysis, and displays the detected key signature and BPM alongside a list of beat timestamps.
import SwiftUI
import MusicUnderstanding
import AVFoundation
struct BeatKeyAnalyzerView: View {
@State private var bpm: Double? = nil
@State private var keyLabel: String = "—"
@State private var beatTimes: [String] = []
@State private var isAnalyzing = false
@State private var showFilePicker = false
@State private var selectedURL: URL? = nil
var body: some View {
NavigationStack {
VStack(spacing: 20) {
Button("Select Audio File") {
showFilePicker = true
}
.buttonStyle(.borderedProminent)
if isAnalyzing {
ProgressView("Analyzing…")
} else {
VStack(alignment: .leading, spacing: 12) {
Label("Key: \(keyLabel)", systemImage: "music.note")
.font(.title2.bold())
Label("BPM: \(bpm.map { String(format: "%.1f", $0) } ?? "—")", systemImage: "metronome")
.font(.title2.bold())
Divider()
Text("First 10 Beat Timestamps")
.font(.headline)
ForEach(beatTimes, id: \.self) { t in
Text(t)
.font(.caption)
.foregroundStyle(.secondary)
}
}
.padding()
.frame(maxWidth: .infinity, alignment: .leading)
.background(.regularMaterial, in: RoundedRectangle(cornerRadius: 16))
.padding(.horizontal)
}
Spacer()
}
.navigationTitle("Music Understanding")
.fileImporter(
isPresented: $showFilePicker,
allowedContentTypes: [.audio]
) { result in
if case .success(let url) = result {
selectedURL = url
Task { await analyzeAudio(url: url) }
}
}
}
}
func analyzeAudio(url: URL) async {
isAnalyzing = true
defer { isAnalyzing = false }
let asset = AVURLAsset(
url: url,
options: [AVURLAssetPreferPreciseDurationAndTimingKey: true]
)
do {
let session = try MusicUnderstandingSession(asset: asset)
let results = try await session.analyze(for: [.rhythm, .key])
// Key
if let keyResult = results.key,
let firstRange = keyResult.ranges.first {
let sig = firstRange.value
keyLabel = "\(sig.tonic) \(sig.mode == .major ? "Major" : "Minor")"
}
// Rhythm
if let rhythmResult = results.rhythm {
bpm = rhythmResult.beatsPerMinute
beatTimes = rhythmResult.beats.prefix(10).map { beat in
let seconds = beat.seconds
return String(format: "%.3f s", seconds)
}
}
} catch {
keyLabel = "Error: \(error.localizedDescription)"
}
}
}
#Preview {
BeatKeyAnalyzerView()
}Foundation Models is a new Apple framework introduced in iOS 27 that gives developers on-device access to the same Apple Intelligence language model powering system features, enabling text generation, structured output, and tool-calling entirely on-device without a network connection.
iOS 27 opens the Foundation Models framework to third-party LLM providers via a new public LanguageModel protocol, enabling anyone to integrate custom, server-based, or open-source models using the same Swift API as Apple's on-device system model.
App Schemas let developers describe their app's content and actions using pre-defined domain schemas (like the Calendar domain) so Siri can understand, search, and act on app data without custom NLP. Entities conforming to IndexedEntity are donated to Spotlight's semantic index, enabling natural-language queries over app content.
In-depth guide
iOS 27 On-Device AI & Apple Intelligence →Set AVURLAssetPreferPreciseDurationAndTimingKey to true on your AVURLAsset for accurate timing results. The beatsPerMinute property on RhythmResult is optional — it returns nil until at least two beats have been detected. When using the targeted analyze(for:) API, all unrequested result fields will be nil.
Runs on-device; no Apple Intelligence hardware required, but performance may vary on older devices
Visual Intelligence brings iOS 17's Visual Look Up capabilities to a new developer-facing API surface in iOS 27, letting apps pipe live camera frames or static images through on-device scene understanding to extract subjects, text, barcodes, and rich semantic labels without any cloud round-trip.