Apple’s Foundation Models Framework Now Accepts Image Inputs

Published by Carl Sanson on June 8, 2026June 8, 2026

What You Need to Know

Apple’s on-device AI framework now accepts image inputs alongside text for multimodal capabilities.
Server-side model execution option added, suggesting on-device processing hit capability limits for developers.
Xcode coding assistant now handles app localization and simulated device interaction without manual intervention.
App Intents expansion opens Siri integration pathway to third-party developers, addressing long-standing complaints.

Apple’s on-device AI framework now accepts image inputs alongside text, a quiet but meaningful expansion that moves it closer to the multimodal capabilities developers have been building around in competing platforms for the past year.

The Foundation Models framework additions also include custom skills and server-side model execution. That last piece matters: Apple has consistently marketed on-device processing as a privacy feature, so offering a server-side path suggests developers were hitting real capability ceilings that local inference couldn’t clear.

The Xcode side of the announcement is where Apple’s agentic ambitions are most visible. The coding assistant can now handle app localization and interact directly with simulated devices, two tasks that previously required manual developer intervention. Craig Federighi calling Xcode the “best place” to build with agentic coding is the kind of competitive framing Apple usually reserves for when it thinks it has something real to show.

Developer Reach Through App Intents

The expanded App Intents support is worth watching for a different reason. By citing Line as an example of third-party Siri integration, Apple is signaling that the pathway for outside developers to expose actions through Siri is genuinely open, not just a feature reserved for Apple’s own apps. That has been a persistent complaint from developers since Siri Shortcuts launched in 2018.

The Core AI framework announcement received the least detail in today’s session, with Apple pointing to the State of the Union for more. That staging suggests it may be the more consequential piece, or at minimum the one Apple wants to control the framing around more carefully.

What Apple is assembling here is a layered AI development stack: on-device models, optional cloud execution, agentic tooling in the IDE, and Siri as an action layer into third-party apps. Whether those layers actually work together as smoothly as the session implied is a question the developer community will answer over the next few months.

Source: Apple Unveils Xcode and Foundation Models Framework Improvements (macrumors.com)