A new Apple-backed study, in collaboration with Aalto University in Finland, introduces ILuvUI: a vision-language model trained to understand mobile app interfaces from screenshots and from natural language conversations. Here’s what that means, and how they did it. ILuvUI: an AI that outperformed the model it was based on In the paper, ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine …
Read More »Apple taught an AI model to reason about app interfaces
