Yesterday, in its spring update, OpenAI released a native Mac app for ChatGPT and also debuted its new flagship model, GPT-4o . The demos generated a lot of excitement, especially among Apple users who were looking forward to the Apple-OpenAI partnership that is rumored to be coming to iOS 18 .
One important thing in the full event video was that the upcoming ChatGPT app feature was demoed but no real details were shared. That is, ChatGPT's ability to see everything that's happening on the user's device screen.
Are you ready to trust ChatGPT and share your screen content?
At launch, the ChatGPT Mac app supports taking a screenshot manually or uploading a file to then query ChatGPT for it. But in the future, as seen in the video below, you'll be able to click a button to give the app access to everything on your screen.
This feature apparently won't be limited to Macs, as iOS and iPadOS apps have demonstrated the same technology in the example below.
It’s unclear when these screen-viewing features will arrive in ChatGPT. However, the change could make for a more seamless, interactive experience. Apple even provides an API for developers to tap into the device’s screen content—with user consent, of course. So OpenAI isn’t doing anything nefarious by adding these features.
Having ChatGPT access to the entire content of your screen can be a bit scary, but it can also be extremely useful.
There will likely be a time limit on how long an app can access a user's screen, requiring a button to be pressed each time you want to grant access. But in theory, it makes sense that one day AI tools will have permanent access to everything you can see, allowing for more seamless assistance with minimal friction.
An AI assistant that can see everything you see on your devices seems like something Apple is aiming for at a system level across its platforms. If Siri gets a new AI boost, with deep system integration, it could prove to be a real asset for iOS 18 and later users.