GPT-4 Gives Visually-Impaired A ‘Virtual Volunteer’ For Practically Everything
OpenAI’s GPT has been blessed with an artificial set of eyes. Now, the bot is sharing its gifts to help others see.
For context, the new GPT-4 can now comprehend images on top of text prompts, enabling the powerful ChatGPT chatbot to digest and interpret visual information too. Its creator OpenAI quickly identified blind individuals and those with low vision as some of the users that could deeply benefit from this super-strength, and it has thus banded together with a free accessibility tool called Be My Eyes to, well, be their eyes.
Launched in 2015, the Danish startup assists the visually-impaired by connecting them with a human volunteer via a live video call to take on everyday tasks. For instance, a user could point their phone camera at a photo and have a sighted person describe it to them.
To date, Be My Eyes has served over six million users.
Now, OpenAI and Be My Eyes are bringing daily assistance closer to the community with the ‘Virtual Volunteer’, an artificially-intelligent helper that can perform those same tasks. Powered by the new GPT-4, Virtual Volunteer can scan and understand images, and then infer from them to answer questions posed by low-sighted users.
To illustrate, the companies have shared a demo showing how the Virtual Volunteer can work as a “fashion advisor” and “gym partner” for the user, among other things. It can compare outfits to help the user make styling decisions for the day, or guide them to an available treadmill at the gym. It can also translate signs into other languages, read maps, and detect plant species.
If you send the Virtual Volunteer a photo of the inside of your fridge, it can look around and suggest how to turn those ingredients into a meal. The AI generates a handful of recipes that include those items, as well as churns out a step-by-step guide on how to prepare them.
All told, the AI assistant is not here to replace the 6.3 million human helpers registered on the Be My Eyes platform. Users can still receive live help from existing volunteers for further assistance when the Virtual Volunteer doesn’t provide the answer they need.
Michael Buckley, CEO of Be My Eyes, lauds that GPT-4’s global implications are “profound” as it can dramatically increase the chances of the visually-impaired finding independence in their lives.
“In the not-so-distant future, the blind and low-vision community will utilize these tools not only for a host of visual interpretation needs, but also to have a greater degree of independence in their lives,” Buckley continues.
“Ultimately, whatever the user wants or needs, they can re-prompt the tool to get more information that is usable, beneficial and helpful, nearly instantly.”
OpenAI and Be My Eyes will continue working closely to expand Virtual Volunteer’s abilities.
For now, the tool is only available to a small group of Be My Eyes users, though circumstances should change in the following weeks or months. In any case, the Virtual Volunteer feature will always be free for iOS and Android users, despite GPT-4 being a paid privilege for ChatGPT Plus subscribers.
[via TechCrunch, Be My Eyes, OpenAI, cover image via Be My Eyes]
