New AI Tool Can Translate Simple Screenshots Of Websites Into Working Code
By Mikelle Leow, 29 Nov 2023
Photo 203614294 © Nadezhda Mikhalitskaia | Dreamstime.com
Imagine taking a glance at something and being able to figure out its skeleton and guts at once. Well, this is practically what this new artificial intelligence tool, called Screenshot-to-Code, is said to be able to do. It studies screenshots of a website or application interface and, with the magic of the algorithm, turns what is seemingly face value into actual code.
So, how does it work? Screenshot-to-Code—developed by independent hacker Abi Raja—utilizes the power of GPT-4 with Vision, which is able to accept visual inputs and answer questions about them, and DALL-E 3, which generates the outputs, to reverse-engineer your screenshot into functional code of HTML, Tailwind CSS, React, Vue, or Bootstrap formats.
During its process, the AI progressively refines the code and continuously compares it to the screenshot to ensure accuracy, explains Siqi Chen, the co-founder of startup Runway and a member of the board of directors at Sandbox VR, on X/Twitter.
screenshot-to-code:
— Siqi Chen (@blader) November 21, 2023
upload a screenshot of any website, watch as AI progressively builds the html, iteratively improving the generated code by comparing it against the screenshot repeatedly. pic.twitter.com/TCrbJj3VS0
In a video shared by Chen, the model impressively scans through a snapshot of the YouTube homepage and recreates its layout, replacing the thumbnails with AI-generated pictures.
For those who only need the code without image generation, there’s an option to disable DALL-E too. You could also choose to share a URL instead of a screenshot.
Under the hood, Screenshot-to-Code features a React/Vite front end and a FastAPI backend. To get started, you'll need access to GPT-4 Vision and an API key from OpenAI.
This innovative tool is hosted on GitHub, which means it’s open-source and available for anyone to use.
As with many AI apps out there, this one isn’t perfect, with users occasionally running into errors. Screenshot-to-Code should thus only be incorporated into the iteration stage, instead of serving as the full picture.
While the idea is for such tools to make complex tasks simpler and more accessible for developers and designers, it simultaneously raises some concerns about the speed at which AI is catching up to human abilities, and the protection of intellectual property. Whatever you do, make sure you have the appropriate permissions and rights first.
[via Creative Bloq and Siqi Chen, cover photo 203614294 © Nadezhda Mikhalitskaia | Dreamstime.com]
This article was crafted with assistance from an AI engine, and has been manually reviewed & edited.