How AI Is Turning Pictures into Recipes

Hey, all you foodies out there roaming the Web for new and exciting ideas to tickle your palate… One day you might be able to get your favorite recipe printed out just by looking at some picture of a lovingly arranged dish, then let your AI-enabled gourmet app go over it, analyze it and come up with all its ingredients, including the instructions how to prepare it.

Sounds miraculous? Not if you follow the latest news coming from Facebook’s Artificial Intelligence Research (FAIR) group in a recent interview with the US Business Magazine Forbes. Until now, the work is just a concept or trial and not yet ready for popular use. But large brand-name food producers are waiting already to present delicious dishes for their easy replication via Facebook and other social media outlets.

Incidentally, FAIR is not the only group working on turning pictures into recipes. From the renowned Massachusetts Institute of Technology (MIT) comes a similar idea. Currently researchers there have primarily focused their attention on tasty desserts because compared to more ambiguous foods like for example sushi rolls or smoothies, their ingredients are easier to determine. Besides analyzing photos of various foods to guess their ingredients, MIT’s Computer Science and Artificial intelligence Laboratory (CSAIL)  is aiming somewhat higher on the ladder of scientific significance: “Seemingly useless photos on social media can actually provide valuable insight into health habits and dietary preferences,” as MIT Professor Antonio Torralba declares.

Living in an interactively empowered social media world, the CSAIL research team actually has a trial website, where you can go and try to determine some food ingredients by uploading your food pictures. It’s called Pic2Recipe. If you are lucky and your dishes appear close enough in appearance to those in the CSAIL AI database, it will return a printable recipe.

To develop their dish-identifying AI algorithm, the MIT research group visited some popular food-related websites such as All Recipes and to assemble a database of more than a million recipes, called Recipe1M. They annotated this collection with information about their known ingredients. Then they presented this information to a neural network and trained it to logically connect pictures and ingredients. The final step was to present their AI database with pictures of similar dishes and let it determine their ingredients. This, by the way, is pretty much how most AI systems work, if only on other, somewhat larger-scale problems and tasks.

All this, of course, exciting as it seems, is only a modest beginning. Accuracy scores are mostly still in the lower 60 to 70 percentile ranges. But there are more and more research teams, academic as well as commercial, around the world, working on the problem of analyzing recipes with AI. The databases required to analyze myriads of dishes from all over the world (sometimes containing exotic ingredients) have to be enlarged considerably. Datasets must include individual tastes and regional styles of preparation to be able to reliably recognize many various flavors. So there is still some way to go. But a foretaste of things to come is already noticeable.


Varsha Shivam

Varsha Shivam

Varsha Shivam is Marketing Manager at Arago and currently responsible for event planning and social media activities. She joined the company in 2014 after graduating from the Johannes Gutenberg University of Mainz with a Master’s Degree in American Studies, English Linguistics and Business Administration. During her studies, she worked as a Marketing & Sales intern at IBM and Bosch Software Innovations in Singapore.

View all posts by