The idea of installing software updates on your fridge already sounds strange, let alone focusing on improving its AI capabilities. But that’s exactly what’s happening with Samsung’s line of bespoke refrigerators this week, and I’m amazed that this patch is making major progress in delivering actually useful machine learning to modern iceboxes.
As a quick recap, Samsung has already offered AI-powered features like automatic food identification and meal planning on its bespoke refrigerators for a few years. However, as I found out after reviewing its flagship model late last year, the company’s AI capabilities are still very much a work in progress. Previously, the fridge could recognize about 60 different types of fresh foods (like fruits and vegetables) as well as 50 or more packaged items like yogurt or popcorn. It seemed like a good start, but given the sheer number of different items you’d find at a typical grocery store, it was far from complete. Plus, you often have to input additional data like the number of items or when something was first added, making the idea of AI-assisted grocery tracking more difficult than I expected. I don’t know about you, but I don’t generally want to type on my fridge and I’m pretty sure Samsung’s engineers agree, which is probably what drove this major update to its bespoke refrigerator software.
So what’s new?
The big change is that Samsung is adding support for Google Gemini, which has several important implications. By combining Samsung’s existing on-device object detection with Google’s cloud-based model, the total number of recognizable food items is growing from just over 100 items to more than 2,000. Now this does mean you’ll need to connect the fridge to Wi-Fi, but considering it supports other smart features like calendar integration and video playback, that’s not a big ask.
Another update is that Samsung is using Gemini to expand voice controls, allowing users to ask the fridge to change things like device settings, check details like when the water filter was last changed or even help troubleshoot problems. And depending on the situation, the fridge may even run a tutorial on how to solve the problem.
Alternatively, for more complex or difficult-to-solve problems, Samsung is introducing Reliability AI, designed to monitor the fridge’s components or help identify faults before they become too serious. Or in cases where the device needs service, AI could provide agents with more detailed information while potentially allowing them to fix certain things remotely.
For example, a Samsung representative told me that if a customer calls and says the cubes from the icemaker are coming out in clumps and stuck together, Reliability could allow AI agents to reduce the amount of water added to the ice tray — all without the need to physically come to your home. Crucially, Samsung says that while the fridge will monitor and track device health metrics, owners will need to provide explicit consent for repair personnel to access that data. Meanwhile, if a problem requires in-person servicing, Samsung says that by sharing this data with repair technicians, it allows people to identify and resolve problems faster instead of having to arrive without any context and diagnose problems from scratch.
How it works in the real world
My test unit hasn’t suffered any mechanical problems in the eight months I’ve used it, so I haven’t been able to evaluate Samsung’s Repairability AI. That said, after first checking it out at Samsung’s headquarters, I’ve had a chance to use an early version of the Bespoke line’s new software over the past two weeks — including its advanced support for cloud-based object detection — and the improvements are profound. Even after using it for a while, I’m still amazed at how many different foods it recognizes. Deep in the back of my fridge, I have a can of Bull’s Head Shallot Sauce, a typical Taiwanese ingredient used almost exclusively in Asian cuisine. However, the AI had no trouble identifying it, automatically tagging it and including it when it was first added to the fridge’s AI Food Manager.
Additionally, the system is now much better at recognizing brands and counting the number of specific ingredients to create a more detailed list. It could distinguish between Diet Coke and Coke Zero and also accurately note whether there were multiples of each item. And even though the fridge often has to ping a cloud-based server for help identifying various items, results appear very quickly, often in less than a few seconds. And for some foods, like avocado, I was pleased that the fridge tracks how long you’ve had it and will bring up a notification that it may be close to expiration. Granted it’s not always perfect, but I really need a reminder to check things and this does just that.
Sam Rutherford for Engadget
I’ve also noticed that the fridge now remembers when you repeatedly take out a specific food and then asks if you want to add that item to your shopping list. It’s a good reminder to refill the staples you use regularly and it’s in a low-abrasive way, so it doesn’t get annoying. From there, you can simply check your phone while you’re at the store instead of needing to manually prepare a list each week. Plus, because the fridge does a better job of identifying and tracking what’s inside, it can make better suggestions about recipes you can cook using ingredients you already have.
That said, like many existing models, the AI doesn’t always get every detail accurate. For example, I was initially impressed when it automatically labeled a tub of imitation cream cheese as “Philadelphia Plant-Based,” until I realized that the label was incomplete and the AI was only reading what was written on the lid and not smart enough to accurately complete the description. Don’t get me wrong, when I look at the Food Manager it provides more than enough information to help me figure out what’s in the fridge. It’s just not in the right place.
Outlook and things that still need work
One problem with this update is that, like many AI services today, Samsung’s new software can be a little overconfident or prone to hallucinations. Once, when my wife was putting something back in the fridge, the algorithm took a picture of a brightly colored strip on her finger and labeled it as a vegetable, which it actually is not. Other times it appears it is just a guess. But I would argue that going from about 100 identifiable objects to over 2,000 is a very welcome improvement, even with the limitations.
The other strange thing is that even though Samsung is leveraging Google’s AI models for many of the fridge’s new features, you won’t see any explicit callouts for Gemini inside the device. It’s kind of strange because Bixby is still the only digital assistant you can use and talk to directly.
I’ve said before that Samsung’s AI food identification is a work in progress and I think that’s still true. With this latest update, the company actually gets closer to delivering on the promise of a fridge with useful AI-powered features. What once seemed like a promising tech demo has quickly become a useful tool for keeping track of your groceries, even with a few hiccups here and there.
<a href