I Tried Amazon’s Prime Day AI Assistant and Found It Lacking

I Tried Amazon’s Prime Day AI Assistant and Found It Lacking

Just a few days before this year’s Prime Day sale, which is taking place July 16 and 17, Amazon rolled out a new app-only chat feature to all US customers to assist with their shopping.

This story is part of Amazon Prime Day, CNET’s guide to everything you need to know and how to find the best deals.

Called Rufus, the AI assistant has been trained on Amazon’s products, reviews and Q&As, along with information from the web. Its goal is to answer questions about your shopping needs and Amazon’s products, as well as to make product comparisons — and, of course, to encourage you to spend. But Rufus also warns: “My answers are powered by AI, so I may not always get things right.”

It’s refreshing to get that kind of caveat right up front. Generative AI software has been plagued by wrong answers and hallucinations. Google’s AI Overviews mishaps in search results a few weeks back weren’t an isolated problem.

In the last 18 months, we’ve seen the arrival of multiple generative AI assistants, including OpenAI’s ChatGPT, Google’s Gemini, Anthropic’s Claude and Adobe’s Firefly. Mistakes notwithstanding, they’re capable of extraordinary tasks, from writing A-plus essays to creating never-before-seen images. But e-commerce is one category where we haven’t seen as much activity, and I was curious how Amazon’s shopping-specific assistant stacked up.

I took Rufus for a test drive  — including asking for tips specific to Prime Day deals — and found an AI assistant that was eager to help answer questions beyond shopping. I also discovered that it was better at diving into details about specific products than responding to broader queries, which is somewhat curious given the bucket loads of data Amazon has on its 200 million Prime members in the US. Plus, the ease of use leaves something to be desired.

How to get started with Amazon Rufus

First off, you need the Amazon app. To use Rufus, look for the orange and blue icon in the bottom right-hand corner. Tap on it to ask Rufus a question by either typing your query or saying it out loud. Once you find products to purchase, you add them to your Amazon cart as usual.

Finding a beach read

I wanted to start off easy, so I went with the product category that started it all: books. Given the season, I asked: What is a good summer beach read?

As you can see in the image below, it seems like Rufus is relying on keywords to do the heavy lifting here. It suggested books like Beach Read, A Novel Summer and The Summer Place.

Amazon/Screenshot by Lisa Lacy/CNET

I was curious how Rufus compares to Amazon’s site search and Alexa, so I asked them about beach reads too.

When I made the same query on-site, I got a completely different set of recommendations than what Rufus served up, including Carley Fortune’s Every Summer After and Jan Moran’s Seabreeze Inn. 

And when I asked Alexa, she said, “There are no books matching ‘summer beach’ in your Audible library.”

Rufus’ response is almost comically keyword-driven, so I have to give the edge to the Amazon website on this one. 

I’ve reached out to Amazon for more insights into how Rufus does its job.

Shopping for a pet’s birthday

I have a puppy with a birthday coming up, so next I asked: What should I get for my dog’s first birthday?

Rufus said, “Celebrate your furry friend’s first birthday with special treats and toys! Look for durable chew toys, interactive puzzles and delicious dog treats made with high-quality ingredients.”

The response was less obviously keyword-driven, but it was still fairly generic. It included links to product categories, but those just took me to basic search pages on Amazon for those terms.

By comparison, Amazon’s on-site results seemed to be more focused on keywords here, offering up a selection of first-birthday bandanas and dog party packages with banners that say, “It’s my bday, motherpuppers!”

When I asked Alexa the same question, this was the answer: “On your dog’s first birthday, it is customary to give them a toy as a present. This can be a stuffed animal, a ball, or any other item that is fun for dogs to play with.”

Alexa also offered additional dog facts, including average lifespan (in case I wasn’t yet pondering my dog’s mortality).

Shopping on a Winnie the Pooh theme

My son has recently discovered the show Playdate with Winnie the Pooh on Disney Plus, and he’s obsessed, so I asked Rufus: What are the best Winnie the Pooh products on Amazon?

Amazon/Screenshot by Lisa Lacy/CNET

The response, shown above in part, wasn’t terrible. However, once you click on a suggested link, it’s hard to go back to the previous page — it’s like Rufus wants you to go further down a rabbit hole. 

When I asked the same question on-site and also put the same query to Alexa, I got a whole different range of Winnie the Pooh items, including different plush toys, a water bottle, a keychain, makeup and a backpack. So each platform seems to have a different idea about what “best” means.

Shopping for home improvement products

I recently finished a home improvement project, and since Rufus can supposedly make recommendations about the products you need for tasks like an at-home manicure, yoga and gardening, I decided to ask something I kind of already knew: What do I need to wallpaper a bathroom?

“When selecting wallpaper for a bathroom, prioritize moisture-resistant and washable materials like vinyl or non-woven fabrics. Look for high scrubbability ratings and mold/mildew resistance to withstand humidity,” Rufus said. 

It included nothing about paste, brushes, a straightedge or a box cutter in its first answer, but it did suggest that my next question should be, “What tools are needed for wallpaper installation?” And then it provided a better answer.

But again, when you click on those product listings, you’re taken to a generic Amazon search page, and it’s hard to go back once you’ve clicked through on the first product.

Alexa was also fairly helpful in terms of product recommendations for a wallpaper project.

They both had better answers than the website, which offered up a vinyl decal that said “Bathroom” in cursive, along with a bunch of signs that said, “You never know what you have until it’s gone,” alongside an image of an empty roll of toilet paper.

Is Rufus ready for Prime Day?

I have some bad news. I tried asking Rufus about the upcoming sales promotion in multiple ways, but the AI shopping assistant simply didn’t know what to say beyond the basics.

First, I asked: What are the best Prime Day deals for me?

Amazon's AI shopping assistant answers a question about Prime Day. Amazon's AI shopping assistant answers a question about Prime Day. Amazon/Screenshot by Lisa Lacy/CNET

As you can see above, Rufus gave me a general nonanswer about a wide variety of products being discounted. Curiously, when I asked this question on site, Amazon recommended nothing but clothing — including a tank top, linen pants and a linen shirt — and though I have bought many products from Amazon over the years, I’ve never bought clothing there, beyond socks for my son.

Alexa called out 10 results for deals, including two books — The Greatest Beer Run Ever and We Were the Lucky Ones — and a poster frame. I do actually buy a lot of frames from Amazon, so that latter one maybe actually made sense.

Looking for Prime Day deals

I tried to broaden the query a bit: What are the best Prime Day deals?

I got the same results as last time, from Rufus and Alexa. The Amazon site shook things up a bit with a toiletry bag, some storage bags — and those same linen pants.

I thought maybe asking about Prime Day deals as a whole was too big a challenge, given the scope, so I got more specific: Do you have any Prime Day deals on dog toys?

Alas, Rufus could only say:

“Prime Day is Amazon’s biggest global shopping event, exclusively for Amazon Prime members, and will take place on July 16 and 17. Head over to the Prime Day page to learn more.”

Alexa and the website called up some dog toys, but nothing was tied to a specific Prime Day deal.

I gave it one more shot: Will there be any Winnie the Pooh deals on Prime Day?

But again, Rufus said: “Prime Day is Amazon’s biggest global shopping event….” Alexa and the website called up some Winnie the Pooh products this time, but nothing was tied to a specific Prime Day deal — and even if these deals aren’t available yet, none of these tools were able to tell me that.

Given the amount of data Amazon has on us and our search and purchase behaviors, I’m surprised not to see something more personalized for Prime Day. After all, it’s kind of like a big garage sale, with a dizzying array of product deals. I know Amazon calls out some products for each of us on the homepage throughout the sale, but I feel like it would be helpful if Rufus or Alexa could help zero in on the products I’d be most interested in. 

Then again, maybe it’s like candy and magazines at the checkout register — Amazon wants you to browse and increase your basket size with the stuff you throw in your cart along the way.

Where does Amazon Rufus excel?

Amazon has some recommendations for using Rufus, which is where it’s most helpful. Those includes questions like: How do I prepare for Prime Day? When is Prime Day? Are Prime Day Deals just for members?

One of the suggested questions Rufus surfaced for me was: Which books are considered classics in literature?

Rufus called out To Kill a Mockingbird, Pride and Prejudice, The Great Gatsby, The Catcher in the Rye and 1984, which isn’t a bad list.

Then it suggested I ask about the themes that literary classics explore, or what defines a book as a literary classic.

And so Rufus’ strength, for now, probably lies in those suggested queries.

But Amazon says Rufus can tell you more about certain products, so I went back to the linen pants. Once on the product page, I asked:

Amazon/Screenshot by Lisa Lacy/CNET

Each response asks for feedback, so you can rate it with a thumbs-up or thumbs-down and then check options boxes like, “This is inaccurate” or “This is irrelevant.”

Amazon says this will help Rufus get better over time.

For now, I think Rufus is best once you have a specific product in mind rather than asking this new AI assistant to sort through the millions of products for sale on the site. Maybe by next Prime Day, it’ll be able to answer more-personalized queries.

Originally Appeared Here