AMD talks Ryzen 9000 controversy: 'We were as puzzled' as reviewers

AMD’s hotly anticipated Ryzen 9000 desktop processors launched earlier this month, and well, the gaming performance improvements seen by independent reviewers weren’t as hot as expected – or as hot as AMD attested to in early marketing statements. So what happened?

It’s been a long, winding road as reviewers and AMD alike hunted down answers. Last night, AMD released a community post pointing the finger at a confluence of various issues: A difference in the Windows mode used for testing, VBS security settings, how rival Intel systems were configured, and – as ever – the specific games benchmarked.

Phew. Today, David McAfee (who leads AMD’s client channel segment) joined us for a special edition of The Full Nerd to untangle the mess and explain exactly what happened.

As you read the blog post (and listen to David’s chat), it becomes clear: A big part of this stems from how AMD tested versus how reviewers tested, and not just the games chosen. Here’s a tidbit from AMD’s post:

“The ‘Zen 5’ architecture incorporates a wider branch prediction capacity than prior ‘Zen’ generations. Our automated test methodology was run in ‘Admin’ mode which produced results that reflect branch prediction code optimizations not present in the version of Windows reviewers used to test Ryzen 9000 Series. We have a further update on accessing this performance for users below.”

PC users will be able to tap into those performance improvements when Windows 11’s annual feature update, dubbed “24H2” for now, starts to roll out later this year – its branch prediction optimizations mimic the Super Admin changes AMD tested with. (The always-excellent Wendell of Level1Techs already has a nice and nerdy analysis video about the patch, which is available to Windows 11 Insider preview testers.)

Update: Hardware Unboxed tested the Ryzen 7 9700X across 40 games with Windows 11 24H2 preview installed and found truly massive gaming gains — an average of 10 percent over the performance of the current Windows build, with several games running 20 percent faster and a handful hitting blistering 30 percent performance improvements. It’s a truly wild, jaw-dropping performance improvement for an operating system upgrade. Read more about it here.

But nobody should be using Super Admin mode to run games, and McAfee stressed that in the interview. Why does AMD do so, and why were the differences so pronounced this generation? It’s all about AMD’s established, automated testing framework, which runs in Super Admin mode and is necessary to be able to tests products at scale across multiple hardware configurations. It created a blind spot, McAfee said.

“Historically, when our automation framework was built, the difference between performance in Super Admin mode and what you’d consider user mode when testing was negligible – there was very little difference there. That has changed over time, and quite honestly, it was a change we were blind to. When we collected this data, we didn’t see that difference because for several generations we were using that same framework to collect data through out automation suite. So clearly an oversight on our part, clearly something where we were a bit disconnected from how reviewers were testing these applications and how users were playing games on their systems. And that’s been corrected in the way that we collect data for our products going forward.”

The change that AMD was blind to is part of what bit its initial marketing numbers for Ryzen 9000.

“What we tested for Computex was running current shipping build of Windows, 23H2, but it was run in the Super Admin mode that bypasses a number of the security layers in Windows and gets closer to the bare metal performance. It’s a case of an oversight of how much a performance delta there was in those two operating modes.”

Willis Lai/Foundry

Willis Lai/Foundry

Willis Lai/Foundry

McAfee – and AMD’s community post – also touches on how game selection and even game scene selection can influence benchmark results, a fact alluded to by Hardware Unboxed’s continued excellent coverage of this saga. Whether you use a built-in benchmark or manually run through in-game scenes for testing matters, as McAfee elaborated.

“Even within an individual game, the balance of the system where portions of a game which lean heavier on the CPU versus lean heavier on the GPU result in massive differences in relative performance between product A and product B. I think you can look at many games where you can look at significant differences between the two products, and move to another scene where you effectively get clarity between the two.”

You can hear McAfee talk much more about how AMD chose its Ryzen 9000 gaming benchmarks, including how the company was surprised by some reviewer tests this generation, around the 30 minute mark of the interview. It’s a fascinating listen – you can tell there’s clear thought and intent going into the process, and it’s simply not a case of “benchmark games that show us in the best light,” as some community members have been implying across the web.

Willis Lai / Foundry

Willis Lai / Foundry

Willis Lai / Foundry

One final bit to touch on before I encourage you to go watch the full, genuinely insightful interview. Towards the end, my cohost Gordon Mah Ung asks simply: “Is AMD blaming reviewers here? Did hardware reviewers screw this up?”

“Not at all,” McAfee said.

“At the end of the day, there were a series of decisions that AMD made, that differed from how reviewers were testing, that led to different conclusions,” McAfee said. “This is not saying that reviewers did anything wrong, or that there’s anything in the process of how these reviews were done that was incorrect. We were as puzzled by some of the results that reviewers were seeing as the Ryzen fans are out there in the world, and that’s really what led us to this sort of multi-layered conclusion that there’s a number of factors going on here that created that separation.

This is no shade whatsoever on reviewers and how they tested, this is completely on things that we didn’t really understand as a part of the way that we tested and configured our products, and how that differed from both reviewer game suites moving in a slightly different direction, as well as how they tested the product versus how we did. It’s not a criticism of reviewers or how they test, simply significant differences in setup and configuration that got us from point A to point B and I think we understand that now.”

And you’ll also understand how a series of molehills turned into a mountain once you watch our full AMD Ryzen interview with David McAfee. This recap is just the tip of the iceberg – check out the entire talk for many more interesting details, such as why AMD configured Intel test systems the way they did, how Windows 11’s VBS feature affects all this, whether Windows 10 users will also get branch prediction optimizations, why AMD didn’t delay Ryzen 9000’s launch given this confusion (it makes sense!), core parking, and a whole lot more.

Don’t expect Ryzen 9000 chips to suddenly leap forward in gaming performance even under the best case scenarios — as McAfee said, independent reviewers did everything right here, and our numbers are out — but it’s good to understand where such a big disconnect stemmed from. This also serves as another reminder to take all vendor-supplied numbers with a big punch of Donnie Brasco-style salt.

Be sure to subscribe to our new, dedicated Full Nerd channel on YouTube for special interviews and nerdy PC podcasts like this week-in and week-out. We’ve split the podcast off from the main PCWorld channel so click that button if you don’t want to miss out!

Editor’s note: This article originally published on August 22, but was updated August 26 to include results from Hardware Unboxed’s Windows 11 24H2 testing.

Author: Brad Chacos, Executive Editor, PCWorld

Brad Chacos spends his days digging through desktop PCs and tweeting too much. He specializes in graphics cards and gaming, but covers everything from security to Windows tips and all manner of PC hardware.

Recent stories by Brad Chacos:

Best of Computex 2024: The most intriguing and innovative PC hardwareHow to check your PC’s CPU temperatureThe Full Nerd awards: Our favorite PC hardware of 2023