AI customers crave speed more than price in high-bandwidth memory

This analysis is by Bloomberg Intelligence Senior Industry Analyst Masahiro Wakasugi. It appeared first on the Bloomberg Terminal.

High-bandwidth memory chips can add value with each new generation, boosting profits for SK Hynix, Micron Technology and Samsung Electronics, as data-transfer speed is a top competitive factor in artificial-intelligence computing. Many AI customers prize high speed more than low price in high bandwidth memory chips because time to market is critical

DRAM speed crucial to AI development

The performance of DRAM and HBM chips is key to development of artificial intelligence, so the added value that can be offered by DRAM, especially its HBM subset, might rise over the long term. DRAM data-transfer speeds haven’t kept up with the calculation speeds of central processing units.

Speed requirements for DRAM, and HBM in particular, may be especially high for AI customers, which means suppliers able to provide high-performance HBM chips can gain market share and increase profit. With each new generation of HBM, the speed — or bandwidth — will improve, and customers are likely to opt for faster HBM chips even if they have to pay higher prices.

Intel central processing units released in 2017 were 166x faster than the DDR4 DRAM that was the mainstream product at the time, according to Kyushu University.

DRAM serves as gating factor for CPUs

HBM chips can gain a competitive edge through technological advances like faster data-transfer speeds and may not face the level of price competition seen in PC DRAMs. Performance of CPUs and graphics processing units can improve rapidly as chip technology evolves.

Yet no matter how fast a CPU’s speed, calculations can’t be executed unless data is passed to it. DRAM and HBM chips, which handle data transfer to and from CPUs and GPUs, have lagged CPUs in improving speed — a gap that may continue to grow. SK Hynix is ​​ahead of peers in HBM chip development, but Micron is rapidly catching up.

We estimate the difference between CPU calculation speed and DRAM data-transfer speed has widened to about 220x in 2024.

Memory chips remain bottleneck in AI devices

Smartphones, PCs and servers in common use are von Neumann-type computers, in which the CPUs and GPUs calculate with data delivered from memory, including DRAM. CPUs and GPUs can’t perform at their best unless data arrives at the processor fast enough. DRAM speed might be a bottleneck not only in AI servers but also in edge AI devices such as AI smartphones and AI PCs.

The CPU of the Fugaku supercomputer processes at 72x the speed of its HBM, according to Kyushu University. With each successive generation of HBM, speed will improve significantly, adding value and boosting prices.

The data included in these materials are for illustrative purposes only. The BLOOMBERG TERMINAL service and Bloomberg data products (the “Services”) are owned and distributed by Bloomberg Finance L.P. (“BFLP”) except (i) in Argentina, Australia and certain jurisdictions in the Pacific Islands, Bermuda, China, India, Japan, Korea and New Zealand, where Bloomberg L.P. and its subsidiaries (“BLP”) distribute these products, and (ii) in Singapore and the jurisdictions serviced by Bloomberg’s Singapore office, where a subsidiary of BFLP distributes these products. BLP provides BFLP and its subsidiaries with global marketing and operational support and service. Certain features, functions, products and services are available only to sophisticated investors and only where permitted. BFLP, BLP and their affiliates do not guarantee the accuracy of prices or other information in the Services. Nothing in the Services shall constitute or be construed as an offering of financial instruments by BFLP, BLP or their affiliates, or as investment advice or recommendations by BFLP, BLP or their affiliates of an investment strategy or whether or not to “buy”, “sell” or “hold” an investment. Information available via the Services should not be considered as information sufficient upon which to base an investment decision. The following are trademarks and service marks of BFLP, a Delaware limited partnership, or its subsidiaries: BLOOMBERG, BLOOMBERG ANYWHERE, BLOOMBERG MARKETS, BLOOMBERG NEWS, BLOOMBERG PROFESSIONAL, BLOOMBERG TERMINAL and BLOOMBERG.COM. Absence of any trademark or service mark from this list does not waive Bloomberg’s intellectual property rights in that name, mark or logo. All rights reserved. © 2024 Bloomberg.