Recent community discussions have reignited the debate over whether Chinese tech giant Baidu may have developed the key theoretical groundwork for large-scale artificial intelligence (AI) models before America’s OpenAI.
Advertisement
Large models, or “foundation models”, are at the forefront of AI development, with their rapid iterations driving cutting-edge applications. While the United States is generally viewed as leading advanced AI model innovation, some argue that China may have started exploring these concepts earlier.
Central to large-model development is the “scaling law” – a principle that asserts the larger the training data and model parameters, the stronger the model’s intelligence capabilities. Widely credited to OpenAI’s 2020 paper, “Scaling Laws for Neural Language Models”, this idea has since become a cornerstone in AI research.
The OpenAI paper showed that increasing model parameters, training data and compute resources boosts performance following a power-law relationship. This insight guided the development of subsequent large-scale AI models.
However, Dario Amodei, a co-author of the OpenAI paper and former vice-president of research at the company, shared in a November podcast that he had observed similar phenomena as early as 2014, during his time at Baidu.
Advertisement
“When I was working at Baidu with [former Baidu chief scientist] Andrew Ng in late 2014, the first thing we worked on was speech recognition systems,” Amodei said. “I noticed that models improved as you gave them more data, made them larger and trained them longer.”