Dive Temporary:
- AI mannequin efficiency improved considerably over the previous two years, based on the latest AI Index report from the Stanford Institute for Human-Centered AI. The analysis, schooling, trade and coverage group analyzed 29 benchmarks, evaluations and leaderboards to create the 400-plus web page examine.
- One benchmark evaluating a mannequin’s potential to resolve GitHub points from well-liked open-source Python repositories discovered the best-performing mannequin on the finish of 2023 scored 4.4%, whereas OpenAI’s o3, launched to researchers and builders in December, solved practically 72% of issues by early 2025.
- OpenAI’s o1, which was introduced in September, landed the highest spot in a multidiscipline job benchmark evaluating multimodal fashions on deliberate reasoning and college-level topic data. The o1 mannequin scored 4.4 factors beneath the human benchmark and 18.8 factors increased than final 12 months’s state-of-the-art rating.
Dive Perception:
AI mannequin prices, accessibility and different areas have room to develop at the same time as evaluation of mannequin efficiency suggests drastic enchancment.
The Stanford Institute for Human-Centered AI analysis discovered power effectivity has elevated by 40% every year, whereas {hardware} prices have declined by 30% yearly. Fashions are additionally turning into smaller and extra environment friendly. Microsoft’s 3.8 billion parameter Phi-3-mini scored increased than 60% on a broadly used benchmark the place the smallest mannequin to achieve the edge had 540 billion parameters.
Value and accessibility have moved to the forefront of standards enterprises are assessing in mannequin choices. China-based AI startup DeepSeek captured consideration earlier this 12 months when it claimed its R1 mannequin rivaled main U.S. fashions at a fraction of the coaching price, underlining the enterprise friction with current price constructions.
Responsible AI is another area the place CIOs are taking a better look. Researchers have created new benchmarks and sounded the alarm on poorly constructed assessments, based on Stanford’s evaluation.
The HELM Security, which supplies a complete analysis of language fashions, and AIR-Bench, which focuses on authorities laws, are two examples of benchmarks that consider fashions based mostly on accountable AI metrics. Anthropic’s Claude 3.5 Sonnet is taken into account the most secure within the HELM Security take a look at, adopted intently by OpenAI’s o1.
Analysts have cautioned CIOs in opposition to going all-in on one mannequin or vendor, and as a substitute beneficial striving for model-agnostic platforms because the tempo of innovation persists.
Expedia Group developed an internal experimentation platform with that in thoughts.
“We actually need to make certain we are able to make the most of the most recent, coolest mannequin,” Shiyi Pickrell, SVP of knowledge and AI at Expedia Group, advised CIO Dive. “A few of them have higher infrastructure or capabilities, so we constructed this generic layer permitting us to make use of totally different fashions based mostly on the use case or price.”
The general efficiency gaps between mannequin rivals have narrowed, too, underlining the necessity for flexibility.
There was an 11.9% efficiency hole between the very best and Tenth-ranked mannequin in a single evaluation included in Stanford’s AI Index report final 12 months. The distinction shrank to 5.4% this 12 months. In the meantime, the hole between the highest U.S. fashions and the very best Chinese language mannequin was 9.26% final 12 months and dwindled to 1.70% in a February evaluation.
