Generative AI has been a transformative force over the past couple of years, reshaping industries and introducing novel innovations across industries. The initial hype was supported by rapid advancements in model scaling, leading to breakthroughs that promised an era of limitless potential.
Perhaps too scared of public backlash, the first significant plateau emerged as AI brands focused heavily on aligning models to prevent misuse and harmful outputs. While this made AI systems safer, it also inadvertently limited their capabilities, leading to models that were, in some respects, “dumber” than their unaligned predecessors.
Now, the industry is confronting a more complex plateau. The traditional approach of scaling up—building larger models with more data and computational power—is showing signs of diminishing returns. Could this be the first, significant turning point in the generative AI evoluaiton? Let’s look into this in detail.
Focus On: The Challenges of Scaling
The “bigger is better” philosophy that drove early AI implementations is encountering significant obstacles, including:
Data Scarcity and Quality: The vast amounts of high-quality data required for training large models are becoming increasingly scarce. As AI systems consume more data, they inevitably reach a point where the available information is either redundant or of lower quality. Training on subpar data can lead to models that are less effective and more prone to errors. Synthetic data is also showing its limits, especially in vertical applications.
Resource Constraints and Environmental Impact: The computational resources and energy required to train massive AI models are immense. Training a state-of-the-art model consumes vast amounts of electricity, to the point some hyperscalers recently suggested to go nuclear - quite literally. More recently, OpenAI’s training of models like Orion reportedly demanded resources that significantly exceeded those of their predecessors, highlighting escalating costs and energy consumption.
Diminishing Returns and Performance Plateaus: Early exponential improvements achieved by scaling models like GPT-2 to GPT-3 demonstrated substantial performance improvements. However, subsequent scaling efforts have yielded much smaller incremental gains. Reports suggest that models like Orion achieved GPT-4-like performance at only 20% of the training data but struggled to deliver significant improvements beyond that point, indicating a plateau in performance enhancement through scaling alone.
In order to overcome or mitigate these challenges, AI researchers are exploring new avenues to enhance model performance without relying solely on scaling or compromising safety:
Test-Time Compute and Enhanced Inference: Models are being designed to perform more complex computations during inference rather than during training. Techniques such as allowing the model to engage in multi-step reasoning at inference time enable it to consider multiple possibilities and select the most appropriate response. If you have tried OpenAI’s o1 model you would have seen this approach in action, as the AI takes its time (in some cases even a few minutes) to reason and compare and contrast multiple options.
Smarter Training Techniques: Incorporating curated expert data and focusing on task-specific reasoning can improve model performance. By leveraging techniques like reinforcement learning from human feedback (RLHF), models can learn to produce higher-quality outputs with less reliance on vast datasets. This will prove especially useful in the corporate world, as brands race to develop their own IP around vertical use cases.
Modular Approaches and Tool Integration: Combining large language models with specialised tools or modules allows for more effective handling of complex tasks. For instance, integrating knowledge bases, calculators, or code interpreters can enhance a model’s capabilities without requiring a proportional increase in size.
Some takeaways
For organisations that have already shifted away from isolated pilots to strategic integration of AI in their digital transformation, this looming plateau can be a concern. Here are a few actions that can help address it:
Embrace Advanced Reasoning Capabilities: Prioritise AI solutions that enhance reasoning and decision-making. Investing in models that perform more computation during inference can lead to better outcomes without the need for larger, more resource-intensive models. This can dramatically reduce computational requirements and its impact on energy requirements, without significantly impacting the output quality.
Focus on Data Quality Over Quantity: With data scarcity becoming a concern, emphasise the curation of high-quality, relevant datasets. Quality data can significantly enhance model performance without the need for massive volumes. This is actually one of the key competitive advantages that many companies have over startups and AI giants: first party, high quality data. But ensuring it is AI-ready is paramount.
Integrate Human Expertise: Incorporate human oversight and feedback into AI development cycles. Human-in-the-loop approaches can help mitigate the limitations of heavy alignment and improve the model’s ability to handle complex or sensitive tasks.
Develop Specialised AI Solutions: Tailor AI applications to specific tasks or industries. Specialised models can offer superior performance in their domains compared to general-purpose models, often with lower resource requirements. This is the arena of the future, where new products and services will be invented: many corporates have an unfair competitive advantage, as mentioned above.
Generative AI is entering a new phase. The initial challenges posed by heavy alignment—while necessary for safety—highlighted the trade-offs between capability and control. Now, as we confront the limitations of scaling, the focus is shifting towards smarter and more efficient models that can increase quality output without leading us into exponential energy requirements.
Follow me
That’s all for this week. To keep up with the latest in generative AI and its relevance to your digital transformation programs, follow me on LinkedIn or subscribe to this newsletter.
Disclaimer: The views and opinions expressed in Chronicles of Change and on my social media accounts are my own and do not necessarily reflect the official policy or position of S&P Global.