Context is sovereign

Positive Futures for AI, part 3

This presentation is a follow-on from:


Quality of AI

Graph showing the quality of AI by an array of different benchmarks

AI improving is not news. The quality of the outputs of AI models is going to carry on improving. It may eventually be bounded by the availability of high-quality training data, but even that is debatable. I’m pretty sure AI is going to be valuable for organisations and individuals alike.

Source: MiniMax


Graph showing the quality of AI by an array of different benchmarks


Investment

Private investment in AI

…so does most of the tech sector. That’s why major ICT companies are investing so heavily.

Source: 2024 AI Index Annual report


Private investment in AI


Private investment in Generative AI

Generative AI investment figures aren’t available for 2024-2025 yet, but I expect them to be higher than these 2023 figures.

Private investment in Generative AI by region


Not news

  • Quality and investment trends are not surprising
  • Growth is inevitable
  • Not about parameter count
  • Not about prompts

Graph showing the growth in context window size


What context means


What bigger context means

  • 1M token window
  • Equivalent to roughly 800,000 words
    • Shakespeare’s complete works
    • 2000 pages of text

Issues

Accuracy drops as context window size increases

Source: NVIDIA paper, What’s the Real Context Size of Your Long-Context Language Models


Meaningful improvement

![Accuracy drops as context window size increases](https://www.lightenna.com/assets/images/presentation_resources/2025-01-17/minimax-text-01-long-context.png)

Reaction to growth

“I can’t emphasize enough how mind-blowing extremely long token context windows are…”

Ted Xiao on X


“For both AI researchers and practitioners, massive context windows will have transformative long-term impact, beyond one or two flashy news cycles.”


AI now

  • AI is knowledgeable
  • AI has near instant recall
  • Scalable
  • Works 24/7
  • Not perfect, expensive, hallucinations
  • Constantly improving

2000 pages of…

  • Product briefs
  • Technical manuals
  • EPKs
  • Company blogs
  • Emails
  • Roadmaps
  • Design documents
  • Repositories of source code

What it means for business

  • Business-specific context
  • Up to now, AI has been a smart new hire
  • Long context allows AI to be a wise old-timer

Buy vs build

  • Can train your own models
    • but can you compete with the big players?
  • Can feed existing models with context
  • Can feed future models with context
  • Hype warning

“God I love AI hype.

Pump it straight into my veins 🤤”

Different-Froyo9497

This quote comes from a Reddit post on significance.


“God I love AI hype. Pump it straight into my veins and let me feel the singularity coming.”

AI auto-correct, Github Copilot/GPT-4


Challenges

  • Data quality
  • Data quantity
  • Relevance
  • Semantic processing
  • Information architecture
  • Business process integration

“The biggest single cost in adopting AI is the one you could have paid 10 years ago.”


Help

If you’d like help organising and preparing your digital strategy, working with your team to foster a data-driven open culture, or just hands-on-keyboard training to embed AI in your workflows, please get in touch.

View as a presentation

Leave a comment