AI Should Augment People, Not Replace Them, But the Technology Isn't There Yet

Introduction

There's an interesting policy idea making its rounds lately: a South Korean politician's proposal for an "AI dividend," modeled on Norway's sovereign wealth fund or Alaska's Permanent Fund. As a Bloomberg piece published this week pointed out, AI companies have built their foundation models on a kind of common resource — the collective data and creativity of billions of people — and the gains from that ought to be distributed back, particularly because the technology built on it will displace the labor of those same people.

Independent of the feasibility of such a policy, it's interesting to consider what the idea presumes. Even at the level of serious policy debate, the working assumption has been that AI's primary near-term economic effect will be displacement substantial enough to require redistribution. And the CEOs of AI companies themselves have been some of the loudest voices for such policy remedies, e.g., universal basic income (UBI), AI dividends, redistribution mechanisms. The people most bullish on AI are also the ones most actively planning for the labor displacement it will cause.

But AI bulls are also quick to point to Jevons' paradox and how the creation of the Internet spurred the creation of new jobs that never would have existed, such as social media managers or website designers.

So these days, the conversation about AI tends to split along two extremes. On one hand, AI will aid new scientific discoveries, solve climate change, cure cancer, etc. On the other hand, AI will displace most knowledge workers, rendering engineers, accountants, analysts obsolete. The literature on what's actually happening in workplaces today is more varied than either extreme suggests:

Thus far, there's been little evidence that AI can replace those who do purely mechanical work at scale. In fact, in February 2024, Klarna laid off around 700 customer support workers, replacing them with an AI customer support agent. By mid-2025, Klarna CEO Sebastian Siemiatkowski admitted that customer satisfaction had dropped, with complaints spiking, and reversed course, rehiring human staff.

So the reconciliation between two extremes boils down to this question: Will AI primarily be used for augmentation or substitution?

• On the enterprise side, the trend has largely been toward substitution. AI is mentioned on roughly 40% of S&P 500 earnings calls — and the framing skews heavily toward cost-cutting. Salesforce, Shopify, and others have publicly tied AI adoption to headcount decisions: Tobi Lütke's internal memo at Shopify now requires teams to demonstrate that AI can't do the work before requesting new hires.
• At the individual level, AI has been broadly augmentative and empowering, particularly for those who invest in learning how to use it well, enabling entrepreneurs, coding novices, and others to do things they couldn't have done alone. Even there, the benefit isn't uniformly distributed; it accrues disproportionately to those with higher "AI literacy," who can evaluate outputs and direct the tool skillfully.
• Anthropic's Economic Index (March 2026) reinforces this split: Individual use on Claude.ai is overwhelmingly biased toward augmentation. People use AI to think with, learn from, iterate against. Enterprise use is automation-dominant. Businesses deploy AI to execute defined workflows and tasks.

So why is AI being deployed as an augmentation technology at the individual level but as a substitution technology at the institutional level?

Note: "Substitution" has been purposefully chosen here as a softer version of "replacement" in order to engage with the objective technological & structural constraints independently of the more subjective topic of morality.

Why Enterprises Default to Substitution

The obvious explanation for the split is structural incentives. Enterprises operate under quarterly ROI cycles, board pressure to demonstrate returns on AI investment, and competitive pressure to match peers who are doing the same.

• Cost reduction is visible on next quarter's P&L; top-line growth from AI materializes implicitly and marginally over years. Anyone in sales knows this already: it's much easier to sell something that cuts costs by a tangible amount than a tool that could in theory make a business more productive.
• Substitution is easier to specify than augmentation. Replacing an already-defined task is straightforward; augmenting a knowledge worker requires understanding their vision or goal — putting to paper something that hasn't been specified yet.
• Competitive pressure compounds. Companies are expected to spend on AI in line with peers and demonstrate AI-enabled returns. Once one firm shows cost reduction, the rest match or lose margin.

So companies choose substitution, forced into deploying AI in a way that produces immediately visible returns. AI then gets evaluated on what it produces under substitution deployments, which guarantees that substitution is what gets refined and scaled. How we deploy AI in turn shapes what AI solutions come to market. The substitution paradigm becomes self-reinforcing.

If AI were capable of producing real institutional value creation — the way it produces individual-level augmentation — the incentive structure would shift. Cost-cutting dominates because it's what AI is most easily capable of today. Augmentation at the enterprise level is harder to justify partly because the technology doesn't yet support it well.

Why Augmentation Requires Continual Learning

Substitutive deployment isn't necessarily a moral failing, but rather a direct consequence of a specific technical gap: No enterprise AI deployment today is capable of continual learning, but rather starts from scratch every time. The chatbot that you talked to on Monday is the same chatbot that you talk to on Friday: it hasn't learned anything about your business, your customers, your preferences, your edge cases in between.

Substitution works fine with today's forgetful ("stateless") AI, because the unit of value is executing discrete tasks. Augmentation can't work without accumulated context because the unit of value is helping a specific employee/team/company do their work.

Without the capability of continual learning, "enterprise augmentation" has to fall back into AI-assisted search & retrieval.

AI memory has been a persistent topic in AI researcher and developer circles for years now. As we haven't been able to master true continual learning, today's memory approaches are largely patchwork in nature: retrieval-augmented generation (RAG), context dumping, agent memory harnesses, etc.

These are all attempts to simulate continual learning without actually solving it. They keep the underlying model frozen, and pile context on top. This is akin to preparing a briefing for a new hire before they do the job. No matter how you format the briefing, the new hire will never become an experienced employee until they're able to learn continually and internalize the knowledge.

In other words, the technological paradigm shift that would actually produce enterprise-level value creation hasn't happened yet.

What Is Continual Learning?

Continual learning is the capacity for a system to update what it knows and how it reasons based on accumulated experience in real or close-to-real time. The model on day 365 isn't the same model that was deployed on day 1. It has absorbed something specific about this business's decisions, its edge cases, its way of working — and reasons or behaves differently because of it. This encompasses a couple of capabilities:

The unifying property is that the system is structurally different over time in ways that reflect what it has learned there, not loaded with more context or given access to more documents. Different. The way a senior employee on day 365 is a different reasoner than they were on day 1, not because they have more files on their desk, but because their judgment has been shaped by everything they've seen.

This is what makes AI compound at the firm level rather than at the foundation model layer. And nothing currently deployed at scale actually does this yet.

Why Today's Memory Patchwork Isn't "Continual Learning"

The AI industry has converged on a set of approaches that get marketed under the umbrella of "AI memory" or "personalization": retrieval-augmented generation (RAG), persistent memory systems, longer context windows, agentic memory harnesses. These are engineering feats and produce real near-term value, but they're also not continual learning.

Each one is a variation on the same workaround: Keep the underlying model frozen, while finding cleverer ways to inject context at query time. Specifically:

• RAG retrieves relevant documents and dumps them into context. The model uses the retrieved text to answer the immediate question, but doesn't learn from it. Run the same query again, and the model still needs to look up the answer the second time, just retrieving the same documents again. This is closer to search-with-summarization than learning.
• Memory systems persist user state across sessions. Tools like ChatGPT's memory or Claude's projects save facts about the user — preferences, ongoing context, prior conversations — and load them at the start of each session. The model reads these and reasons from them, but the model itself hasn't changed. This isn't learning, it's just being handed a revised or longer note before it starts.
• Longer context windows let you stuff more in upfront. Some argue that as context windows expand, this functionally substitutes for learning — you can just include the entire relevant history each time. (Claude, Gemini, GPT all have ~1M token context now vs. ~100K tokens in 2023). But this conflates access with learning. The model still doesn't update based on what it processed. The window resets at the next session, and the model returns to baseline. A worker who re-read your entire company handbook before every conversation wouldn't be learning your business; they'd just be a fast reader.
• Fine-tuning updates the model itself, but discretely and expensively. Training a custom fine-tune on enterprise data costs tens of thousands to hundreds of thousands of dollars per run, takes engineering time most companies don't have, and produces a static model snapshot that still needs to be updated continually. To approximate continuous learning, you'd need to retrain on a rolling basis, which almost no one does. It's continual learning the way a quarterly software release is "continuous" deployment.

The pattern is consistent. Every current approach keeps the model itself static and compensates with infrastructure around it. Better retrieval. Better memory. Better prompting. The model on day 365 is the same model that shipped on day 1, just with more elaborate scaffolding attached.

This isn't a failure of execution. The patchwork is the rational response to current architectural constraints — foundation models aren't built to update at inference time, weights are expensive to modify, and the alternative (deploying a different model per customer) doesn't scale. So the industry has done what engineering teams always do when the underlying technology doesn't fit the use case: build clever middleware.

But middleware doesn't substitute for the missing capability. The patchwork solutions are being marketed as continual learning, but they aren't. And the marketing is convincing enough that the real gap has stopped feeling urgent, even as it continues to limit enterprises to substitution rather than augmentation.

What Continual Learning Could Enable

AI that compounds would actually internalize feedback and adapt over time. Imagine the strategy team's AI on month six of working with them — it would have absorbed which arguments leadership finds compelling, which considerations get weight in decisions, where the team has been burned before. It would recognize what good looks like at this company, not because someone wrote it down, but because the system has been corrected enough times to develop instincts. Style, tone, taste, judgment about what matters — these would converge over time. Over months, it would stop making the same class of mistakes and start anticipating the considerations its users actually weight. The compounding isn't in stored data; it's in how the system actually reasons.

This is what real continual learning would unlock: AI as an asset that gets more valuable through engagement with a specific context, rather than a tool that resets each session. Investing time to interact with AI actually pays off. This enables augmentation at the institutional level, because the system actually understands your goals, your constraints, your priorities to truly collaborate with you.

The full picture isn't a binary between substitution and augmentation. It's a spectrum, and each tier asks more of the underlying technology than the one before:

We're seeing early versions of tiers 3 and 4 emerge with today's patchwork solutions, mostly at the individual or consumer level. Tier 5, the most durable form of AI value creation, remains structurally out of reach without continual learning.

The paradigm shift hasn't happened, partly because the patchwork solutions have made it look like we're already there. And the gap between where the technology actually is and where companies are being told to deploy it has consequences. Boards demand AI returns; CEOs commit to AI roadmaps; teams are restructured around AI capabilities that don't yet exist in the form being sold. The deployments that result are shaped less by what AI can genuinely do well than by what produces a defensible number on next quarter's slide. Substitution wins by default — not because it's the best application of AI, but because it's the only use case that fits the gap between what AI can do and the returns that companies and investors demand.

The technical problem isn't unsolvable. It's that the current research and deployment trajectory has routed around it. The patchwork is fundable, deployable, and produces visible near-term ROI. Real continual learning requires architectural work whose payoff could be years out and whose research path doesn't fit the current commercial incentive structure.

The labor pressure, then, is being driven by a misread of what AI currently does — Klarna itself reversed course on headcount cuts within a year. The urgency to replace is real, but the basis for it is shakier than the discourse suggests. And the version of AI that everyone keeps promising — the one that genuinely drives innovation and growth as a true collaborator — depends on the same missing capability. This leaves us in a strange "dark age" of AI: not because AI isn't impressive, but because the version that would actually deliver on its promise hasn't been built. The grand applications keep getting promised because they're plausible, but fail to materialize because the technology that would deliver them doesn't yet exist.