Why certain human roles are structurally permanent, and what philosophy can teach us about the limits of automation
Every few months, an AI system does something we thought required a human mind. It writes essays that pass university examinations. It predicts protein structures that took researchers decades to solve. It diagnoses disease from imaging with accuracy that matches specialist physicians. Each advance triggers the same anxious question, and each time the answer retreats a little further: first creativity, then empathy, then contextual judgment, then common sense, then "real understanding," whatever that means. The retreat looks orderly, but its direction is unmistakable. Follow the trajectory far enough and nothing remains.
But what if the trajectory is an illusion, produced by asking the wrong question?
The standard framing treats the human role as a bundle of capabilities: tasks we happen to perform, arranged on a spectrum from "easily automated" to "not yet automated." On this account, progress is a matter of time. AI improves, the frontier advances, the bundle shrinks. The horse-and-automobile analogy feels apt: we are the horses, perhaps not yet retired, but the automobile is gaining.
Five philosophers, working across different centuries and traditions, suggest this picture is fundamentally wrong. Their arguments converge on a conclusion that is at once reassuring and demanding: certain functions cannot be automated not because the technology is immature, but because automating them would destroy what those functions are for. The human role in an AI-enabled world is not transitional. It is structural. And beyond even these philosophical arguments lies a stranger truth: some of the most valuable things human beings do actually degrade when subjected to optimization.
I. Wittgenstein: When Automation Dissolves the Practice
Ludwig Wittgenstein argued that meaning does not reside in words themselves but in the practices that give words their life, what he called language games. "Check" means one thing in chess, another in a restaurant, another in a medical examination. The word is identical; the practice determines its significance.
Here is where the argument becomes interesting. Some practices survive mechanization perfectly well. Chess is the obvious case: a computer plays chess, and it is still chess, because chess just is the rule-governed manipulation of pieces on a board. The practice is fully constituted by its formal structure. Mechanize it and nothing is lost. Indeed, something is gained: the machine plays better chess than any human.
But other practices are not like chess. Consider what happens when a physician asks a patient, "Where does it hurt?" A pain-detection algorithm could, in principle, locate the neural signature with greater accuracy. Yet the question is not merely data collection. It is an entry into a relationship in which one person acknowledges another as someone whose suffering matters, in which the patient can say not just "level seven" but "it is the kind of pain that makes me afraid I am dying" or "it is bearable but I have not slept in three days." The practice of medical care is constituted not by information transfer alone but by the encounter between two persons, one of whom commits to attending to the other through illness. Automate it completely and you have not made medicine more efficient. You have replaced it with something else: data processing that wears the costume of care but has lost what made the practice medicine in the first place.
Think of it this way. A handshake is, mechanically, just two hands gripping each other. You could build a device that grips hands with precisely the right pressure and duration. But a handshake between two people sealing an agreement is not a mechanical event; it is a social act whose meaning depends on the fact that two agents are doing it voluntarily, looking each other in the eye, accepting mutual commitment. The device produces grip. It does not produce agreement.
This is Wittgenstein's contribution: some practices resist complete formalization not because we have not yet found the right algorithm, but because the algorithm, however perfect, would dissolve what makes the practice what it is. The meaning lives in the form of life, not in the information exchanged within it.
None of this prohibits AI assistance. Pattern recognition in imaging, pharmaceutical interaction checking, literature retrieval: these are tools that serve the practice without replacing it. The distinction is between augmenting a human practice and eliminating the human element that constitutes it.
II. Kant: The Origination of Ends
Immanuel Kant drew a distinction that matters more now than at any point since he formulated it in the eighteenth century: the distinction between theoretical reason and practical reason.
Theoretical reason figures out what is the case. It discovers patterns, draws inferences, predicts outcomes. Practical reason determines what ought to be the case. It sets goals, establishes values, decides what matters.
Consider a hiring system. An AI can learn from a decade of successful hires and identify the profile that predicts retention and performance. It can process thousands of applications, weighing hundreds of variables, with flawless consistency. This is theoretical reason operating at scale: given the objective (candidates resembling past successes) and the data (application materials), determine the optimal ranking.
But here is what the system cannot do. It cannot ask whether "candidates resembling past successes" is the right objective. Perhaps the past criteria perpetuated patterns that should not be reproduced. Perhaps the organization needs to change direction, requiring capabilities it has never valued before. Perhaps the very definition of "successful hire" should be reconsidered, measuring something other than retention and performance metrics.
These are not optimization problems within a framework. They are questions about which framework to adopt. They require what Kant called autonomy: the capacity to step back from any given set of objectives and ask whether those objectives are the right ones. AI, regardless of sophistication, operates within the space of theoretical reason. It optimizes brilliantly toward whatever target you specify. But it does not, and logically cannot, originate the target itself.
This is not a claim about current limitations. It is a claim about logical structure. Optimization presupposes a criterion. Someone has to supply the criterion before anything can optimize toward it. That someone occupies a position that is logically prior to the optimization, and that position is the domain of practical reason: the human capacity to set ends, not merely to pursue them.
The corporate executive who says "I am just following economic logic" has made exactly the error Kant identified: treating a principle of theoretical understanding (this is how markets behave) as a principle of practical reason (this is what we should do). Understanding how markets allocate resources does not tell you whether that allocation is desirable. That judgment requires stepping outside the model, which is precisely what the model cannot do.
III. Gödel: The Incompleteness of Every Formal System
In 1931, Kurt Gödel proved something that reverberates far beyond mathematics: no formal system powerful enough to express basic arithmetic can prove all the truths expressible within it. Every such system is incomplete. There are always true statements the system's own rules cannot reach.
The proof is technical, but the intuition is not. Imagine you are playing a board game with elaborate rules. You play long enough and a situation arises that the rules do not cover. Not because the game designers were careless, but because any finite set of rules, if it is powerful enough to generate interesting play, will inevitably produce situations that fall outside its own adjudication. You need a human being to look at the situation, look at the rules, and make a judgment about what the rules should say, a judgment the rules themselves cannot provide.
AI systems, however sophisticated, operate within formal structures. They follow learned patterns, optimize according to objective functions, apply rules (even if those rules were extracted from data rather than hand-coded) to generate outputs. Gödel demonstrated that any such system will encounter questions it cannot answer using its own logic. Not because the system is poorly designed. Because formal systems, as a matter of mathematical necessity, cannot be both consistent and complete.
This is why human oversight is not a temporary concession to immature technology. It is a structural requirement. Someone must be able to stand outside the automated system and ask: is this system still doing what we want it to do? Is its objective function still appropriate? Are its categories still carving reality at the joints? The system itself cannot perform this evaluation. That is not a bug. It is a theorem.
Together, Kant and Gödel establish complementary necessities. Kant shows that someone must set the objectives. Gödel shows that someone must evaluate whether the system pursuing those objectives remains adequate. Neither function can be absorbed into the system it governs. Both are permanently human.
IV. Von Foerster: The Preservation of Undecidability
Heinz von Foerster, working in cybernetics, took Gödel's insight and recast it as an ethical principle.
He distinguished two kinds of questions. Decidable questions have procedures: What is the shortest route between two cities? Which patients with these symptoms typically have this disease? What is the optimal portfolio allocation given these risk parameters? These questions, however complex, admit algorithmic solutions. AI excels at them, and should.
Undecidable questions have no procedure. Should I change careers? What kind of society do we want to build? When does mercy override the rules? Is this person trustworthy? These resist mechanization not because we lack sufficiently clever algorithms, but because no algorithm exists to be found. The answer depends on values, context, relationships, and choices about who we want to be.
Von Foerster's radical move was to refuse the conventional framing in which undecidability represents a problem. He saw it as a space for freedom. These are the questions where genuine agency operates, where we choose rather than calculate. His ethical imperative followed: act always to increase the number of choices.
Consider criminal sentencing. An algorithm could optimize for consistency, recidivism reduction, proportionality, and cost. It would almost certainly produce more uniform outcomes than human judges. But von Foerster's question is not whether the algorithm is more consistent. His question is whether we want to eliminate the undecidability of "what is justice in this specific case for this specific person." Is that not precisely the kind of question where we want human beings to struggle, deliberate, and ultimately choose, accepting responsibility for the choice?
When we force algorithmic solutions onto questions that should remain choices, we do not merely lose efficiency in some competing direction. We lose something about what it means to be an agent in the world. We convert questions about how to live into engineering problems. Von Foerster's insight: certain human roles are permanent not because we answer undecidable questions better than machines, but because these questions should remain spaces for decision rather than becoming targets for computation.
This is compatible with AI assistance. A judge may consult risk assessment tools, sentencing guidelines, statistical analyses of comparable cases. But the act of sentencing, the moment where a person says "I sentence you," must remain a human choice: an exercise of practical reason (Kant) on an undecidable question (von Foerster), subject to evaluation from outside the system (Gödel).
V. Luhmann: The Necessity of an Addressee
Niklas Luhmann studied how complex systems maintain legitimacy without transparency. His answer is deceptively simple: through attribution. We identify who is responsible for decisions, and we direct our expectations, questions, and challenges to them. Legitimacy requires not just good outcomes but an addressee, someone who can be asked "Why did you decide this?" and who can engage with the question.
The distinction is concrete. Your loan application is rejected. In one scenario, you speak to a person who explains: "Your debt-to-income ratio exceeded our threshold, and given current conditions we assessed the risk as too high." You may disagree. But you can respond: "What about my payment history over fifteen years?" You can argue for reconsideration. The officer can exercise judgment about whether the rules are being applied well in your case. The decision exists within a relationship of accountability.
In the other scenario, the algorithm rejects you. You call the bank. They say the AI system determined ineligibility based on thousands of factors, and the decision is final. There is no one to address. No one who can hear your counter-argument, weigh your particular circumstances against the general rule, accept responsibility for the outcome. You can debug an algorithm, retrain it, shut it off. But you cannot hold it accountable in the sense that matters: the sense in which it must stand behind its choices and answer for them.
Luhmann's insight illuminates why certain roles resist complete automation regardless of capability. A judge sentencing a defendant, a physician recommending treatment, a manager deciding between candidates: these are not merely information-processing tasks. They are social positions. The person occupying the position can be addressed, questioned, held to expectations, required to justify their reasoning, and potentially compelled to revise their judgment through dialogue. No algorithm occupies a social position in this sense, and no increase in capability will change that.
Again, the argument is not against AI assistance. The loan officer may use machine-generated risk scores as one input among several. The physician may rely on algorithmic pattern recognition in imaging. But the decision must be made by someone who inhabits the role of accountable agent, someone the affected person can confront and from whom they can demand reasons. Without this, the decision may be technically superior. It will not be legitimate.
VI. The Beauty Problem: Where Optimization Is Poison
Beyond the five philosophical arguments lies a limit on automation that is stranger and, in some respects, more fundamental.
Consider a system designed to maximize "fun" at a gathering. It analyzes thousands of social events, measuring laughter frequency, interaction density, activity engagement. It identifies the optimal playlist, the ideal ratio of structured activity to free conversation, the most productive guest mix. It produces the algorithmically perfect party.
Anyone who has attended both a spontaneous gathering and a meticulously engineered "fun-optimized" event knows what goes wrong. The optimization itself kills what it is trying to capture. Fun emerges from genuine spontaneity, from the unexpected, from the very inefficiencies and surprises that optimization systematically eliminates. You cannot engineer serendipity. The attempt is self-defeating.
This is not a problem of insufficient data or inadequate models. It is a structural feature of a whole class of human experiences and creations that are destroyed by the attempt to optimize them.
Art becomes formulaic when you try to maximize aesthetic value. Train a system on what sells, what wins prizes, what scores highest in audience ratings. It will produce technically proficient work that feels hollow, and the hollowness is not incidental. It is the direct consequence of the optimizing stance. Real art often breaks rules, surprises its creator, emerges from a vision that was not trying to hit a target. The moment artistic creation becomes target-hitting, it ceases to be art and becomes production.
Play becomes work when you try to make it efficient. Structure children's free play into "optimized learning activities" and you destroy what made play valuable: the autotelic quality of doing something purely for its own sake. Play has rules, but those rules serve the play itself. The moment they serve external objectives, play is over.
Beauty becomes mechanical when you try to formalize it. Kant, in the Critique of Judgment, identified the distinctive character of aesthetic experience as "purposiveness without purpose": we perceive the beautiful object as if it were designed for our appreciation, yet we cannot specify what purpose it serves. This quality is annihilated the moment beauty is made instrumental, the moment it becomes a metric to be optimized.
The pattern is general. Goodhart's Law captures it in economics: when a measure becomes a target, it ceases to be a good measure. The paradox of hedonism captures it in psychology: direct pursuit of happiness undermines it, while happiness emerges as a by-product of meaningful engagement. The problem of instrumental friendship captures it in ethics: friendship pursued for networking purposes is not friendship.
The implication for AI is severe. Even if a system could perfectly optimize for fun, beauty, or artistic value (even if it could hit every target with superhuman precision) the optimizing stance itself would destroy the thing being sought. These are not capabilities AI currently lacks. They are domains where the computational orientation is fundamentally wrong.
Human beings are capable of adopting non-instrumental stances. We can appreciate beauty for its own sake, create without optimizing for market value, play without maximizing learning outcomes. This capacity to value things as ends rather than as means may be as distinctively human as practical reason or social accountability. AI, by its nature, optimizes. For many domains, optimization is exactly right. For this class of experiences and creations, it is poison.
VII. Convergence: The Architecture of the Permanent Human Role
These six arguments (five philosophical, one aesthetic) converge on a single structure. The human role in an AI-enabled world is not a residual category of tasks machines have not yet mastered. It is an architecture of functions that require beings who can:
Participate in practices whose meaning depends on human relationship (Wittgenstein): where formalization would dissolve the practice itself, not merely alter its efficiency.
Originate ends through practical reason (Kant): setting goals and determining what ought to be pursued, a function logically prior to any optimization.
Evaluate formal systems from outside (Gödel): questioning whether the rules remain adequate, a judgment no system can perform on itself.
Exercise choice on undecidable questions (von Foerster): preserving spaces for genuine agency rather than converting all decisions into calculations.
Occupy addressable social positions (Luhmann): bearing responsibility in a sense that requires dialogue, justification, and the possibility of challenge.
Adopt non-instrumental stances (the beauty problem): engaging with experiences and creations that are destroyed by the optimizing orientation that defines computation.
None of these functions is a capability gap that future AI development will close. Each reflects a structural feature of legitimate decision-making, meaningful practice, or authentic experience. Together they define not the current boundary of automation (which shifts constantly) but its permanent horizon.
VIII. Implications
For regulation, the argument suggests a reorientation. "Meaningful human oversight" should not be defined in terms of which tasks humans currently perform better than AI. It should be defined in terms of which decisions require practical reason, addressable accountability, judgment on undecidable questions, the capacity to evaluate the system from outside, or non-instrumental engagement. Where these features are necessary, human involvement is structurally required, irrespective of how capable the technology becomes.
For organizations, the operative question shifts from "Can AI do this task?" to something more precise: Does this task require setting ends or merely finding means? Would full automation dissolve what makes this practice valuable? Do we need someone who can be addressed and held accountable? Is this an undecidable question that should remain a space for choice? Would optimization degrade the very quality we are trying to produce?
Where the answer to any of these is yes, human beings should be kept in meaningful decision-making roles, not residual oversight positions where they rubber-stamp algorithmic outputs.
For individuals, the implication is that competing with AI on AI's terms is a losing strategy and, more importantly, a misunderstanding of the situation. The capabilities worth cultivating are those the six arguments identify: the ability to set purposes through practical reason, the habit of questioning rules rather than merely following them, the judgment to recognize undecidable questions and preserve them as spaces for choice, the willingness to occupy positions of genuine accountability, and the capacity for non-instrumental engagement with beauty, art, and play.
The anxiety about replacement rests on a flawed model: a single spectrum of capability, with AI advancing toward the human position. The philosophical arguments suggest a different geometry. Humans and AI do not occupy the same axis. They inhabit fundamentally different positions in the landscape of possible intelligence. AI handles formalized inference with superhuman power. Humans occupy the positions where decisions are made about what to formalize, how to evaluate results, when to revise approaches, and when to resist optimization entirely.
The future is not humans against AI. It is humans and AI, each operating where they belong, with humans retaining authority over how far automation extends in any given domain, making that judgment on the basis of the structural considerations these philosophers have identified. AI will continue to improve at what it does: pattern recognition, optimization, consistent application of learned rules, processing at scales no human can match. Good. That is valuable.
But we need human beings in the positions where only persons can stand: originating purposes rather than merely optimizing, questioning the rules rather than executing them, preserving undecidability as freedom, engaging in practices whose meaning depends on relationship, occupying positions where they can be addressed and held accountable, and experiencing beauty, creating art, and playing genuinely, without converting these into targets for algorithmic improvement.
The headline is not that humans are becoming obsolete. It is that we are discovering what makes us irreplaceable, and it is not what we thought.