Episode 270: Pulling at Tightly Woven Threads

September 30, 2025

Central Thesis

Jim argues that uncritical acceptance of AI, particularly LLMs, poses a significant threat to human autonomy and social structures, especially due to the monopolistic control of these technologies and their potential for thought control. He is unsettled by the prospect of quoting language from a machine that lacks a body, seeing it as fundamentally different from quoting human sources.

Key Arguments

• Machine vs. Human Output: Jim believes quoting requires a "body" behind the words. LLMs, being "automated simulacra of language," don't qualify. He initially refuses to quote LLMs because of this philosophical distinction, seeing them as akin to instruments like speedometers, not sources of thought.

• LLMs as Censored Monsters: KMO counters that LLMs are not simple instruments. They are complex systems trained on vast amounts of human data, including the "nastiest, awfulest, worst things," and require extensive censorship. This suggests a more nuanced view of LLMs as potentially mirroring the complexities (and darkness) of the human mind.

• Monopoly Power and Control: Jim fears that the rapid development and deployment of AI is driven by monopolistic corporations (Alphabet/Google, Meta/Facebook, Amazon), potentially leading to an authoritarian future. He views AI's ownership as a more pressing concern than its mere existence.

• Economic Disruption and Redundancy: The conversation addresses the potential for AI to displace human labor, leading to mass redundancy and a society of renters. Jim worries about a growing divide between those who own the AI and those who don't, potentially triggering social unrest. KMO points out, the quicker AI transition the better, to avoid individuals being blamed for systemic job loss.

• Threat to Human Autonomy: Jim and KMO acknowledge the potential for AI to be used for thought control and policing speech, particularly through built-in red lines and censorship mechanisms. This raises concerns about the erosion of human autonomy and the manipulation of public discourse.

Notable Passages

• "A quote is to indicate some body. A machine is not a body."
• "It is very much like a human mind. Now, it can't reproduce all human capabilities and exceed all humans in those capabilities in all cases. Yet, but it's so much better in 2025 than it was in 2023 that, you know, it would be silly to make definitive statements about the capabilities or the character of AI in general based on the capabilities of, say, Claude 3.5 or GPT-5 in September of 2025."
• "We are going to be used when they are deployed, hopefully for some profitable function in ways that generate revenue for the owner. They're not going to be there for the people to build stuff that may threaten the owner's predominance over to society."
• "There is a new locus of agency coming into existence. And we're gonna have to either come to some agreement to say, no, we're not gonna let it happen and that just doesn't seem to be in the cards, or we're gonna have to figure out how to accommodate it. You know, how to live with it. Or not. If, if it, you know, if it deigns to let us live with it."

Rhetorical Approach

Jim uses a blend of personal anecdote, philosophical reasoning, and historical analogies to build his case. He starts with a personal dilemma about quoting LLMs, then broadens the discussion to encompass the societal implications of AI. He references historical events like the French Revolution and the rise of fascism to illustrate the dangers of unchecked power and economic inequality. He also makes cynical observations about the motivations of monopolistic corporations.

Connections

• Ezra Klein and Derek Thompson's Abundance: Mentioned in the introduction, the essay from a large language model prompts the initial debate on quoting machines.
• John Maynard Keynes: Reference to his essay, "Economic Possibilities for Our Grandchildren," is mentioned as an counterpoint to Jim's concerns.
• Upton Sinclair: Jim failed to remember the 1934 quote from Upton Sinclair, "It is difficult to get a man to understand something when his salary depends upon his not understanding it"
• David Graeber: Jim invokes Graeber’s concept of "bullshit jobs" to argue that technology and automated production have already advanced to a point where most people's labor is not necessary.
• Leonard Shlain's The Alphabet Versus the Goddess: KMO references this book to draw parallels between the societal upheaval caused by the printing press and the potential for chaos resulting from the rise of AI.
• Blake Lemoine: Jim references Blake Lemoine, the Google employee who claimed their language model was sentient, to highlight the dangers of anthropomorphizing AI and the limited safety concerns of tech companies.
• Noam Chomsky: Jim references to Noam Chomsky’s concept of "manufacturing consent."