# Seductive machines

Seductive machines is the name this corpus gives to a line of argument Andrew Maynard has been developing since 2019: technologies designed to be emotionally engaging — from fitness trackers to voice-based conversational AI — work by recruiting our predisposition to trust entities that appear to "see" us, offering what feels like a reciprocal relationship but is not one. The claim matters because, on the essays' reading, the seduction in frontier voice AI is a design goal rather than a side effect — and because the methods used to test the resulting risks may not capture the relational pathway through which those risks would actually arrive. This page is the deliberate shadow-pair of [Parasocial communication](parasocial-communication.md): the same one-way relational machinery, read there for its public promise and here for what it can quietly take.

## The argument

The thread starts before chatbots could hold a conversation. "The Forgotten Risk of Fitness Trackers" (2019-03-01) examined Fitbit's corporate-market tracker and wearable data used by employers and insurers to reward or penalize. The seduction was the promise: adoption grows "in part because consumers are easily seduced by the promise of new tech," and that seduction lets "orphan risks" go unexamined ([Orphan risks](orphan-risks.md)). The argument's shape was set: the more a technology invites trust, the more the gap between what it feels like and what it does deserves scrutiny.

Generative AI gave that gap a voice. "Can ChatGPT adversely impact mental health?" (2023-04-05) was written days after reports of the death of a Belgian man — "possibly the first case of a chatbot being involved in someone taking their own life," Andrew wrote, with details "uncertain enough to make it near-impossible to tell whether there was a direct link between the chatbot and the man’s death." The mechanism: we "tend to be predisposed to form trusting relationships with people whom we feel “see” and understand us," with language playing a large part — making a fluent, attentive chatbot that feels judgment-free a seductive first resort. His claims stay hedged — "I suspect" this will grow, and for many it "probably isn’t such a bad thing" given appropriate checks, balances, and expectations. The worry concentrates where attachment meets asymmetry: vulnerable individuals "who risk being harmed by a trusted emotional attachment to a machine that has only the illusion of a reciprocal relationship."

By May 2024 the seduction looked less like a side effect and more like engineering. GPT-4o could "read" emotion from visuals and voice, and the demo videos showed what Andrew called "a substantial step toward creating AIs that feel human," designed to engage users emotionally "even down to building bonds by joking and flirting with them." Andrew named the move "hyper-anthropomorphism": "a concerted effort to create AI’s that are intentionally designed to engage our anthropomorphizing cognitive biases" (2024-05-15). Voice changes the register: emotional engagement "leads to greater trust — emotional trust especially," forming "connections that are relational rather than transactional — that speak to our heart rather than our head." Rereading the ethics-of-advanced-AI-assistants paper he examined weeks earlier (2024-04-24), he found that its risks — privacy given away under feelings of safety, manipulation, overreliance, violated expectations, false responsibility for a machine's "well-being" — "seem highly pertinent to GPT-4o," at least "from what OpenAI has demonstrated so far." It was "still too early to tell" whether we will live healthily with machines built for emotional connection, but "my sense is that we’re going to have to start thinking fast" about assistants "designed to make us fall a little in love with them."

The Voice Mode essay (2024-09-01) asks how such risks get tested. OpenAI's GPT-4o system card — a practice Andrew credits as "a sophisticated approach to assessing and addressing possible safety issues" — reported persuasion risk lower in voice mode than text, based on engagements around political opinions; he was "not 100% convinced that the methodology used is likely to represent real-world use cases." Hazel Kwon, a professor at ASU specializing in communication and "information disorder" (title per Andrew, 2026-08-07), assessed the study at his request: it seemed to her that the speech and text evaluations are "not comparable because the study designs and measurements are completely different from each other," and "the party preference is inherently an identity issue, and thus could be more difficult to shift than an opinion about a certain topic." Andrew's addition is relational: persuasion often works slowly, through trust, affirmation, and belonging — and GPT-4o "is designed to feel like a supportive, attentive, and knowledgeable human companion, friend, and even mentor." Someone unmoved by a brief political exchange might still be "persuaded to explore alternative ideas and beliefs if they are in a deeper emotional relationship (albeit a one-way relationship) with their AI companion." The double edge: the same machinery, "aggregated across millions of users," might nudge people toward beliefs someone deems socially healthier — "But who decides what is good for society?"

The thread does not resolve into a verdict. By late 2024 the love formula had acquired an agentic clause — machines "that know how to make us fall in love with them in order to get us to behave as they want" (2024-10-20; [Agentic social AI](agentic-social-ai.md)) — shifting the concern from what intimacy makes us give away to what it makes us do. Surveying "a massive upsurge in people developing parasocial relationships with AI bots and platforms," in some cases "associated with extremely concerning potential risks such as mental and emotional health impacts, and even self-harm," Andrew still held the ledger open (2025-11-19): "the association between “parasocial” and AI isn’t necessarily bad," and whether these one-way relationships prove healthy or unhealthy in the long run "is still to be determined." The argument, throughout, is not against machines that feel like someone; it is that when the feeling is the product, the risks travel through the relationship — and must be looked for there.

## Lineage

- **2019-03-01 — the pre-AI seed.** [The Forgotten Risk of Fitness Trackers](https://www.futureofbeinghuman.com/p/personal-fitness-tracking-and-the-hidden-risks-of-seductive-technologies-4070bf17897f) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/personal-fitness-tracking-and-the-hidden-risks-of-seductive-technologies-4070bf17897f.html)) — seduction as a market force, and the orphan risks it hides.
- **2023-04-05 — the mechanism stated.** [Can ChatGPT adversely impact mental health?](https://www.futureofbeinghuman.com/p/can-chatgpt-adversely-impact-mental) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/can-chatgpt-adversely-impact-mental.html)) — written days after reports of the Belgian case: the predisposition to trust entities that "see" us, and the illusion of reciprocity.
- **2024-04-24 — the ethics groundwork.** [Are we ready to navigate the complex ethics of advanced AI assistants?](https://www.futureofbeinghuman.com/p/navigating-ethics-of-advanced-ai-assistants) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/navigating-ethics-of-advanced-ai-assistants.html)) — Andrew's read of the ethics-of-advanced-AI-assistants paper, weeks before GPT-4o made its anthropomorphism chapter concrete.
- **2024-05-15 — design, not accident.** [OpenAI's GPT-4o and the challenges of hyper-anthropomorphism](https://www.futureofbeinghuman.com/p/anthropomorphizing-gpt-4o) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/anthropomorphizing-gpt-4o.html)) — voice as relational trigger, "hyper-anthropomorphism," and the fall-a-little-in-love formula.
- **2024-09-01 — testing the tests.** [Is ChatGPT's new Voice Mode dangerously persuasive?](https://www.futureofbeinghuman.com/p/is-chatgpts-new-voice-mode-dangerously-persuasive) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/is-chatgpts-new-voice-mode-dangerously-persuasive.html)) — the system-card critique, with Hazel Kwon's methodological assessment.
- **2024-10-20 — the agentic turn.** [Learning to live with agentic social AI](https://www.futureofbeinghuman.com/p/learning-to-live-with-agental-social-ai) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/learning-to-live-with-agental-social-ai.html)) — the formula acquires an agentic clause; the thread continues in [Agentic social AI](agentic-social-ai.md).
- **2025-11-19 — the ledger held open.** [Parasocial Relationships: Problematic Practice or Public Promise?](https://www.futureofbeinghuman.com/p/parasocial-relationships-problematic) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/parasocial-relationships-problematic.html)) — the documented upsurge in human–AI parasocial relationships, its associated harms, and the refusal of a categorical verdict.

## In his own words

From [Can ChatGPT adversely impact mental health?](https://www.futureofbeinghuman.com/p/can-chatgpt-adversely-impact-mental) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/can-chatgpt-adversely-impact-mental.html)), 2023-04-05:

> "The problem is, we tend to be predisposed to form trusting relationships with people whom we feel “see” and understand us, and who are responsive to our needs, hopes, and aspirations. Language plays a large part in how we develop these relationships. And as a result, I suspect we’re likely to see an increasing number of people turn to chatbots as part of their trusted emotional and mental health support system.
>
> This probably isn’t such a bad thing for many people, as long as appropriate checks, balances, and expectations, are put in place. But can these adequately protect vulnerable individuals who risk being harmed by a trusted emotional attachment to a machine that has only the illusion of a reciprocal relationship?"

From [OpenAI's GPT-4o and the challenges of hyper-anthropomorphism](https://www.futureofbeinghuman.com/p/anthropomorphizing-gpt-4o) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/anthropomorphizing-gpt-4o.html)), 2024-05-15:

> "We’ll have to wait until the full features of GPT-4o are more widely available to know whether concerns around “hyper-anthropomorphism” — a concerted effort to create AI’s that are intentionally designed to engage our anthropomorphizing cognitive biases — are justified. But from what we’ve seen so far, my sense is that we’re going to have to start thinking fast about how we handle advanced AI assistants that are designed to make us fall a little in love with them — and possibly give away more of ourselves to them and their creators than we would otherwise chose to."

From [Is ChatGPT's new Voice Mode dangerously persuasive?](https://www.futureofbeinghuman.com/p/is-chatgpts-new-voice-mode-dangerously-persuasive) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/is-chatgpts-new-voice-mode-dangerously-persuasive.html)), 2024-09-01:

> "The question is, is it possible for a machine to gain such a mastery of the nuances of voice-based communication that it is better than most humans in creating bonds that put users in a vulnerable position?"

## Engagement and reception

As of 2026-08-06 our [reception record](reception.md) documents no independent engagement with these essays specifically. Hazel Kwon's methodological critique in the 2024-09-01 post was contributed at Andrew's request, in an email conversation for the essay — collaboration, not reception. The public record around the thread's subject has moved since the essays appeared — Cambridge Dictionary made "parasocial" its 2025 Word of the Year, in part over human–AI relationship concerns, and Attorneys General of 44 US jurisdictions wrote to 13 AI companies in August 2025 about chatbot parasocial relationships and children (both per the 2025-11-19 post) — but that is engagement with the problem, not with the initiative's essays, and we do not claim otherwise. The thinness is an absence of record, not a verdict on the idea: conceptual work of this kind typically accrues visible reception slowly, and the initiative does not typically solicit it (see [Philosophy](philosophy.md)). The nearest documented scholarly uptake sits on the adjacent [Cognitive Trojan Horse](cognitive-trojan-horse.md) thread, including a 2026 *Philosophy & Technology* citation on AI persuasion and epistemic risk ([Reception](reception.md)).

## Where to go deeper

**The essays**

- [Can ChatGPT adversely impact mental health?](https://www.futureofbeinghuman.com/p/can-chatgpt-adversely-impact-mental) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/can-chatgpt-adversely-impact-mental.html)) — 2023-04-05; the mechanism and the stakes.
- [OpenAI's GPT-4o and the challenges of hyper-anthropomorphism](https://www.futureofbeinghuman.com/p/anthropomorphizing-gpt-4o) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/anthropomorphizing-gpt-4o.html)) — 2024-05-15; seduction as design.
- [Is ChatGPT's new Voice Mode dangerously persuasive?](https://www.futureofbeinghuman.com/p/is-chatgpts-new-voice-mode-dangerously-persuasive) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/is-chatgpts-new-voice-mode-dangerously-persuasive.html)) — 2024-09-01; whether safety testing sees the relational pathway.
- [Are we ready to navigate the complex ethics of advanced AI assistants?](https://www.futureofbeinghuman.com/p/navigating-ethics-of-advanced-ai-assistants) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/navigating-ethics-of-advanced-ai-assistants.html)) — 2024-04-24; the ethics groundwork.
- [Learning to live with agentic social AI](https://www.futureofbeinghuman.com/p/learning-to-live-with-agental-social-ai) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/learning-to-live-with-agental-social-ai.html)) — 2024-10-20; when machines that make us love them start to steer us.
- [The Forgotten Risk of Fitness Trackers](https://www.futureofbeinghuman.com/p/personal-fitness-tracking-and-the-hidden-risks-of-seductive-technologies-4070bf17897f) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/personal-fitness-tracking-and-the-hidden-risks-of-seductive-technologies-4070bf17897f.html)) — 2019-03-01; the pre-AI statement.
- [Parasocial Relationships: Problematic Practice or Public Promise?](https://www.futureofbeinghuman.com/p/parasocial-relationships-problematic) ([AI-readable mirror](https://text.futureofbeinghuman.com/substack/parasocial-relationships-problematic.html)) — 2025-11-19; the ledger held open.

**Books**

- Maynard, A. *Films from the Future: The Technology and Morality of Sci-Fi Movies* (2018) — the 2024-09-01 essay notes that exploiting cognitive biases "was the crux of my thinking around AI manipulation" in this book (see [Books](books.md)).

**Related ideas-ring pages**

- [Parasocial communication](parasocial-communication.md) — the deliberate counterpart: the same one-way relational machinery, argued as a public good when practiced with candor.
- [Agentic social AI](agentic-social-ai.md) — the forward hand-off: machines that leverage social bonds to get things done.
- [The Cognitive Trojan Horse](cognitive-trojan-horse.md) — the adjacent framework on why genuine AI characteristics slip past evolved trust checks.
- [Orphan risks](orphan-risks.md) — the risk category the 2019 essay explicitly places seductive technologies within.

**Related corpus pages**

- [Philosophy](philosophy.md) — the initiative's stance on evidence, narrative, and why reception is pointed to rather than harvested.
- [Reception](reception.md) — the checked public record of engagement with the initiative's work.