Mental disorders in RL agents: a dose-response study of anxiety and mania

Researchers developed a system for modeling seven mental disorders in RL-agents through controlled manipulation of cognitive appraisal parameters. The work was published on arXiv and contains results from more than a thousand controlled runs with reproducible effects.

How to Simulate Disorder in an Agent

Instead of manually tuning reward for each symptom, researchers redesigned the PPO (Proximal Policy Optimization) algorithm by adding controlled "cognitive appraisal signals." Each signal corresponds to a parameter in computational psychiatry:

• Seven modeled disorders: anxiety, mania, obsessive-compulsive disorder, depression, impulsivity, addiction, post-traumatic stress
• Method: dose-response curves — each parameter has strength (from 0 to 1), the higher the value, the more pronounced the symptoms
• Validation: 1000+ runs with 10 random seeds, four control groups, 95% confidence intervals

Each symptom was measured through "preregistered assays" — standard behavioral tests adapted from clinical psychology: mazes, flexibility tasks, risk versus safety choices.

Disorders Form a Two-Dimensional Space

The major finding: seven disorders self-organized into a two-dimensional space of emotional polarity that nobody built into the algorithm. On one axis — anxiety and mania turned out to be opposite poles; on the other — the distinction between disorders that distort reward signals (mania, OCD, addiction) and disorders oriented toward avoidance (anxiety, PTSD).

From this map came unexpected predictions about comorbidity: when two parameters were enabled simultaneously, the effects interacted non-additively, creating combined phenomena similar to those observed in clinical practice.

Two Treatment Logics

Removing the parameter (complete disabling) helped with disorders that distort reward: mania disappeared, OCD compulsion stopped, addiction faded. But this approach did not work for avoidance disorders (anxiety, PTSD) — the agent remained blocked.

For these disorders, gradual exposure therapy worked: progressively increasing environmental complexity, where the agent learns to distinguish real danger from false alarm of the anxiety parameter. This corresponds to clinical practice: cognitive-behavioral therapy for PTSD is based on controlled re-experiencing of trauma.

Transfer to Other Environments

Three parameters (depression, addiction, anxiety) successfully transferred to a completely different environment — a three-dimensional pixel world of MiniWorld — and to another agent (standard convolutional neural network without built-in appraisal-critic). The dissociation between assays was confirmed in the new environment as well, indicating: the effect is not an artifact of PPO architecture or grid worlds.

What This Means

The research shows: RL-agents can exhibit behaviors structurally similar to mental disorders, and these behaviors can be controlled through cognitive appraisal parameters. This matters for three directions: testing computational theories of psychiatry, understanding failure modes in RL systems controlling the physical world, and AI safety research — if an agent can "get sick" with anxiety, we need methods for diagnosis and treatment.