Stationary Alone, Drifting Together

Many multi-agent systems are coupled through a physical medium. Vehicles share roads, solar inverters share the lines of a distribution feeder, and actuators share a power supply or a rigid body. What happens to one agent then has two parts: the effect of its own actions, and the effect of everyone else's use of the medium.

External conditions often change only the second part. Rain does not change how a car accelerates or how long its route is. It lowers the capacity of the road, so the same number of other cars now causes more delay. A car alone on the road at night hardly notices the rain. The same car in rush-hour traffic does.

We call this interaction-mediated nonstationarity. Each agent, taken alone, faces a stationary task. The fleet as a whole faces a drifting one.

What happens to one agent, in one equation

Write \(s_i\) for the part of the state that belongs to agent \(i\). Over one step, its evolution separates into a private part and a coupling part:

\(F_i(s_i, a_i)\)

The agent on its own: how its state would evolve if no one else used the medium.

\(C_i(\peer{s_{-i}, \ba_{-i}};\, s_i)\)

The coupling: what the other agents (written \(-i\)) do to agent \(i\) through the medium.

\(\drv{\btheta(t)}\)

The coupling gains: what one unit of the others' activity costs agent \(i\) at time \(t\). External conditions act here and only here.

\(\oplus\)

How the two parts combine: by addition (an offset to injected power), by multiplication (a loss of force), or through a realised outcome (a travel time).

Three kinds of drift, and two tests that separate them

From agent \(i\)'s point of view, the environment is nonstationary whenever one ingredient of the equation above changes over time. There are three possibilities, and each has been studied by a different community. Two simple experiments tell them apart: freeze the policies of all other agents, or remove the other agents altogether.

Interaction-mediated drift is the only kind that survives freezing every partner and disappears for a lone agent. It survives freezing because a frozen fleet still loads the medium, and the external condition still acts on that load. It disappears for a lone agent because, with no one else on the medium, the coupling term \(C_i\) is zero whatever the condition.

Seen as one joint system, (C) is a special case of exogenous drift. What sets it apart is its structure: the drift enters only through the coupling, and it scales with the fleet's own load.

Why this drift needs its own treatment

• The harm is a product of two factors

  Below saturation, the disturbance an agent receives in our feeder, lift and Ant benchmarks has the form

  \[ y_i^{\star}(t) \;=\; \underbrace{\drv{\sigma\,\bar k\,A(t)}}_{\text{set by the condition}} \;\times\; \underbrace{\send^{\top}\peer{\bx_i(t)}}_{\text{set by the other agents}} \]

  and routing has the same structure through the ratio of traffic volume to rain-reduced capacity. A method that is told the weather still lacks the traffic, and a method that sees the traffic still lacks the weather. With either factor missing, the executed action misses its target by an amount that no amount of training removes.

• Only a few numbers are unknown

  Each agent knows the infrastructure layout (who shares its medium, and how strongly) and receives its peers' executed actions. Only the gains are unknown: \(r+1\) numbers per agent, where \(r\) is the number of coupling classes (three in our benchmarks), independent of the size of the fleet. These can be estimated online from the agent's own measured shortfall, which is what our method, PACT, does.

• Reactions feed back

  A correction is itself an action on the medium, so it becomes part of the neighbours' load. Inverters that react to one another can start to oscillate, and vehicles that all follow the same accurate forecast can all switch to the route predicted to be faster and make it slower. How strongly agents should act on their estimates is therefore itself a multi-agent question.

Each benchmark is used exactly as published when \(\sigma = 0\). Raising \(\sigma\) strengthens an external condition whose effect on an agent scales with the other agents' load on the shared medium. The reward function is never modified: agents earn less only because the physics delivers less.

Every instance declares the same four objects.

M · medium

What the agents use together, and the load \(\phi_j\) that each user \(j\) puts on it.

W · coupling

A fixed operator computed from the infrastructure layout: which agents affect which, and how strongly.

D · driver

An external condition \(\drv{A(t)} \in [0,1]\), from no effect (0) to full effect (1). No agent can influence it.

H · harm

How the condition, at intensity \(\drv{\sigma A(t)}\), turns the peers' load into a disturbance.

At severity \(\sigma\), the action \(a_i\) that agent \(i\) sends is executed as

The channels \(\peer{\bx_i}\) summarise the other agents' executed load through \(W\), one channel per coupling class, possibly filtered over time. They are exactly zero when agent \(i\) is alone. The harm map does nothing when either factor is absent, \(h(0, \bx; a) = h(\xi, \mathbf 0; a) = a\), which is what makes both tests of Section 1 hold by construction. In routing the harm acts on the realised travel time instead of the action.

All four drivers share one shape

Each driver is a smooth bump over the active half of its cycle and exactly zero over the other half:

The zero half is a built-in control: on those steps every benchmark is the unmodified original at every \(\sigma\). The agents are told neither the period nor the phase.

Is PACT simply given the answer?

On the feeder, the lift and the Ant, the harm is exactly linear in PACT's channels below saturation (Proposition C.2 of the paper). The form is not tailored to the method. It is how each field already writes the effect:

PACT is given the layout of the network and its peers' executed actions, and it estimates the gains online from its own measured shortfall. The same product is what defines the class: remove the peers and it vanishes, freeze them and the gain still moves. We first met the structure in traffic, where it is measured, and designed PACT for the class it defines.

Is using the operator W an unfair advantage?

No. PACT uses only what the environment exposes, and the same inputs are open to every method.

PACT works from three inputs: the infrastructure layout, the executed actions its peers broadcast, and its own measured shortfall. The operator \(W\) is computed from the layout before training: from the road network and the departure schedule on URB, from the feeder's line reactances on MAPDN, from the actuators' positions and hardware classes on VMAS, and from the robot's own model on the Ant. It records who can affect whom through the medium, and how strongly the structure transmits. It does not contain the gains, the severity \(\sigma\), the driver \(\drv{A(t)}\), or the driver's period and phase, and PACT is given none of these.

A known layout is also a realistic input. The people who operate these systems have it: a utility knows its feeder's impedances, a traffic authority its road network, a robot maker its robot's inertia. What they cannot know in advance is how strongly the layout transmits under the day's conditions, and that is the part PACT estimates.

The baselines receive the same kinds of input, in the form their designs call for. On VMAS every learner observes its own shortfall, and on MAPDN every learner also observes the channels, which are the peers' actions weighted by \(W\). The mean-field learners receive the peers' actions, and URLS and ESO use the shortfall exactly as PACT does. The oracle run, Oracle-A(t), goes further: it receives the one quantity PACT is never given, the true driver \(\drv{A(t)}\), on top of everything a learner normally observes. Methods that make no use of the layout do so by design, since they have no place for a known coupling structure. That is a property of those methods, not a restriction of the benchmark, and any method is free to use the layout.

Having the right model and the right inputs is still not the same as solving the task. The table lists what several baselines are given and how they perform at \(\sigma = 3\), with values as reported in Section 5 of the paper; Section 4 below describes each baseline.

The MAPDN values are over the last three evaluations. On MAPDN every learner, the host included, also observes the channels and its own one-step-old shortfall, and on VMAS every learner observes its own shortfall, so the comparisons there are matched in information. The unharmed VMAS return is 56.

• An exact model with the same data still falls behind

  URLS runs PACT's own recursion, with the same forgetting, trust and correction, on the peers' raw actions. On the feeder its model is also exact below saturation, since every coefficient is proportional to \(\drv{A(t)}\). It must identify up to \(n\) coefficients where PACT identifies \(r+1\), and the learners' actions do not vary enough for that. Between two correct models, the declared structure decides.

• Knowing the driver is not enough

  In every benchmark the gains move together with \(\drv{A(t)}\), so \(\drv{A(t)}\) is the only quantity that drifts. Oracle-A(t) is told \(\drv{A(t)}\) at every step and still does not improve on its host, because the harm is \(\drv{A(t)}\) multiplied by the peers' load, and the load changes whenever the co-learners change.

• The right measurement, used as feedback, makes things worse

  The disturbance observer uses the same shortfall that PACT measures, but it learns each reaction one step late. In the feeder's feedback loop this lowers the instability threshold below that of the uncompensated fleet (the observer proposition in Appendix D). On the 6-inverter feeder at \(\sigma = 3\), with a scripted droop host, its voltage loss is 4.32 times the unharmed host's, against 4.21 with no compensation at all; PACT's is 1.04 and the oracle's 1.00.

If the model is right, what remains hard?

Three things, and a correct model class removes none of them.

• The regressor comes from learners, and it runs dry

  Identification needs the peers' actions to vary, and convergent learning removes that variation (the excitation lemma in Appendix C). When we estimated the coupling operator itself instead of declaring it, the fitted coefficient on URB was 0.13 ± 5.0 and never significant in 5004 windows.

• Acting on the estimate changes what is estimated

  Every correction adds to the neighbours' load. At full trust 2 of 5 runs became unstable, and 1 of 5 at trust 0.95 (the trust sensitivity table in Appendix B). If the task were only to track a few numbers, full trust would be best.

• The outcome is not guaranteed

  On VMAS PACT reaches 83% of the unharmed return and closes a little over half of the gap. On the Ant, PACT and its host end at the same return, and the gain is in sample efficiency.

How realistic is each harm?

Each benchmark combines parts taken from its domain with parts we added to create the drift. The table separates the two.

Two details matter when reading the results. SUMO has no model of rain, so the URB delay is applied to recorded travel times rather than inside the car-following dynamics. It is applied to every traveller, human drivers included; the benchmark keeps the human drivers' policies fixed, so for them only the reported times change.

On MAPDN the agent's action plays the role of the curve's reference voltage. On the sloped part of an IEEE 1547 curve, moving the reference voltage by \(\Delta V\) shifts the reactive power by \(k\,\Delta V\), so a dispatched level with the curve reacting around it matches how the standard lets operators steer a fleet of inverters. The same linear loop is the standard model for analysing local Volt/VAr stability (Farivar et al., 2013). Our curve reacts only to the voltage that the peers cause. A complete curve would also react to load and solar variation, which adds type (B) drift on top of type (C).

All methods are evaluated at \(\sigma = 3\) with 5 seeds. Tables report the mean and the standard deviation over seeds, and figures the mean with one standard deviation. PACT wraps an unmodified learner, its host, and changes only the action sent to the environment, so every host is also run on its own, which is PACT at trust 0, with the same seeds. The Demo tab shows a side-by-side rollout on VMAS Balance.

Results on URB

On URB the score is the mean travel time of the agents' vehicles in the test phase, in minutes, so lower is better. Entries are the mean over seeds with the standard deviation in brackets. The video ranks every method at \(\sigma = 3\); the tables below give the same numbers.

\(\Delta\) is the change from \(\sigma = 0\) to \(\sigma = 3\). PACT-IQL wraps IQL with a fixed trust and is reported over 4 seeds at \(\sigma = 3\).

Baselines

The baselines take one recent representative from each class of method that could plausibly handle this drift, preferring methods with public code; those without public code were implemented from their papers. Every baseline runs through the same entry point as PACT, inside the same harmed environment. The tags show the benchmarks each one runs on. Four names differ in Figure 4: RLS is URLS, DEDA-FP is Deep FP, IPGA is PMPG, and QCD+ is QCD.

The benchmark's own reference methods

• Greedya heuristic, not a learner

  shipped with URB

  • URB

  Code

  github.com/COeXISTENCE-PROJECT/URB

  Greedy keeps one shared table holding, for each origin, destination and departure time, the shortest travel time ever recorded on each route, and sends every vehicle to the route with the lowest entry; a route nobody has tried yet is entered at its free-flow time. There is no network and nothing is learned. Two consequences make it a reference line rather than a competitor. It is centralised, reading every controlled vehicle's realised travel time from a single record that no decentralised learner sees. And because it stores the minimum ever observed, which is attained on a dry day, its estimate is effectively the dry-day travel time, so the harm barely reaches it: its travel time rises by 0.1% at \(\sigma = 3\) while every learner loses between 1.3% and 3.8%.

• IQLindependent Q-learning

  shipped with URB

  • URB

  Code

  github.com/COeXISTENCE-PROJECT/URB

  One Q-network per agent, trained independently. A URB episode is a single decision, so the next state is terminal and the Q-learning target reduces exactly to the immediate reward: each agent regresses its own travel time on its observation and chosen route, over a replay buffer of the last 2048 days, and acts greedily once exploration has decayed to zero. That is the smallest machinery the task requires, and pooling days averages out the noise that the other 88 vehicles inject into any single day's reward.

  This explains the ordering of the learners. The policy-gradient hosts estimate the same quantity from single on-policy samples and discard them, paying variance for a policy parameterisation that a one-step decision does not need. QMIX and VDN exist to split a shared team reward into per-agent shares, and MAPPO fits a centralised critic over all 89 agents, but the reward here is already each vehicle's own travel time and the horizon is one step, so that machinery is carried without being used. The advantage is one of estimator fit to the task's shape, and it is orthogonal to the drift: IQL loses 1.3% at \(\sigma = 3\), inside the band of the other hosts.

Trust regions and sequential updates

• HAPPO and HATRPOwith HAA2C on the Ant

  ICLR 2022 · JMLR 2024

  • URB
  • MAPDN
  • VMAS
  • Ant

  Paper

  arxiv.org/abs/2304.09870

  Code

  github.com/PKU-MARL/HARL

  Heterogeneous-agent trust-region methods update the agents one at a time in a random order and weight each update by the change the preceding agents have just made. This is the standard multi-agent remedy for drift caused by co-learners. They test whether stabilising learning helps when the drift continues with every partner frozen.

• HASAC

  ICLR 2024

  • URB
  • MAPDN
  • VMAS

  Paper

  arxiv.org/abs/2306.10715

  Code

  github.com/PKU-MARL/HARL

  The off-policy, maximum-entropy member of the same family: soft actor-critic with sequential agent updates. Its entropy bonus keeps every agent's actions varied, which is the excitation an estimator of the coupling needs, while its critic is fitted on replayed data that mix phases of the driver.

Memory

• R-MAPPO and R-IPPO

  NeurIPS 2022

  • URB
  • MAPDN
  • VMAS

  Paper

  arxiv.org/abs/2103.01955

  Code

  github.com/marlbenchmark/on-policy

  MAPPO and IPPO with a recurrent layer between the observation encoder and the policy head, trained by backpropagation through time. They test the simplest alternative explanation: that a policy with memory can infer the drift from its own history. On URB the recurrent state is carried across days.

Graphs, mean fields and agent models

• DGN

  ICLR 2020

  • URB
  • VMAS

  Paper

  arxiv.org/abs/1810.09202

  Code

  github.com/PKU-RL/DGN

  Graph convolutional reinforcement learning: each agent combines its features with its neighbours' through multi-head attention, and a Q-network acts on the result, with a regulariser that keeps the attention stable over time. It tests whether a learned graph over the agents can discover the coupling that PACT is given.

• MF-Q and MF-ACMF-AC and MF-AC-T in Figure 4

  ICML 2018

  • URB
  • VMAS
  • MAPDN, via the host

  Paper

  proceedings.mlr.press/v80/yang18d

  Code

  github.com/mlii/mfrl

  Mean-field reinforcement learning summarises an agent's neighbourhood by its mean action and learns the value of the agent's own action against that mean, with a Q-function (MF-Q) or an actor-critic (MF-AC); MF-AC-T uses the team mean with the agent included. The coupling in our benchmarks is a weighted mean of the peers' load, so mean-field RL is its classical alternative. On MAPDN the host already observes the weighted mean of the peers' actions and represents this class.

• LIAM

  NeurIPS 2021

  • URB
  • MAPDN
  • VMAS

  Paper

  proceedings.neurips.cc/…/a03caec5…-Paper.pdf

  Code

  github.com/uoe-agents/LIAM

  Local information agent modelling: an encoder learns a representation of the other agents from the controlled agent's own observations and actions, trained to reconstruct what the others observe and do, and the policy conditions on it. It tests whether modelling the peers' behaviour is enough when what drifts is how much that behaviour costs.

Context, latent tasks and restarts

• LCPO

  ICLR 2025

  • URB
  • MAPDN
  • VMAS

  Paper

  openreview.net/pdf?id=l6QnSQizmN

  Code

  github.com/pouyahmdn/LCPO

  Built for environments driven by an observed exogenous context. It anchors the policy on states from past contexts, so that learning in the current context does not overwrite behaviour that other contexts still need. We give it the driver \(\drv{A(t)}\) as its context.

• Oracle-A(t)the oracle driver

  Control run

  • URB
  • MAPDN
  • VMAS

  Code

  Part of our code, released upon acceptance

  The host with the true driver \(\drv{A(t)}\) appended to every observation: the one quantity that drifts, and one that PACT is never given. It measures what knowing the condition is worth on its own.

• LILAC

  ICML 2021

  • MAPDN

  Paper

  proceedings.mlr.press/v139/xie21c.html

  Code

  github.com/EmiliyanGospodinov/LILAC (third-party implementation)

  Models the environment as a sequence of MDPs indexed by a latent variable that changes between episodes. A variational model infers the latent from each episode, an LSTM prior predicts the next one, and a soft actor-critic learner conditions on it.

• WISDOM

  arXiv 2025

  • URB
  • MAPDN
  • VMAS

  Paper

  arxiv.org/abs/2510.04507

  Code

  github.com/MinWangcs/WISDOM

  Learns a representation of the current task from recent transitions, decomposes the sequence of representations over several time scales with wavelets, and predicts how it will evolve; the policy conditions on the prediction. It is the learned counterpart of PACT's declared basis and forecast.

• QCDQCD+ in Figure 4

  arXiv 2024

  • URB
  • MAPDN
  • VMAS

  Paper

  arxiv.org/abs/2410.13772

  Code

  No public code; implemented from the paper

  Quickest change detection with restarts: a Bernoulli generalised likelihood-ratio test watches the reward stream and resets the learner when it detects a change. Variants restart at random times or follow the multi-scale tests of MASTER. It tests whether detecting change and relearning is enough, without identifying what changed.

Online identification

• RMAwith UP-OSI as a variant

  RSS 2021

  • URB
  • MAPDN
  • VMAS

  Paper

  roboticsproceedings.org/rss17/p011.html

  Code

  Implemented from the paper

  Rapid motor adaptation trains a policy on privileged environment factors, then distils an adaptation module that recovers them from the recent history of observations and actions. It is the closest methodological neighbour of PACT: both identify something online, but RMA learns its identifier from history while PACT estimates a few gains on a declared basis. The privileged factor is the drifting gain on MAPDN and the driver on VMAS and URB.

Robustness

• DRdomain randomisation

  IROS 2017

  • URB
  • MAPDN
  • VMAS

  Paper

  arxiv.org/abs/1703.06907

  Trains the host on severities drawn uniformly from \([0, 3]\) in every episode, without telling the policy which severity it faces. It hedges against the drift instead of identifying it.

• DORAEMON

  ICLR 2024

  • URB
  • MAPDN
  • VMAS

  Paper

  arxiv.org/abs/2311.01885

  Code

  github.com/gabrieletiboni/doraemon

  Domain randomisation with an adaptive curriculum: it maximises the entropy of the training distribution over \(\sigma\) subject to a success-rate constraint and a trust region, so the range of severities widens as fast as the policy can handle it.

• ERNIE

  NeurIPS 2023

  • URB
  • MAPDN
  • VMAS

  Paper

  proceedings.neurips.cc/…/d6f8517f…-Abstract-Conference.html

  Code

  github.com/abukharin3/ERNIE

  Makes each policy robust to changes in its input with an adversarial regulariser, solved as a Stackelberg game. Its analysis bounds the change of return under perturbed observations or transitions for policies that are smooth in their input.

Model-based MARL

• M3W

  NeurIPS 2025

  • URB
  • MAPDN
  • VMAS

  Paper

  openreview.net/forum?id=fi24ry0BX5

  Code

  github.com/zhaozijie2022/m3w-marl

  Learns a world model whose dynamics and reward predictors are mixtures of experts, and chooses joint actions by model-predictive path-integral planning. It tests whether a learned model of the dynamics, drift included, can take the place of a declared one.

Fictitious play and performative learning

• Deep FPDEDA-FP in Figure 4

  NeurIPS 2025

  • URB
  • MAPDN
  • VMAS

  Paper

  arxiv.org/abs/2510.22158

  Code

  No public code; implemented from the paper

  Deep fictitious play for mean-field games: each iteration computes a best response to the averaged population, and an average policy fitted by supervised learning plays at evaluation. It tests whether seeking an equilibrium, with no estimator at all, reaches what PACT reaches.

• PMPGIPGA in Figure 4

  arXiv 2025

  • URB
  • MAPDN
  • VMAS

  Paper

  arxiv.org/abs/2504.20593

  Code

  No public code; implemented from the paper

  Independent policy gradient (IPGA) and natural policy gradient (INPG) under repeated retraining, from the theory of performative Markov potential games, in which the deployed policies reshape the game. It tests whether a gain could come from the deployment schedule rather than from identification.

Estimators without the declared operator

• ESOextended state observer

  IEEE TIE 2009

  • URB
  • MAPDN
  • VMAS

  Paper

  doi.org/10.1109/TIE.2008.2011621

  Toolbox

  arxiv.org/abs/2112.01614

  Code

  pyadrc.readthedocs.io

  The observer at the heart of active disturbance rejection control. It estimates each agent's lumped disturbance from its own measured shortfall and cancels the estimate at the next step, with no peers and no structure. It is the classical control answer to an unknown disturbance.

• URLSRLS in Figure 4

  Control run

  • URB
  • MAPDN
  • VMAS

  Code

  Part of our code, released upon acceptance

  Unstructured recursive least squares: PACT's own estimator, with the same forgetting, trust and correction, applied to the peers' raw broadcast actions instead of the declared channels. The two differ only in the design matrix, so URLS isolates the value of the declared operator.

State of the art on the feeder

• TPA

  KDD 2024

  • MAPDN

  Paper

  dl.acm.org/doi/10.1145/3637528.3671790

  arXiv

  arxiv.org/abs/2406.17818

  Code

  github.com/Canyizl/TPA-for-AVC

  Temporal prototype-aware learning for active voltage control. It extends an off-policy learner with a transformer over the zone's bus features, a recurrent summary of recent solar and load profiles, and 24 seasonal prototypes that condition the policy. It is the closest published method on MAPDN and was built for time-varying operating conditions.