✨ TL;DR
This paper shows that decentralized machine learning can achieve the same performance as centralized learning by having clients share Gibbs measures (probability distributions over models) instead of raw data. The key innovation is using each client's Gibbs measure as a reference measure for the next client, effectively encoding prior information while preserving privacy.
Decentralized machine learning typically requires either sharing local datasets (compromising privacy) or accepting degraded performance compared to centralized learning. The challenge is to design a decentralized learning framework that achieves centralized performance guarantees without requiring clients to share their private data.
Clients adopt an empirical risk minimization with relative-entropy regularization (ERM-RER) framework and establish forward-backward communication. The key mechanism is that each client k produces a Gibbs measure (a probability distribution over models weighted by their empirical risk), which is then used as the reference measure by client k+1. This creates a chain where local inductive biases are propagated through reference measures rather than raw data. The regularization factors must be scaled appropriately with local sample sizes to achieve centralized performance.
What the paper shows.
The paper demonstrates that when clients follow the ERM-RER framework with forward-backward communication and share Gibbs measures, they achieve the same performance as centralized ERM-RER that has access to all datasets. The specific scaling requirement for regularization factors with local sample sizes is identified as necessary for this equivalence.
The paper does not provide explicit numerical experiments or empirical validation of the theoretical results. The forward-backward communication structure may impose constraints on the network topology and communication patterns. The practical computational cost of computing and sharing Gibbs measures compared to other decentralized approaches is not discussed. The scalability to very large numbers of clients or high-dimensional model spaces is not addressed.
✨ Generated by Claude · Apr 25, 2026 · Read the PDF for authoritative content.