Differentially Private Clustered Federated Learning with Privacy-Preserving Initialization and Normality-Driven Aggregation

Jie Xu; Haaris Mehmood; Rogier Van Dalen; Karthikeyan Saravanan; Mete Ozay

✨ TL;DR

PINA is a differentially private clustered federated learning framework that uses low-rank adapters and compressed sketches for privacy-preserving initialization, followed by normality-driven aggregation to improve convergence. It achieves 2.9% higher accuracy than state-of-the-art DP-FL methods while maintaining formal privacy guarantees.

01 · Problem

Federated learning enables distributed training while keeping raw data on devices, but it still leaks private information. While differential privacy and secure aggregation provide formal privacy guarantees, combining them with clustered federated learning (which addresses data heterogeneity) is challenging. The core issue is that injected DP noise makes individual client updates too noisy for the server to initialize cluster centroids effectively, leading to poor convergence and generalization.

02 · Approach

PINA operates in two stages. First, each client fine-tunes a lightweight low-rank adaptation (LoRA) adapter and privately shares a compressed sketch of the update with the server. The server uses these sketches to construct robust cluster centroids. Second, PINA introduces a normality-driven aggregation mechanism that improves convergence and robustness while maintaining differential privacy guarantees against an untrusted server.

03 · Key insights

What the paper shows.

01Low-rank adapters combined with compressed sketches enable effective cluster initialization despite DP noise

02Normality-driven aggregation improves both convergence speed and robustness in the presence of DP noise

03Clustered federated learning benefits are preserved while providing formal privacy guarantees

04The two-stage approach separates the challenges of initialization and aggregation for better handling of privacy-utility tradeoffs

04 · Results

PINA outperforms state-of-the-art DP-FL algorithms by an average of 2.9% in accuracy across privacy budgets of epsilon in {2, 8}. The method successfully maintains the benefits of clustered federated learning while providing formal differential privacy guarantees against an untrusted server.

05 · Limitations

The paper does not explicitly discuss computational overhead of the LoRA fine-tuning and sketch compression steps on resource-constrained devices, scalability to very large numbers of clusters, or sensitivity analysis for key hyperparameters. The evaluation appears limited to specific privacy budget values and does not provide detailed ablation studies on individual components.

✨ Generated by Claude · Apr 25, 2026 · Read the PDF for authoritative content.

What the paper shows.

↘ Related papers