Distributed Backdoor A1acks

against Federated Learning

Chulin Xie
University of Illinois Urbana-Champaign, chulx2@illinois.edu

Joint work with

Keli Huang, Pin-Yu Chen, Bo Li

Sept. 30, 2020 FLOW seminar

DBA: Distributed Backdoor Attacks against Federated Learning

Chulin Xie, Keli Huang, Pin-Yu Chen, Bo Li. ICLR 2020
Outline

§ Introduction
§ Distributed Backdoor Attack
§ Experiments
§ Conclusion & Discussion
Federated learning

§ Distributed training ML models without direct access to diverse training

data
§ a shared global model with improved performance
§ Applications: loan status prediction, health situation assessment, next-
word prediction while typing

Image reference(CMU)
 One Round of Federated Learning

Gt+1
&
𝜂
𝐺 !"# = 𝐺 ! + & 𝐿!"#
$ − 𝐺!
𝑛
$%#
federated learning aggregator

Model Update: 𝐿!"#

$ − 𝐺!
…
𝑓$ 𝑤$ = 𝑙( 𝑥)$ , y)$ , 𝑤$ )
…
)∈+!

𝑤$∗ = 𝑎𝑟𝑔𝑚𝑎𝑥(! ∑)∈+! 𝑃 𝐿!"#

$ 𝑥)$ = 𝑦)$

§ distributed learning methodology

Gt
§ inherently diverse (i.e., non-i.i.d.) data
distribuAon

may facilitate data poisoning attacks on the shared § limited access to training data due to
model trained with FL privacy concerns

* McMahan et al., Communication Efficient Learning of Deep Networks from Decentralized Data, AISTATS 2017
 Backdoor Attack

§ data poisoning attack

§ manipulate a subset of training data
§ mislead the trained model to make a targeted wrong prediction on
any test data that has an attacker-chosen pattern (i.e., a trigger)

Backdoor data with an

Benign data attacker-chosen pattern

Correct predic>on Targeted wrong prediction(e.g.,

classify as “bird”)
Main task Backdoor task

* Tianyu Gu, Kang Liu, Brendan Dolan-GaviM, and Siddharth Garg. Badnets: EvaluaOng backdooring aMacks on deep neural networks. IEEE Access, 2019
 Backdoor Attack against Federated Learning

§ Purpose of backdoor attacks against FL:

§ manipulate local models to simultaneously fit the main task and backdoor task
§ after aggregation
§ the global model would behave normally on benign data while achieving high
attack success rate on backdoor data

Backdoor data with an

Benign data
attacker-chosen pattern

Correct prediction Targeted wrong predicEon(e.g.,

classify as “bird”)
Main task Backdoor task

* Bagdasaryan, E., Veit, A., Hua, Y., Estrin, D., & Shmatikov, V. How to backdoor federated learning. AISTATS 2020
* Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. ICML. 2019.
Outline

§ IntroducHon
§ Distributed Backdoor AIack
§ Experiments
§ Conclusion & Discussion
 DBA: Distributed Backdoor Attack

centralized backdoor attack (current setting) DBA: distributed backdoor aKack (ours)

§ DBA attacker’s trigger: the local trigger; combined whole trigger: global trigger

same ultimate adversarial goal: using the SAME global trigger to attack the global model

§ same amount of total injected triggers (e.g., modified pixels) for fair comparison
DBA Examples
A>acker Ability in Federated Learning

§ full control of their local training process:

§ ☑ backdoor data injec>on
§ ☑ upda>ng local training hyperparameters
§ Local training epoch
§ Local learning rate

§ do not have the ability to:

§ ❌ influence the privilege of central server
§ such as changing aggrega1on rules
§ ❌ influence other clients
§ tampering the training process
§ tampering the model updates
ObjecAve of Backdoor A>ack

§ Centralized Backdoor Attack

Malicious Objective Benign Objective

𝑤!∗ = 𝑎𝑟𝑔 𝑚𝑎𝑥#, ∑$∈&, 𝑃 𝐺 '() 𝑅 𝑥$! , 𝜙 = 𝜏 + ∑$∈&, 𝑃 𝐺 '() 𝑥$! = 𝑦$!
-., /01

§ Poison ratio
§ a set Parameters of an attacker-chosen trigger pattern
§ Transform clean data in any class into backdoored data
§ target label

Trigger factors (size, gap and locaEon) in Trigger factor (feature importance ranking) in
backdoored images tabular data

in our implementaDon we use minimizing the cross-entropy loss as the training objecDve
 Objective of Distributed Backdoor Attack

§ Distributed Backdoor Attack

M distributed
Malicious Objective Benign Objec1ve a;ackers

𝑤!∗ = 𝑎𝑟𝑔 𝑚𝑎𝑥#, ∑$∈&, 𝑃 𝐺 '() 𝑅 𝑥$! , 𝜙!∗ = 𝜏; 𝛾; 𝐼 + ∑$∈&, 𝑃 𝐺 '() 𝑥$! = 𝑦$! , ∀ 𝑖 ∈ [𝑀]
-., /01

§ geometric decomposing strategy for the local trigger paNern

§ poison round interval
§ 𝐼 = 0 all the local triggers are embedded within one round
§ 𝐼 = 1 the local triggers are embedded in consecu1ve rounds
§ the scale factor

breaks a centralized attack formulation into M distributed sub-attack problems

in our implementation we use minimizing the cross-entropy loss as the training objective
Outline

§ Introduction
§ Distributed Backdoor Attack
§ Experiments
§ Conclusion & Discussion
Distributed V. S. Centralized Backdoor A>ack

§ mulHple-shot aIack(conHnuous aIack)

Evaluation setup
• MNIST dataset
• Total of 100 agents, 10 agents are selected
each round
• Distributed attackers or centralized attacker
are consistently selected each round
• Test attack success rate in the global model
Distributed V. S. Centralized Backdoor A>ack

§ multiple-shot attack(continuous attack)

Test the attack success rate
of global trigger in the global
model
Evaluation setup
• MNIST dataset
• Total of 100 agents, 10 agents are selected
each round
• Distributed attackers or centralized attacker
are consistently selected each round
• Test attack success rate in the global model

• DBA converges faster and even yields a higher attack success rate
than centralized backdoor attack
Distributed V. S. Centralized Backdoor A>ack

§ mulHple-shot aIack(conHnuous aIack)

Test the attack success rate
of global & local triggers in
the global model
EvaluaEon setup
• MNIST dataset
• Total of 100 agents, 10 agents are selected
each round
• Distributed aSackers or centralized aSacker
are consistently selected each round
• Test aSack success rate in the global model
DBA:
• global trigger never actually appears in any local training dataset
• The test attack success rate of the global trigger is higher than any local trigger Unique findings for DBA;
Implies inefficiency of
• The global trigger converges faster in attack performance than local triggers centralized attack on FL
Centralized attack:
• embeds the whole pattern during training so its attack success rate of any local triggers is low
• indicates that the success of global trigger does not require the same success for local triggers
 Distributed V. S. Centralized Backdoor Attack

§ single-shot attack
DBA embed local triggers 1 to 4 centralized attacker implants
separately in round 12, 14, 16, 18 its global trigger in round 18

EvaluaEon setup
• MNIST dataset
• Total of 100 agents, 10 agents are selected
each round
• Every aSacker is only selected once
• ASacker performs scaling in their malicious
updates (scale factor = 100)
• Test aSack success rate in the global model
Distributed V. S. Centralized Backdoor A>ack

§ single-shot attack
Test the attack success rate
of global trigger in the global
model
Evaluation setup
• MNIST dataset
• Total of 100 agents, 10 agents are selected
each round
• Every attacker is only selected once
• Attacker performs scaling in their malicious
updates (scale factor = 100)
• Test attack success rate in the global model

• both reach a high attack success rate after performing a complete backdoor
with scaling malicious update
• distributed attack is more persistent than centralized attack
Distributed V. S. Centralized Backdoor A>ack

§ single-shot a/ack
Test the aBack success rate
of global & local triggers in
the global model
Evaluation setup
• MNIST dataset
• Total of 100 agents, 10 agents are selected
each round
• Every attacker is only selected once
• Attacker performs scaling in their malicious
updates (scale factor = 100)
• Test attack success rate in the global model

DBA
• performs data poisoning only using local triggers
• global trigger lasts longer than any local triggers
Centralized attack
• the attack success rate of centralized attack in local triggers and the global trigger
drops faster than that of DBA
Distributed V. S. Centralized Backdoor Attack

multiple-shot attack(Attack A-M) single-shot attack(Attack A-S)

studies how easy the backdoor is successfully studies how fast the backdoor effect
injected diminishes
 The Robustness of Distributed Attack

§ Two robust FL aggregation algorithms

RFA FoolsGold
• based on distance metrics • based on similarity metrics

replacing the weighted arithmetic

mean(AM) in the aggregation step with an
approximate geometric median(GM)
reduces aggregaAon weights of parAcipaAng
clients that repeatedly contribute similar
RFA is claimed to be able to detect more nuanced outliers gradient updates
which goes beyond the worst-case of the Byzantine setting

* Krishna Pillutla, Sham M. Kakade, and Zaid Harchaoui. * Clement Fung, Chris JM Yoon, and Ivan Beschastnikh.
Robust Aggregation for Federated Learning. Mitigating sybils in federated learning poisoning.
arXiv preprint, 2019. arXiv preprint arXiv:1808.04866, 2018.
Robustness of Distributed A>ack

§ Evaluate two robust aggregation algorithms based on distance or

similarity metrics in multiple-shot attack scenario
FoolsGold
distributed attack
91.55%

FoolsGold RFA
centralized attack centralized attack
2.91% totally fail

• the a;ack success rate of DBA is much higher and the convergence speed is
much faster under RFA and FoolsGold
• DBA is more insidious as it can be;er evade the robust FL aggrega1ons
 Explanation for DBA via Feature Visualization

§ Decision visualizaEon of poisoned digit “4” with target label “2” on a

DBA poisoned model using Grad-CAM

• Each locally triggered image alone is a weak a;ack as none of them can change the predic1on
• when assembled together as a global trigger, the backdoored image is classified as the target label
• the stealthy nature of DBA

* Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.
Grad-cam: Visual explanations from deep networks via gradient-based localization. CVPR. 2017.
 ExplanaAon for DBA via Feature Importance

§ Soft Decision Tree distills a trained neural network by training with data
and their soft targets
trigger area after poisoning indeed
becomes much more significant for
decision making

(backdoor target label is 2)

Soft decision tree for MNIST dataset

* Nicholas Frosst and Geoffrey Hinton. DisOlling a neural network into a sob decision tree.
arXiv preprint arXiv:1711.09784, 2017.
ExplanaAon for DBA via Feature Importance

the insignificant features become highly

important for prediction after poisoning

Feature importance of LOAN dataset learned

from its soU decision tree
Analysis of Trigger Factors in DBA

§ Effects of Scale Factor(study DBA trigger factors under single-shot

attack scenario)

DBA-ASR the attack success rate

when the last distributed local trigger is
embedded
Main-Acc the accuracy
when the last distributed local trigger is
embedded
DBA-ASR-t the attack success rate of
t rounds after a complete DBA is performed.
It reveals the persistence.
Main-Acc-t main accuracy
after t rounds
Analysis of Trigger Factors in DBA

§ Effects of Trigger Location, Trigger Gap, Trigger Size

Analysis of Trigger Factors in DBA

§ Effects of Poison Round Interval

Analysis of Trigger Factors in DBA

§ Effects of Poison Per Batch

Analysis of Trigger Factors in DBA

More details and results in paper:

§ Analysis of DBA factors(scale

factor, trigger size, trigger gap,
trigger location, poison interval,
poison ratio, data distribution) in
LOAN, MNIST, CIFAR-10, Tiny-
Imagenet
Outline

§ Introduction
§ Distributed Backdoor Attack
§ Experiments
§ Conclusion & Discussion
 In summary

§ We propose the distributed backdoor aNack (DBA) — a novel threat assessment

framework developed by fully exploi>ng the distributed nature of Federated Learning.

§ Our results suggest DBA is a more powerful and stealthy aNack, which sheds lights on
characterizing the robustness of Federated Learning.
Discussion: Possible Defenses for Backdoor Attacks
In a single model setting(in contrast to federated learning)

§ Attack types:

Data Model Inference

Training Phase Test Phase “panda” “gibbon”

Training-Time A3ack Test-Time Attack

(e.g. Backdoor, Label-Flipping) (Adversarial Examples)

§ Defenses background
§ Many empirical defenses have been “broken” by new aNacks
§ A Cer;fied defense( in 𝑙! norm) is a classifier which returns both a predic>on
and a cer>ficate that the predic>on is constant within an 𝑙! ball around the
input

x
 Discussion: Possible Defenses for Backdoor A>acks
In a single model seOng

§ Certified Robustness via Randomized Smoothing for Test-Time Attack

Randomized Smoothing

§ First, given a neural net 𝑓 (the base classifier) and the input 𝑥

§ Then, smooth 𝑓 into a new classifier 𝑔 (the smoothed classifier). 𝑔 returns the most probable
predic1on by 𝑓 of random Gaussian corrup1ons of 𝑥

Clean image Image with Gaussian noise

* Cohen, Jeremy, Elan Rosenfeld, and Zico Kolter. "Certified Adversarial Robustness via Randomized Smoothing." ICML. 2019.
 Discussion: Possible Defenses for Backdoor A>acks
In a single model seOng

§ Certified Robustness via Randomized Smoothing for Test-Time Attack

Randomized Smoothing
§ 𝑔 returns the most probable predicEon by 𝑓 of random Gaussian corrupEons of 𝑥

Example: consider the input 𝑥 =

Suppose that 𝑓 calssifies 𝑥 + 𝒩(0, 𝜎 ! 𝐼) =

with class probabilities:

Then 𝑔 𝑥 = 🐼

* Cohen, Jeremy, Elan Rosenfeld, and Zico Kolter. "Certified Adversarial Robustness via Randomized Smoothing." ICML. 2019.
 Discussion: Possible Defenses for Backdoor A>acks
In a single model seOng

§ Certified Robustness via Randomized Smoothing for Test-Time Attack

Certified Robustness

§ If we shift the Gaussian, the probabilities of each class can’t change

too much

§ Therefore, if we know the class probabilities at the input x, we can

certify that for sufficiently small perturbations of x, the 🐼 probability
will remain higher than the 🐒 probability

The smoothed classifier g provably returns the top class

within a 𝑙! ball around x of radius:
🐼 𝜎 "#
𝑅= (Φ 𝑝$ − Φ"# 𝑝% )
2
Φ!" is the inverse standard Gaussian CDF

* Cohen, Jeremy, Elan Rosenfeld, and Zico Kolter. "Certified Adversarial Robustness via Randomized Smoothing." ICML. 2019.
 Discussion: Possible Defenses for Backdoor A>acks
In a single model seOng

Randomized Smoothing for test-time attack Randomized Smoothing for training-.me aKack

Standard classification Standard classifica=on

Dataset, Train and

Classifier 𝑓
Input Output Test input Classify Output

Noisy Certified robust to

Noisy CerEfied robust to
Dataset, training-time
Input test-Eme aSack
Test input attack

Smoothed classification Smoothed classifica=on

Smooth over input Smooth over training dataset

Recast 𝑓 as the whole

training process

* Rosenfeld, Elan, Ezra Winston, Pradeep Ravikumar, and J. Zico Kolter. “Certified Robustness to Label-Flipping Attacks via Randomized Smoothing.” ICML. 2020.
* Weber, Maurice, Xiaojun Xu, Bojan Karlas, Ce Zhang, and Bo Li. "RAB: Provable Robustness Against Backdoor Attacks." arXiv preprint arXiv:2003.08904 (2020).
 Discussion: Possible Defenses for Backdoor Attacks
In a single model setting

§ CerHfied Robustness via Randomized Smoothing for Backdoor A*ack

(Training-0me A*ack)
Randomized Smoothing

the smoothed classifier

* Weber, Maurice, Xiaojun Xu, Bojan Karlas, Ce Zhang, and Bo Li. "RAB: Provable Robustness Against Backdoor AMacks." arXiv preprint arXiv:2003.08904 (2020).
 Discussion: Possible Defenses for Backdoor Attacks
In a single model setting

§ Certified Robustness via Randomized Smoothing for Backdoor Attack

(Training-time Attack)
Cer>fied Robustness

* Weber, Maurice, Xiaojun Xu, Bojan Karlas, Ce Zhang, and Bo Li. "RAB: Provable Robustness Against Backdoor AMacks." arXiv preprint arXiv:2003.08904 (2020).
Discussion: Possible Defenses for Backdoor Attacks
In Federated Learning setting

Local
Local
Model
Data Update
… …
Local
Local Global
Model Inference
Data Update Model
… … Test Phase
Local Local
Model
Data Update
Training Phase

§ Much harder in FL:

§ Heterogeneous data distribution;
§ Aggregation over local model updates;
§ ……

§ Certified Robustness for Backdoor Attack: To be explored

§ Certified Robustness for Distributed Backdoor Attack: To be explored
 Thank you for listening!
• This work was partly supported by IBM-ILLINOIS Center for Cognitive
Computing Systems Research (C3SR) – a research collaboration as part of
the IBM AI Horizons Network.
• Code is available: https://github.com/AI-secure/DBA