avatar
textmodels
textmodels's Blog
Writings, Papers and Blogs on Text Models
textmodels
textmodels's Blog
Writings, Papers and Blogs on Text Models
  • AI Models on HackerNoon
About Writings, Papers and Blogs on Text Models
Login
  • About Writings, Papers and Blogs on Text Models
  • Login

Adversarial Training in Multi-Exit Networks: Proposed NEO-KD Algorithm and Problem Setup

cover
30 Sept 2024

Authors:

(1) Seokil Ham, KAIST;

(2) Jungwuk Park, KAIST;

(3) Dong-Jun Han, Purdue University;

(4) Jaekyun Moon, KAIST.

Table of Links

Abstract and 1. Introduction

2. Related Works

3. Proposed NEO-KD Algorithm and 3.1 Problem Setup: Adversarial Training in Multi-Exit Networks

3.2 Algorithm Description

4. Experiments and 4.1 Experimental Setup

4.2. Main Experimental Results

4.3. Ablation Studies and Discussions

5. Conclusion, Acknowledgement and References

A. Experiment Details

B. Clean Test Accuracy and C. Adversarial Training via Average Attack

D. Hyperparameter Tuning

E. Discussions on Performance Degradation at Later Exits

F. Comparison with Recent Defense Methods for Single-Exit Networks

G. Comparison with SKD and ARD and H. Implementations of Stronger Attacker Algorithms

3 Proposed NEO-KD Algorithm

3.1 Problem Setup: Adversarial Training in Multi-Exit Networks

Figure 1: NEO-KD consists of two parts that together improve the adversarial robustness: NKD and EOKD. (a) NKD guides the output of the adversarial data to mimic the ensemble outputs of neighbor exits of clean data. (b) EOKD reduces adversarial transferability of the network by distilling orthogonal knowledge of the clean data to adversarial data for the non-ground-truth predictions, in an exit-wise manner. Although omitted in this figure, EOKD normalizes the likelihood before distilling the soft labels. The overall process operates in a single model, although we consider two cases depending on the input (clean or adversarial example) for a clear presentation.

This paper is available on arxiv under CC 4.0 license.


← Previous

Advancing Robustness in Multi-Exit Networks Through Exit-Wise Knowledge Distillation

Up Next →

A Robust Self-Distillation Strategy for Multi-Exit Networks

avatar
textmodels
textmodels's Blog
Writings, Papers and Blogs on Text Models
textmodels
textmodels's Blog
Writings, Papers and Blogs on Text Models
  • About
  • Stories
  • Random Story
  • Terms
  • Privacy
  • Publish Your Story