AdapterFusion

本文目录导读：

1. The Core Problem: The "Stiffness" of Multi-Task Learning
2. The Architecture: A Two-Phase Process
3. How the Fusion Layer Works (Technical Detail)
4. Why is AdapterFusion Important?
5. Comparison: Adapter vs. AdapterFusion
6. Concrete Example
7. Limitations
8. Summary

AdapterFusion is a parameter-efficient transfer learning technique proposed in the paper "AdapterFusion: Non-Destructive Task Composition for Transfer Learning" (Pfeiffer et al., 2021).

While standard Adapters (Houlsby et al., 2019) allow a model to learn a specific task by inserting small bottleneck layers, AdapterFusion solves a different problem: How do you combine knowledge from multiple, separate tasks to solve a new, unseen task without forgetting or degrading the original skills?

Here is a breakdown of how it works, the architecture, and why it matters.

The Core Problem: The "Stiffness" of Multi-Task Learning

A naive approach to transfer learning from multiple tasks is Multi-Task Learning (MTL) . You train one model on all tasks at once.

• The Downside: This creates a "stiff" model. To add a new task, you must retrain the entire model. Furthermore, tasks can negatively interfere with each other (negative transfer).

AdapterFusion solves this by treating task-specific knowledge (Adapters) as modules that can be dynamically composed.

The Architecture: A Two-Phase Process

AdapterFusion operates in two distinct phases:

Phase 1: Train the Adapters (Independent)

First, you train a separate Adapter for each source task.

• You freeze the large pre-trained model (e.g., BERT, T5).
• You insert a small, trainable "Adapter" module (a down-projection, non-linearity, and up-projection) into each Transformer layer.
• You train one Adapter for Task A, one for Task B, etc.
• Result: You have a set of "skill modules" stored in these Adapters. The base model remains unchanged.

Phase 2: Train the Fusion Layer

Now you have a new target task (Task C) and you have the frozen Adapters from Tasks A and B.

• The Goal: Learn how to combine the outputs of Adapters A and B to best solve Task C.
• The Mechanism: You introduce a new, small module called the AdapterFusion layer.
  • This layer sits on top of the standard adapter stack.
  • Instead of just taking the output of one adapter, the Fusion layer learns to compute a weighted sum of the outputs of all available adapters.
  • These weights are task-specific and layer-specific.
  • Key Insight: The base model AND the source adapters remain frozen. You only train the tiny Fusion layer.

How the Fusion Layer Works (Technical Detail)

Within a single Transformer layer $l$:

1. The output of the main feed-forward network (FFN) layer is fed into a set of pre-trained Adapters (A & B).
2. Each Adapter outputs its own feature vector (the "skill").
3. The AdapterFusion layer takes these vectors and learns a query-key-value mechanism.
   • It learns a query vector $q_l$ (representing what the current target task needs).
   • It computes a key vector $k_l^{(t)}$ for each source adapter $t$.
   • It calculates an attention weight $s_l^{(t)} = \text{softmax}(q_l \cdot k_l^{(t)})$.
4. The final output of the block is the weighted sum of the adapter outputs: $\sum_{t} s_l^{(t)} \cdot \text{Adapter}_t(\text{context})$.

Why is AdapterFusion Important?

1. Non-Destructive: Adding a new task doesn't modify or damage the knowledge stored for previous tasks. This is a massive advantage over traditional fine-tuning or MTL.
2. Modularity: You can think of Adapters as a "skill library." You can add or remove skills (Adapters) from the library without retraining the base model or other skills.
3. Parameter Efficiency: You only need a very small number of parameters for the Fusion layer (usually < 1% of the original model).
4. High Performance: It can match or even outperform full fine-tuning on target tasks where the source tasks are relevant.

Comparison: Adapter vs. AdapterFusion

Concrete Example

Imagine you have a pre-trained language model (like RoBERTa).

1. Train Adapters: You train a Sentiment Adapter (on movie reviews) and a Grammar Adapter (on grammatical error correction).
2. The Target Task: You need to build a Question Answering system for a legal FAQ.
3. AdapterFusion:
   • You freeze RoBERTa.
   • You freeze the Sentiment Adapter and Grammar Adapter.
   • You insert an AdapterFusion layer.
   • You train the Fusion layer on your legal FAQ dataset.
   • Result: The Fusion layer learns to pay high attention to the Grammar Adapter (for precise legal wording) and low attention to the Sentiment Adapter (since legal text is objective). It effectively combines the relevant "lawyering" skills without losing the original model's general knowledge.

Limitations

• Requires Phase 1: You need a good set of pre-trained adapters. If you don't have relevant source tasks, it's not useful.
• Fusion is a Layer: It only learns a static weighted combination per layer. It doesn't dynamically change which adapter to use based on the input token, just on the task.
• Complexity: While efficient, the system is more architecturally complex to implement than a simple single adapter.

Summary

AdapterFusion is a "meta-learning" or "skill-composition" technique. It treats adapters as Lego bricks representing distinct skills. The Fusion layer is the "instruction manual" that tells the model how to assemble those bricks to best solve a new problem, without gluing them together permanently.