Attention Meets Perturbations: Robust and Interpretable Attention with Adversarial Training

Shunsuke Kitada and Hitoshi Iyatomi

Attention mechanisms¹ are widely applied in natural language processing (NLP) field through deep neural networks (DNNs). As the effectiveness of attention mechanisms became apparent in various tasks^{2 3 4 5 6 7}, they were applied not only to recurrent neural networks (RNNs) but also to convolutional neural networks (CNNs). Moreover, Transformers ⁸ which make proactive use of attention mechanisms have also achieved excellent results. However, it has been pointed out that DNN models tend to be locally unstable, and even tiny perturbations to the original inputs ⁹ or attention mechanisms can mislead the models ¹⁰. Specifically, Jain and Wallace ¹⁰ used a practical bi-directional RNN (BiRNN) model to investigate the effect of attention mechanisms and reported that learned attention weights based on the model are vulnerable to perturbations.

To tackle the models' vulnerability to perturbation, Goodfellow et al.¹¹ proposed adversarial training (AT) that increases robustness by adding adversarial perturbations to the input and the training technique forcing the model to address its difficulties. Previous studies^{11 12} in the image recognition field have theoretically explained the regularization effect of AT and shown that it improves the robustness of the model for unseen images.

In this paper, we propose a new general training technique for attention mechanisms based on AT, called adversarial training for attention (Attention AT) and more interpretable adversarial training for attention (Attention iAT). The proposed techniques are the first attempt to employ AT for attention mechanisms. The proposed Attention AT/iAT is expected to improve the robustness and the interpretability of the model by appropriately overcoming the adversarial perturbations to attention mechanisms ^{13 14 15}. Because our proposed AT techniques for attention mechanisms is model-independent and a general technique, it can be applied to various DNN models (e.g., RNN and CNN) with attention mechanisms. Our technique can also be applied to any similarity functions for attention mechanisms, e.g, additive function ¹ and scaled dot-product function ⁸, which is famous for calculating the similarity in attention mechanisms.

To demonstrate the effects of these techniques, we evaluated them compared to several other state-of-the-art AT-based techniques ^{16 17} with ten common datasets for different NLP tasks. These datasets included binary classification (BC), question answering (QA), and natural language inference (NLI). We also evaluated how the attention weights obtained through the proposed AT technique agreed with the word importance calculated by the gradients ¹⁸. Evaluating the proposed techniques, we obtained the following findings concerning AT for attention mechanisms in NLP:

• AT for attention mechanisms improves the prediction performance of various NLP tasks.
• AT for attention mechanisms helps the model learn cleaner attention and demonstrates a stronger correlation with the word importance calculated from the model gradients.
• The proposed training techniques are much less independent concerning perturbation size in AT.

Especially, our Attention iAT demonstrated the best performance in nine out of ten tasks and more interpretable attention, i.e., resulting attention weight correlated more strongly with the gradient-based word importance.

Common Model Architectures

Illustration of the base models to apply our proposed training technique: (a) a single sequence model for the binary classification (BC) task and (b) a pair sequence model for question answering (QA) and natural language inference (NLI) tasks. In (a), the input of the model is word embeddings, {${\bf w_1}$, $\cdots$, ${\bf w_{T_S}}$} associated with the input sentence $X_S$. In (b), the inputs are word embeddings {${\bf w_{1}^{(p)}}$, $\cdots$, ${\bf w_{T_P}^{(p)}}$} and {${\bf w_{1}^{(q)}}$, $\cdots$, ${\bf w_{T_Q}^{(q)}}$} from two input sequences, $X_P$ and $X_Q$, respectively. These inputs are encoded into hidden states through a bi-directional RNN (BiRNN) model. In conventional models, perturbation ${\bf r}$ is added to the hidden state of the words ${\bf h}$. In our adversarial training for attention mechanisms, we compute and add the worst-case perturbation ${\bf r}$ to attention ${\bf a}$ to improve both prediction performance and the interpretability of the model.

Visualization

The following tables are visualizations of the attention weight for each word and gradient-based word importance in the Stanford Sentiment Treebank (SST) ¹ test dataset.

Vanilla vs. Attention AT vs. Attention iAT

Attention AT yielded clearer attention compared to the Vanilla model or Attention iAT. Specifically, Attention AT tended to strongly focus attention on a few words.

Attention and Gradient

Regarding the correlation of word importance based on attention weights and gradient-based word importance, Attention iAT demonstrated higher similarities than other models.

Vanilla

Attention AT

Attention iAT

Conclusion

We proposed robust and interpretable attention training techniques that exploit AT. In the experiments with various NLP tasks, we confirmed that AT for attention mechanisms achieves better performance than techniques using AT for word embedding in terms of the prediction performance and the interpretability of the model. Specifically, the Attention iAT technique introduced adversarial perturbations that emphasized differences in the importance of words in a sentence and combined high accuracy with interpretable attention, which was more strongly correlated with the gradient-based method of word importance. The proposed technique could be applied to various models and NLP tasks. This paper provides strong support and motivation for utilizing AT with attention mechanisms in NLP tasks.

In the experiment, we demonstrated the effectiveness of the proposed techniques for RNN models that are reported to be vulnerable to attention mechanisms, but we will confirm the effectiveness of the proposed technique for large language models with attention mechanisms such as Transforme ¹ or BERT ² in the future. Because the proposed techniques are model-independent and general techniques for attention mechanisms, we can expect they will improve predictability and the interpretability for language models.