ArXiv Preprint
Demonstration-based learning has shown great potential in stimulating
pretrained language models' ability under limited data scenario. Simply
augmenting the input with some demonstrations can significantly improve
performance on few-shot NER. However, why such demonstrations are beneficial
for the learning process remains unclear since there is no explicit alignment
between the demonstrations and the predictions. In this paper, we design
pathological demonstrations by gradually removing intuitively useful
information from the standard ones to take a deep dive of the robustness of
demonstration-based sequence labeling and show that (1) demonstrations composed
of random tokens still make the model a better few-shot learner; (2) the length
of random demonstrations and the relevance of random tokens are the main
factors affecting the performance; (3) demonstrations increase the confidence
of model predictions on captured superficial patterns. We have publicly
released our code at https://github.com/SALT-NLP/RobustDemo.
Hongxin Zhang, Yanzhe Zhang, Ruiyi Zhang, Diyi Yang
2022-10-19
