In Knowledge distillation: A good teacher is patient and consistent, Beyer et al. investigate various existing setups for performing knowledge distillation and show that all of them lead to ...

Adaptive Span is a novel self-attention mechanism that can learn its optimal attention span. This allows us to extend significantly the maximum context size used in Transformer, while maintaining ...

Abstract: Deep neural networks (DNNs) have always been a popular base model in many image classification tasks. However, some recent works suggest that there are some man made images will easily lead ...