Vision Transformer: Farewell Convolutions?

Vision Transformer (ViT) is a pure self-attention-based architecture (Transformer) without CNNs. ViT stays as close as possible to the Transformer architecture that was originally designed for text-based tasks.  One of the most key characteristics of ViT is its extremely simple way of encoding inputs and also using vanilla transformer architecture with no fancy trick. In ViT, […]

Long Range Arena: A Benchmark for Efficient Transformers

Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem (check out this post). However, there was no well-established way on how to evaluate this class of models in a systematic way […]

Marzieh's Thesis Cover

Marzieh Fadaee, who is an old friend of mine, has just defended her PhD thesis, on "Understanding and Enhancing the Use of Context for Machine Translation". Marzieh's thesis is one of best thesis I've ever seen. Super cool stuff! I've designed Marzieh's thesis cover, (of course with her help and great suggestions):  

Efficient Transformers

Transformers has garnered immense interest lately due to their effectiveness across a range of domains like language, vision and reinforcement learning.  The self-attention mechanism is a key defining characteristic of Transformer models. The mechanism can be viewed as a graph-like inductive bias that connects all tokens in a sequence with a relevance-based pooling operation. A […]

MetNet: A Neural Weather Model for Precipitation Forecasting

Weather has an enormous impact on renewable energy and markets, which is expected to reach 80% of the world’s electricity production. There are many social and economic benefits of accurate weather forecasting, from improvements in our daily lives to substantial impacts on agriculture, energy and transportation and to the prevention of human and economic losses […]

PhD Thesis

I have defended my PhD dissertation, with "cum laude" ( highest distinction in the Netherlands), on Friday, February 28, 2020, at 10:00AM in de Agnietenkapel . Thesis My PhD thesis is about "Learning with Imperfect Supervision for Language Understanding". You can download my PhD thesis here, or get it from the UvA repository. Abstract Humans […]