2 research outputs found
MLP Can Be A Good Transformer Learner
Get PDFSelf-attention mechanism is the key of the Transformer but often criticized
for its computation demands. Previous token pruning works motivate their
methods from the view of computation redundancy but still need to load the full
network and require same memory costs. This paper introduces a novel strategy
that simplifies vision transformers and reduces computational load through the
selective removal of non-essential attention layers, guided by entropy
considerations. We identify that regarding the attention layer in bottom
blocks, their subsequent MLP layers, i.e. two feed-forward layers, can elicit
the same entropy quantity. Meanwhile, the accompanied MLPs are under-exploited
since they exhibit smaller feature entropy compared to those MLPs in the top
blocks. Therefore, we propose to integrate the uninformative attention layers
into their subsequent counterparts by degenerating them into identical mapping,
yielding only MLP in certain transformer blocks. Experimental results on
ImageNet-1k show that the proposed method can remove 40% attention layer of
DeiT-B, improving throughput and memory bound without performance compromise.
Code is available at https://github.com/sihaoevery/lambda_vit.Comment: efficient transforme
Data-driven global ocean modeling for seasonal to decadal prediction
No full textAccurate modeling of ocean dynamics is crucial for enhancing our understanding of complex ocean circulation processes, predicting climate variability, and tackling challenges posed by climate change. Although great efforts have been made to improve traditional numerical models, predicting global ocean variability over multiyear scales remains challenging. Here, we propose ORCA-DL (Oceanic Reliable foreCAst via Deep Learning), a data-driven three-dimensional ocean model for seasonal to decadal prediction of global ocean dynamics. ORCA-DL accurately simulates the three-dimensional structure of global ocean dynamics with high physical consistency and outperforms state-of-the-art numerical models in capturing extreme events, including El Ni & ntilde;o-Southern Oscillation and upper ocean heat waves. Moreover, ORCA-DL stably emulates ocean dynamics at decadal timescales, demonstrating its potential even for skillful decadal predictions and climate projections. Our results demonstrate the high potential of data-driven models for providing efficient and accurate global ocean modeling and prediction
