DeepSeek’s conditional memory fixes silent LLM waste: GPU cycles lost to static lookups

Through systematic experiments DeepSeek found the optimal balance between computation and memory with 75% of sparse model ...
IEEE

Spa-L Transformer: Sparse-self attention model of Long short-term memory positional encoding based on long text classification

Abstract: The emergence of Transformer and its derivative models brings new opportunities to tasks of NLP (Natural Language Processing). Transformer is not only a separate model, but also the core of ...