openPangu 2.0 Flash targets long context and lower inference cost
openPangu 2.0 Flash is an MoE model trained on Ascend hardware. It has 92 billion total parameters, but only 6 billion are used when producing an answer. Its is 512,000 tokens, and its pretraining used 34 trillion tokens.
The model mixes several attention methods to reduce compute, memory use, and memory access cost during long-context inference. SWA handles nearby information, while DSA is used to capture important information spread across a wider context. The model also uses , with three heads that draft three extra tokens at each step, so answers can be generated faster.
After pretraining, it was trained with slow and fast thinking behavior, several specialists, and .
Key points
- The model has 92 billion total parameters and 6 billion .
- It supports a 512,000-token .
- Its attention design aims to lower compute and memory costs for long-context inference.
- drafts three extra tokens per step to speed up generation.
- It is relevant for AI agents that need to process large amounts of text at lower cost.