SAS (Simple Attention Sparsification)

Post-training attention sparsification picks a small set of context blocks per query; existing trainable selectors score blocks with a lightweight router and hard top-k, which blocks the language-modelling gradient, so they fall back to distilling dense attention weights that are not aligned with what matters under a fixed budget. Tencent's HY LLM Frontier team (with HKUST) instead adds continuous gates on the selected blocks so context ranking is optimised end to end by the LM loss, with a Triton kernel on FlashAttention. Released as 33M to 42M-parameter gate packages for Qwen3-4B, 8B, and 14B: at a 1,024-token budget the paper reports gains over SeerAttention-R of 6.0 to 7.7 points on MATH-500 and 10.6 to 15.5 on GPQA Diamond.