PanGu-pi

Efficient LLM architecture addressing feature collapse via nonlinearity compensation. Introduces a series-based activation function in FFN and augmented shortcuts in MSA. PanGu-pi-7B achieves comparable performance with 10% inference speed-up. PanGu-pi Pro further optimizes for tiny language models (1B-1.5B), achieving 8.87-point average improvement through tokenizer compression, architecture tweaking, and parameter inheritance.