PaLM 2

Google I/O 2023 successor to PaLM, "significantly smaller than the largest PaLM model but us[ing] more training compute" — the tech report validated Chinchilla's 1:1 compute-optimal scaling at larger compute budgets. Trained with a tuned mixture of objectives (UL2-style) on a mixture spanning 100+ languages, code, and math. Four sizes: Gecko, Otter, Bison, Unicorn (Gecko small enough for on-device use).

Big gains over PaLM on reasoning (BIG-Bench Hard 78.1 CoT vs 65.2) and math (GSM8K 91.0 with self-consistency vs 74.4, "outperform[ing] Minerva and GPT-4"), and C2-level passes on language-proficiency exams in every tested language. Launched powering 25+ products including Bard, Workspace Duet AI, Med-PaLM 2, and Sec-PaLM. Parameters and training tokens officially undisclosed; CNBC (from internal docs) reported ~340B parameters on ~3.6T tokens. Proprietary.