Technology Innovation Institute Open-Sourced Falcon LLMs: A New AI Model That Uses Only 75 Percent of GPT-3's Training Compute, 40 Percent of Chinchilla's, and 80 Percent of PaLM-62B's

Abstraction: TII open-sources Falcon-40B and Falcon-7B compute-efficient decoder LLMs

Key points:

• Falcon-40B is a decoder-only model trained on 1,000B tokens from RefinedWeb plus curated corpora; outperforms LLaMA, StableLM, RedPajama, and MPT on the OpenLLM Leaderboard
• Falcon-7B uses 7B parameters trained on 1,500B tokens from RefinedWeb; similarly tops comparable open-source models
• Both models incorporate FlashAttention (Dao et al. 2022) and multi-query attention (Shazeer 2019) for inference optimization
• Training compute: 75% of GPT-3, 40% of Chinchilla, 80% of PaLM-62B — significant compute efficiency gains
• Released under TII Falcon LLM License permitting commercial use; Falcon-40B-Instruct variant available for chat/instruction formats
• Developed by the Technology Innovation Institute (TII), a UAE government research organization

Connections: Technology Innovation Institute · Falcon · Large Language Models · Open Source AI · Model Training

Source: https://www.marktechpost.com/2023/05/28/technology-innovation-institute-open-sourced-falcon-llms-a-new-ai-model-that-uses-only-75-percent-of-gpt-3s-training-compute-40-percent-of-chinchillas-and-80-percent-of-palm-62b/