ChatGPT and generative AI are booming, but the costs can be extraordinary

Abstraction: Compute economics of training and serving large language models

Key points:

• Latitude (AI Dungeon) spent ~$200K/month at peak on OpenAI API + AWS; switched to AI21 Labs to cut costs below $100K/month
• Training GPT-3 estimated at $4M+; Meta LLaMA (65B params, smaller than GPT-3) used 1M GPU hours (~$2.4M); Hugging Face Bloom v2 retraining expected to cost up to $10M
• Nvidia A100 GPU costs $10K; Nvidia holds ~95% of the AI chip market; inference differs from training — ChatGPT inference at 100M MAU estimated at $40M/month
• Microsoft Bing AI chatbot requires at least $4B of infrastructure to serve all Bing users
• Structural cost issue: LLM inference requires billions of calculations per response vs. typical web serving; GPUs were designed for training/graphics, not inference spikes
• Nvidia CEO Jensen Huang projects AI will be "a million times" more efficient in 10 years through combined chip, software, and algorithmic improvements

Connections: Openai · Nvidia · Hugging Face · Microsoft · Large Language Models · AI Infrastructure · Model Training

Source: https://www.cnbc.com/2023/03/13/chatgpt-and-generative-ai-are-booming-but-at-a-very-expensive-price.html