NeurIPS 2023: Key Takeaways From Invited Talks

Abstraction: NeurIPS 2023 invited talk summaries on LLM efficiency, generative AI, and responsible data

Key points:

• Björn Ommer (Stable Diffusion creator, LMU Munich) argued GPU supply grows 9x slower than compute demand; proposed combining diffusion models with ConvNets and flow matching for efficiency rather than scaling alone
• Lora Aroyo (Google Research) introduced "truth by disagreement" — treating rater disagreements as signal; safety experts disagree on 40% of examples and diversity of raters is essential for realistic AI safety evaluation
• Linda Smith (Indiana University) showed children learn 1,000+ object categories in under 16,000 waking hours through learner-controlled input, a constrained developmental curriculum, and coherent learning episodes
• Christopher Ré (Stanford) presented Flash Attention as a database-style memory optimization (blocking and aggressive fusion); introduced S4 as a signal-processing-based architecture potentially more efficient than Transformers for long sequences
• Beyond Scaling panel (Percy Liang/Together AI, Meta's Angela Fan, Google DeepMind's Aakanksha Chowdhery): open models contribute positively to AI safety research; synthetic data increasingly blends with human-curated real data; annotation now requires much more domain expertise than five years ago

Connections: Neurips · Stable Diffusion · Google · Meta · Together AI · Large Language Models · Diffusion Models · Generative AI

Source: https://www.topbots.com/neurips-2023-invited-talks/