AI Prompt Engineering Is Dead

Abstraction: Automated prompt optimization outperforms human prompt engineering

Key points:

• VMware's Rick Battle and Teja Gollapudi tested 3 open-source LLMs with 60 prompt combos on grade-school math; found "the only real trend may be no trend" — chain-of-thought sometimes helped, sometimes hurt; odd positive primers ("This will be fun") occasionally boosted performance.
• Automatic prompt-optimization tools (given examples + a scoring metric) beat the best hand-tuned prompts in almost every case, and in hours instead of days.
• Auto-generated prompts were bizarre and non-human — one top math prompt was an extended Star Trek reference ("Command, we need you to plot a course through this turbulence...").
• Intel Labs (led by Vasudev Lal) built NeuroPrompts: trains an LLM to expand simple prompts into expert ones, tuned via reinforcement learning against PickScore for Stable Diffusion XL; beat human-expert prompts.
• Counterpoint: prompt engineering survives rebranded — Red Hat's Tim Cramer and ex-Microsoft's Austin Henley note productionizing LLMs (reliability, testing, safety, compliance) is hard; the emerging role is LLMOps, evolving from MLOps. Article ran May 2024 as "Don't Start a Career as an AI Prompt Engineer."

Connections: Prompt Engineering · Automatic Prompt Optimization · Vmware · Intel Labs · Llmops

Source: https://spectrum.ieee.org/prompt-engineering-is-dead