Meet 'Stack,' A 3TB of Permissively Licensed Source Code for LLMs (Large Language Models)

Abstraction: BigCode releases 3TB permissively licensed code dataset for LLMs

Key points:

• The BigCode project (ServiceNow + Hugging Face) released The Stack: 3.1 TB of permissively licensed source code in 30 programming languages
• Dataset was built from 137.36M GitHub repositories; most common licenses were MIT and Apache 2.0, detected via GHArchive and go-license-detector
• Training 350M decoder-only transformers on Python data showed near-deduplication substantially improves model performance
• Developers can opt out of having their repositories included via a dedicated portal at huggingface.co/spaces/bigcode/in-the-stack using their GitHub username
• The Stack is more than 3x the size of the comparable CodeParrot dataset; compared favorably against AlphaCode, CodeGen, and PolyCoder
• Released with a Data Governance plan to promote transparency in code LLM pretraining data

Connections: Hugging Face · Servicenow · Bigcode · Large Language Models · Code Generation · Training Data

Source: https://www.marktechpost.com/2022/12/16/meet-stack-a-3tb-of-permissively-licensed-source-code-for-llms-large-language-models/