The Stack v2: Massive Multilingual Code Corpus for AI

The Stack v2, released by the BigCode team, is a gargantuan dataset of source code harvested from over 600 programming languages. Tagged for text-generation tasks, it provides raw code files along with rich metadata (repository identifiers, licensing information, timestamps, star/fork counts, and language detection) stored in parquet format. The collection spans billions of lines of code (size category 1B < n < 10B) and is intended for training large language models that understand and generate code. Access to the full dataset requires a bulk‑download agreement with SoftwareHeritage and INRIA, and users must respect the original licenses of each code snippet, as outlined in the dataset’s gated terms of use.

The dataset is multilingual and includes both crowdsourced and expert‑generated content, making it suitable for research on cross‑language code generation, code summarization, and licensing compliance. Its tabular schema enables efficient filtering and analysis with libraries such as Dask, Polars, and the HuggingFace datasets framework. The inclusion of arXiv references (e.g., 2402.19173) points to accompanying research papers that detail the dataset’s construction and intended applications.