• 2 Posts
  • 7 Comments
Joined 3 years ago
cake
Cake day: June 11th, 2023

help-circle
  • From the linked webpage readme:

    2.1 Classify. Each file is labelled permissive (at least one permissive license detected, no conflicting non-permissive license), no_license (no licenses detected, or only non-license legal texts such as CLAs), or non_permissive. The permissive allowlist follows the Blue Oak Council list plus licenses categorized as Permissive or Public Domain by ScanCode. Files classified as non_permissive are excluded from both released datasets.

    From https://www.bigcode-project.org/docs/about/the-stack/:

    v1.1: The three copyleft licenses (MPL/EPL/LGPL) were excluded and the list of permissive licenses extended to 193 licenses in total. The list of programming languages was increased from 30 to 358 languages. Also opt-out request submitted by 15.11.2022 were excluded from this ersion of the dataset. The resulting near-deduplicated dataset is 6TB in size.

    So MPL/EPL/LGPL are already not part of the dataset.

    So… why were they in there? Was this added for v1.1?

    “one permissive license” - So if my project includes a lib and I include the license file for that for the license notice…?



  • Noteworthy: They crawled only the default branch HEAD and inlined all source content.

    • The file contents are included inline. The decoded UTF-8 source text is embedded directly in the dataset, so it is fully self-contained — you can start training the moment the download finishes.
    • It reflects the state of GitHub in August 2025. The corpus is a direct crawl of GitHub repositories at their default-branch HEAD, capturing roughly two additional years of open-source code compared to The Stack v2.

  • We want to give developers agency over their source code by letting them decide whether or not it should be used to develop and evaluate machine learning models.

    crawled directly from GitHub and built to pre-train code LLMs with full-repository context

    Repositories that opted out are removed from the dataset before each patch release.

    “agency”

    Which AI company will not use v1 which has all of the data but will use later patch releases instead which have less data?



  • Unfortunately, these crawlers instead try to read every single page from Codeberg, no matter if it makes sense. This includes all the different issue filter variants, Git history, as well as the actual files at any point in Git history - even if they are still equal.

    I’m surprised they don’t lock some things behind account logins. Is it that hard to decide what would be acceptable to no longer serve without an account?

    Does the full commit and change history have to be available without an account, under these circumstances? If we think about what would be minimally enough:

    • Current branch head tree
    • Tag trees (you can link to release source state and [potentially/manually] compare between releases)

    Not having a change log history nor code change diff seems like a big loss to me, but you have to draw the line somewhere, and that seems acceptable to me.

    To me, Codeberg is in a better position to do so than smaller instances. I already have an account because it has many [relevant/significant] projects, is a home of public good, and is under an appropriate org.