Researchers have released NAICS-GH, a dataset of 6,588 GitHub repositories each tagged with a two-digit NAICS industry sector code, validated at 96.98% precision on a human-checked sample [S1]. GitHub hosts hundreds of millions of public repositories — but the platform has never mapped any of them to the industries that actually produce the code [S1]. That blind spot blocks entire fields of research into where open-source innovation comes from and how it spreads. The pipeline that fills it starts with 1.37 million repositories and ends with 6,588 labels you can trust — and the way it gets there is the part worth reading for.

The gap nobody could see

GitHub is the world's largest open-source archive, holding hundreds of millions of public repositories [S1]. Yet the platform exposes no native mapping from those repositories to standardised industry sectors [S1]. You can count stars, languages, commits, and contributors — but you cannot ask a basic question like "how much open-source code comes from finance versus manufacturing?"

The researchers behind NAICS-GH argue this gap limits empirical work on three fronts: understanding where code creation is geographically concentrated, measuring which industries contribute most to open-source output, and tracking how emerging technologies move from one economic sector to another [S1]. In other words, we know a lot about what is being built on GitHub, but almost nothing about which industries are building it.

The funnel: from 1.37 million to 6,588

The team started with approximately 1.37 million source repositories drawn from pools covering the United States, the European Union, and Australia [S1]. Their pipeline — which they describe as a retrieve-and-verify approach — narrowed that pool to 31,178 candidate repository-sector pairs [S1]. From those candidates, only labels scoring at least 8 on a rubric scale were kept, leaving 6,588 high-confidence labels [S1].

The Hugging Face dataset page confirms the corpus spans 19 retained two-digit NAICS 2022 codes across the three regions [P2]. NAICS — the North American Industry Classification System — is the framework that statistical agencies in the US, Canada, and Mexico use to sort every business into sectors like manufacturing, finance, or information technology.

How the labels were made

The pipeline chains three components [S1]:

  • BAAI/bge-large-en embeddings — the system converts each repository's metadata into numerical vectors so it can gauge how closely a repo resembles the textual description of a given NAICS sector.
  • FAISS retrieval — an open-source similarity-search toolkit, originally developed at Facebook, that quickly surfaces the most plausible sector match for each repository.
  • GPT-4.1 rubric scoring — the large language model checks every candidate pairing against a defined rubric and returns a confidence score from 1 to 10.

Re-running the full pipeline end to end reproduces the candidate set to within 0.03 percent [S1] — though that reproducibility hinges on using the exact same model versions, a caveat we'll return to.

The precision test

On a random sample of 2,421 repositories validated by humans, the released labels achieved 96.98 percent precision, with a Wilson 95 percent confidence interval of [96.23, 97.59] [S1]. Precision here means the share of labels that human reviewers agreed with — roughly 97 out of every 100.

The researchers also benchmarked six pretrained encoders on the corpus [S1]. The standout was RoBERTa-large, which reached 86.45 percent F1 and 86.35 percent accuracy on a held-out 20 percent test set [S1]. A fine-tuned version of that classifier — which takes a repository's name, description, topics, and README and predicts its NAICS sector — is publicly available on Hugging Face [P3].

What it means

For the first time, there is a reusable, standardised bridge between the world's largest open-source platform and the industry classification system that governments and statisticians already use.

That bridge makes previously unanswerable questions tractable. Which industries contribute most to open-source? Does a sector's open-source output predict how quickly it adopts new technologies? How does the geography of code production differ between the US, the EU, and Australia?

The dataset is deliberately small and selective. The researchers chose precision over coverage, keeping only labels scored 8 or above. With 6,588 repos drawn from an initial 1.37 million, this is a high-confidence starting point — not a census of GitHub.

Everything is released openly: the dataset, Croissant metadata, pipeline code, prompts, and the fine-tuned RoBERTa-large checkpoint, under CC-BY-4.0 and MIT licenses [S1].

What it means for business

For a two-person analytics consultancy, NAICS-GH and its companion classifier offer something that previously required a custom data-science project: the ability to sort GitHub activity by industry sector without building a labelling pipeline from scratch.

A suburban software agency could use the fine-tuned RoBERTa-large classifier [P3] to categorise repositories by sector, then benchmark its own open-source footprint against peers in the same NAICS category. A policy researcher studying tech diffusion could query which industries are producing AI-related repositories — and whether that concentration looks different in Sydney versus San Francisco.

The practical limit is coverage. With 6,588 labelled repos across 19 sectors, the dataset is too thin for granular analysis of niche industries. It is a foundation — the first map of a much larger territory — not the territory itself.

What we don't know yet

This is a preprint and has not been peer-reviewed [S1]. All performance and methodology claims are self-reported by the authors and may change after review.

The 96.98 percent precision figure comes from a 2,421-repository sample selected and validated by the authors — not an independent audit [S1]. The full 6,588-label corpus has not been externally verified.

The 0.03 percent reproducibility claim depends on using the same versions of BAAI/bge-large-en and GPT-4.1 [S1]. API updates or model version changes could shift results in ways the authors have not tested.

Coverage is limited to three regions — the US, EU, and Australia — and 19 of the possible two-digit NAICS sectors [P2]. Asia, Africa, and South America are absent. And with 6,588 repos from an initial 1.37 million, the corpus tells us about high-confidence labels, not about GitHub as a whole.

The next concrete signal to watch: whether independent researchers can reproduce the pipeline end-to-end, and whether the dataset expands beyond its initial release.

Sources: arXiv preprint, "Industry Classification of GitHub Repositories Using NAICS" [S1]; Hugging Face dataset page, aquiro1994/naics-gh [P2]; Hugging Face model page, aquiro1994/naics-github-classifier [P3].

If this kind of plain-English decode is why you're here, the subscribe button is right where you'd expect it.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.