# Wikilangs - Open NLP for 340+ Wikipedia Languages 🌐

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwikilangs-open-nlp-for-340-wikipedia-languages%2F174076%3E%5C
**Category:** Languages at Hugging Face\
**Created:** [March 8, 2026, 11:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwikilangs-open-nlp-for-340-wikipedia-languages%2F174076 "2026-03-08T11:18:31Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![omarkamali](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/omarkamali/32/58369_2.png) [@omarkamali](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fomarkamali)%5C
**Post date:** [March 8, 2026, 11:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwikilangs-open-nlp-for-340-wikipedia-languages%2F174076%2F1 "2026-03-08T11:18:31Z")

</div>

Hi everyone! I’m Omar, an ML/NLP researcher from Berlin and I’ve been building **Wikilangs** , open-source NLP infrastructure and models trained on Wikipedia across **340+ languages** , including many that have little to no existing model coverage. 🌎🌍🌏

The project currently provides, for each supported language:

- **Vocabulary** , a word list with usage frequencies

- **Custom tokenizers** , trained on native Wikipedia text per language

- **N-gram language models** , lightweight, fast, usable offline

- **Word embeddings** , cross-lingual vector spaces, monolingual and english-aligned

- **Markov chains** , to generate all the non-sensical text you ever dreamed of

- **Morphological tokenizers** , to make stemming easier (uses an experimental statistical approach)

- And a **Wordle-like game** to kill time while your models are training 🙂

You can explore everything at **[wikilangs.org](https://wikilangs.org/)** or install directly:

```bash
pip install wikilangs

```

The hub page is at 🤗 [`huggingface.co/wikilangs`](https://huggingface.co/wikilangs). Each language has its own model card with download stats, training corpus size, and evaluation notes.

This project builds on my dataset 🤗 [`omarkamali/wikipedia-monthly`](https://huggingface.co/datasets/omarkamali/wikipedia-monthly), which publishes a monthly text corpus for every language on wikipedia (3 years ahead of the official Wikipedia dataset on HF).

* * *

**Why I’m posting here:** I’d love to connect with researchers, engineers, and community members working on NLP for any of these languages, especially low-resource ones. If you’re working on **African languages, Arabic, indigenous languages, creoles and pidgins, or any other underrepresented language** , I’d genuinely love to hear what’s missing, what’s broken, and what would make these resources actually useful for your work.

**A few things I’m actively looking for:**

- Native speakers / language experts who can help validate tokenization quality

- Collaborators interested in building downstream projects (LLMs, search and semantics, morphological tokenization, user-facing apps and games …)

- Feedback on any specific language’s model quality or ideas how to take the project further

Drop a reply and introduce yourself, which language(s) are you working on? 👇
