---
license: apple-ascl
datasets:
- orai-nlp/ZelaiHandi
- HuggingFaceFW/fineweb
language:
- eu
base_model:
- apple/OpenELM-270M
---

# OpenELM-270M-eu continual

OpenELM 270M for Basque continually pretrained on [ZelaHandi-v1](https://huggingface.co/datasets/orai-nlp/ZelaiHandi) for 5 epochs.

📝 Paper: [Sub-1B Language Models for Low-Resource Languages: Training Strategies and Insights for Basque](https://aclanthology.org/2025.mrl-main.35/) accepted in the [5TH MULTILINGUAL REPRESENTATION LEARNING (MRL) WORKSHOP 2025](https://sigtyp.github.io/ws2025-mrl.html) (EMNLP)


## Acknowledgments 

The creation of this dataset has been partially funded by the Basque Government (ICL4LANG project, grant no. KK-2023/00094) and the European Union (EFA 104/01-LINGUATEC IA project, INTERREG POCTEFA 2021-2027 program). 
Pre-training and fine-tuning of SLMs were conducted using the Hyperion system at the Donostia International Physics Center (DIPC).
Finally, we thank Idoia Davila Uzkudun for her contributions to manual data curation and evaluation.

## Citation

If you use this dataset please cite the following paper:

```bibtex
@inproceedings{urbizu2025sub,
  title={Sub-1B Language Models for Low-Resource Languages: Training Strategies and Insights for {B}asque},
  author={Urbizu, Gorka and Corral, Ander and Saralegi, Xabier and San Vicente, I{\~n}aki},
  booktitle={Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025)},
  pages={519--530},
  year={2025}
}

```

## Contact

- Gorka Urbizu (g.urbizu@orai.eus)
- Xabier Saralegi (x.saralegi@orai.eus)