# Fine-Tune GPT2 for German Rap

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-gpt2-for-german-rap%2F7097%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 23, 2021, 12:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-gpt2-for-german-rap%2F7097 "2021-06-23T12:08:53Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 23, 2021, 12:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-gpt2-for-german-rap%2F7097%2F1 "2021-06-23T12:08:53Z")

</div>

# GPT2 for German Rap

For this project, I propose to use a pretrained GPT2 model in Gerrman and fine-tune it to learn to German rap.

## Model

A GPT2 model, pretrained in German can be found here: [dbmdz/german-gpt2 · Hugging Face](https://huggingface.co/dbmdz/german-gpt2)

## Datasets

The model can be fine-tuned on a variety of publicly available lyrics of rap songs in German. The team can choose whatever lyrics they like.

## Available training scripts

A training script to fine-tune a GPT2 model in Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py)

## (Optional) Desired project outcome

The desired project output is a GPT2 model that can [drop bars](https://hinative.com/en-US/questions/2837028) in German. Ideally the model is able to generate a whole rap song.

## (Optional) Challenges

It might be difficult to find enough data and to homogenize it well.

## (Optional) Links to read upon

There are some nice articles on GPT2 models that can rap:

- [GitHub - ewenme/gpt-2-raps: generate lyrics with GPT-2](https://github.com/ewenme/gpt-2-raps)
- [https://medium.com/analytics-vidhya/create-a-rapping-ai-using-deep-learning-part-2-fine-tune-a-pre-trained-gpt-2-instance-for-a-1eb19ac557c3](https://medium.com/analytics-vidhya/create-a-rapping-ai-using-deep-learning-part-2-fine-tune-a-pre-trained-gpt-2-instance-for-a-1eb19ac557c3)
