# GPT2 for German poetry generation

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-for-german-poetry-generation%2F7069%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 22, 2021, 3:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-for-german-poetry-generation%2F7069 "2021-06-22T15:54:49Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 22, 2021, 3:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-for-german-poetry-generation%2F7069%2F1 "2021-06-22T15:54:49Z")

</div>

# GPT2 for German poetry generation

For this project, I propose to use a pretrained GPT2 model in German and fine-tune it to learn German poetry.

## Model

A GPT2 model, pretrained in German can be found here: [dbmdz/german-gpt2 · Hugging Face](https://huggingface.co/dbmdz/german-gpt2)

## Datasets

The model can be fine-tuned on the publicly available “Faust” dataset: [mobverdb/faust.txt at master · martinth/mobverdb · GitHub](https://github.com/martinth/mobverdb/blob/master/faust.txt)

## Available training scripts

A training script to fine-tune a GPT2 model in Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py)

## (Optional) Desired project outcome

The desired project output is a GPT2 model that can generate sensible German poetry. This can be showcased directly on the hub or with a streamlit app.

## (Optional) Challenges

The data is written as a dialogue with the speakers names `"MEPHISTOPHELES:"` before every paragraph. One might need to remove such data for better model quality.

## (Optional) Links to read upon

There are lots of English GPT2 models for poetry:

- [https://scottmduda.medium.com/generating-an-edgar-allen-poe-styled-poem-using-gpt-2-289801ded82c](https://scottmduda.medium.com/generating-an-edgar-allen-poe-styled-poem-using-gpt-2-289801ded82c)
- [https://www.gwern.net/GPT-2](https://www.gwern.net/GPT-2)
- [OpenAI’s new language AI is available to try yourself | Vox](https://www.vox.com/2019/5/15/18623134/openai-language-ai-gpt2-poetry-try-it)
