# Domain adaptation from Causal LM to a Seq2Seq model

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdomain-adaptation-from-causal-lm-to-a-seq2seq-model%2F33330%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 8, 2023, 11:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdomain-adaptation-from-causal-lm-to-a-seq2seq-model%2F33330 "2023-03-08T11:09:09Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Johnguitarist](https://avatars.discourse-cdn.com/v4/letter/j/67e7ee/32.png) [@Johnguitarist](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohnguitarist)%5C
**Post date:** [March 8, 2023, 11:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdomain-adaptation-from-causal-lm-to-a-seq2seq-model%2F33330%2F1 "2023-03-08T11:09:09Z")

</div>

Hey all, I’m trying to retrain the CodeGen model to generate code from prompts. It seems the model was initially trained to complete code in a Causal LM domain, similar to how GitHub Copilot works. I want to use this model to generate code from prompts IE: ‘function to perform this operation on a dataframe’ would return the code to perform this. I’m pretty sure that means it would need to be a Seq2Seq LM type, the issue is the Seq2Seq classes for fine-tuning don’t support Codegen as it’s a Causal LM. How can I fine-tune this model to perform a different task while retaining the models knowledge through transfer learning? There doesn’t seem to be any literature on this that I have come across.

Thanks!  
John.
