# Using T5 pre-trained weight for Text style transfer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-t5-pre-trained-weight-for-text-style-transfer%2F4791%3E%5C
**Category:** Models\
**Created:** [March 20, 2021, 4:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-t5-pre-trained-weight-for-text-style-transfer%2F4791 "2021-03-20T16:28:35Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kyoritzu](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@Kyoritzu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKyoritzu)%5C
**Post date:** [March 20, 2021, 4:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-t5-pre-trained-weight-for-text-style-transfer%2F4791%2F1 "2021-03-20T16:28:35Z")

</div>

Hi, I am trying to create a similar model to Riley et al., 2020 ([[2010.03802] TextSETTR: Label-Free Text Style Extraction and Tunable Targeted Restyling](https://arxiv.org/abs/2010.03802)), their model uses the the pre-trained weights for T5 model.

My approach is similar to theirs as I am trying create a model with a encoder decoder structure , which both are initialized with the T5 weights. Similar to Riley´s model mine will also include a “style extractor” which has same structure as the encoder which also needs to be initialized with the T5 weights.

I am able to access the weights using the _from\_pretrained()_ and _state\_dict()_ functions. The problem I am stuck with is loading/initializing my model with the weights. Since the model need to have same structure as the T5 model (from my understanding) to be able to load the weights. Any tips on this front ?

---

<div class="post-metadata">

**Author:** ![kav24](https://avatars.discourse-cdn.com/v4/letter/k/b38774/32.png) [@kav24](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkav24)%5C
**Post date:** [June 25, 2021, 8:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-t5-pre-trained-weight-for-text-style-transfer%2F4791%2F2 "2021-06-25T20:18:17Z")

</div>

Were you able to figure this out? I am trying to implement a similar model and I am having a hard time understanding how the pieces connect.