# KoCLIP: Pretraining CLIP on Korean

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280%3E%5C
**Category:** Flax/JAX Projects\
**Created:** [June 24, 2021, 5:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280 "2021-06-24T17:03:42Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![jaketae](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@jaketae](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjaketae)%5C
**Post date:** [June 24, 2021, 5:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280%2F1 "2021-06-24T17:03:42Z")

</div>

# KoCLIP

Building on top of Korean Language models that are publicly available, we want to train a multimodal generative system. Specifically, we train CLIP on Korean datasets using [KoBERT](https://github.com/SKTBrain/KoBERT) and ViT as backbones.

## Model

ViT, KoBERT (or any other Korean encoder LM)

## Datasets

KETI has released a Korean image captioning dataset, available on [AI Hub](https://aihub.or.kr/keti_data_board/visual_intelligence). We can also utilize the [WIT dataset](https://github.com/google-research-datasets/wit), which is a multilingual dataset scraped from Wikipedia.

## Training Script

The training script is (almost) already available, see [here](https://github.com/huggingface/transformers/pull/12285/commits/9ef23f1d911565bd5935451377b7a2298080bb6a).

## Challenges

1. KoBERT likely performs slightly worse than the default English BERT. While this could be a bottleneck, we can always write the script in such a way that it is easy to plug-and-play different LMs. This way, when a better Korean LM is released, it can easily be used.
2. There might be minor architectural adjustments we have to make (e.g. adding projection layers).
3. Attaining fluency in JAX will take time and effort.

## Desired Outcomes

The final deliverable of this project will most likely be an open source repository, accompanying documentation, model weights, and potentially a demo Streamlit app.

---

<div class="post-metadata">

**Author:** ![tree-park](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/tree-park/32/1957_2.png) [@tree-park](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftree-park)%5C
**Post date:** [June 25, 2021, 2:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280%2F4 "2021-06-25T14:14:42Z")

</div>

> [@jaketae](#):
>
> . While this could be a bottleneck, we can always write the script in such a way that it is easy to plug-and-play different LMs. This way, when a better Korean LM is released, it can easily be used

Great! Iet me join this fun project!

---

<div class="post-metadata">

**Author:** ![jaketae](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@jaketae](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjaketae)%5C
**Post date:** [June 25, 2021, 2:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280%2F5 "2021-06-25T14:42:07Z")

</div>

Thanks everyone! Please feel free to share any thoughts you have regarding the direction or details of this project. Looking forward to the next couple of weeks.

---

<div class="post-metadata">

**Author:** ![jaketae](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@jaketae](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjaketae)%5C
**Post date:** [June 26, 2021, 1:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280%2F8 "2021-06-26T13:52:00Z")

</div>

Agreed. I mean I don’t think timezone should ever prevent anyone from joining, but for the purposes of arranging logistics, it would certainly be helpful. I’m also on KST/GMT+9.

---

<div class="post-metadata">

**Author:** ![srisweet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/srisweet/32/1829_2.png) [@srisweet](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsrisweet)%5C
**Post date:** [June 27, 2021, 5:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280%2F10 "2021-06-27T05:31:57Z")

</div>

Hello @jaketae & Team,  
I am interested to be a part of such an amazing project & team. I will try my best to contribute to the Korean version of the CLIP model. It would be nice if we could discuss some learning resources that would be useful for this project. I can work in any time zone that is comfortable for everyone in the team.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 28, 2021, 4:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280%2F11 "2021-06-28T16:34:53Z")

</div>

Great! let’s officially define this project 🙂

Putting everybody in the official sheet [here](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?usp=sharing). More people can still join! Leave a comment here or on the sheet if you want to change something.

---

<div class="post-metadata">

**Author:** ![junhsss](https://avatars.discourse-cdn.com/v4/letter/j/43a26b/32.png) [@junhsss](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjunhsss)%5C
**Post date:** [June 28, 2021, 4:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280%2F12 "2021-06-28T16:56:26Z")

</div>

IMHO aligning Korean encoders with the pre-trained CLIP text encoder will probably suffice. It would be great if we could do better. Count me in 😌 (I’m on GMT+9)

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 28, 2021, 5:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280%2F13 "2021-06-28T17:06:14Z")

</div>

Added you to the team 🙂

---

<div class="post-metadata">

**Author:** ![kyungeun](https://avatars.discourse-cdn.com/v4/letter/k/45deac/32.png) [@kyungeun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkyungeun)%5C
**Post date:** [June 30, 2021, 1:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280%2F15 "2021-06-30T01:08:08Z")

</div>

Hi @jaketae & Team,  
I am interested to be a part of KoCLIP team. I am Korean and will try my best to contribute to this project as possible as i can. It would be nice if i could participate in this project and discuss about it. I can work in any time zone that is comfortable for everyone in the team.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 30, 2021, 9:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkoclip-pretraining-clip-on-korean%2F7280%2F17 "2021-06-30T09:11:48Z")

</div>

Hi @amphora , @kyungeun added you to the team 🙂
