# Llama-2 on colab

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-2-on-colab%2F50013%3E%5C
**Category:** Beginners\
**Created:** [August 8, 2023, 7:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-2-on-colab%2F50013 "2023-08-08T19:25:32Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![suhaaspk](https://avatars.discourse-cdn.com/v4/letter/s/34f0e0/32.png) [@suhaaspk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsuhaaspk)%5C
**Post date:** [August 8, 2023, 7:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-2-on-colab%2F50013%2F1 "2023-08-08T19:25:33Z")

</div>

Hello!

I am trying to download llama-2 for text generation on google colab free version. I tried simply the following

```auto
model_name = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name, token=True)
model = AutoModelForCausalLM.from_pretrained(model_name, token=True)

```

But this gives me an “ran out of RAM” error and the runtime crashes. I noticed that the GPU RAM wasn’t being used and the CPU RAM was going past the limit and causing the runtime to crash. I saw some potential solutions of trying to checkpoint online – I haven’t done this before so I have to learn how but will learn if that is useful. Are there any ways to successfully get this model running on colab. Additionally, as a more general question – How can I predict how much memory it takes to run a specific model?

Any advice is much appreciated. Thank you!

---

<div class="post-metadata">

**Author:** ![obi77](https://avatars.discourse-cdn.com/v4/letter/o/a4c791/32.png) [@obi77](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fobi77)%5C
**Post date:** [August 8, 2023, 8:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-2-on-colab%2F50013%2F2 "2023-08-08T20:47:51Z")

</div>

I think I am having the same issue [Colab RAM Limit Exceeded: Unable to Run 3B Model Even with Quantization](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcolab-ram-limit-exceeded-unable-to-run-3b-model-even-with-quantization%2F50022%2F1) I will share a solution if I find it

---

<div class="post-metadata">

**Author:** ![philippetatel1](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@philippetatel1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphilippetatel1)%5C
**Post date:** [August 9, 2023, 10:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-2-on-colab%2F50013%2F3 "2023-08-09T22:10:00Z")

</div>

by any chance you found something

---

<div class="post-metadata">

**Author:** ![Amansoni](https://avatars.discourse-cdn.com/v4/letter/a/e480ec/32.png) [@Amansoni](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAmansoni)%5C
**Post date:** [November 28, 2023, 4:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-2-on-colab%2F50013%2F4 "2023-11-28T04:50:24Z")

</div>

You can use llama 2 in colab using 4 bit quantization this shorten the memory usage but this will not work without GPU below is the link:

> **[Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA](https://huggingface.co/blog/4bit-transformers-bitsandbytes)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

To use the model below is the main code:

if torch.cuda.is\_available():  
torch.set\_default\_device(‘cuda’)

```
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-chat-hf",
                                            torch_dtype="auto", load_in_4bit=True)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf",
                                          torch_dtype="auto")
tokenizer.use_default_system_prompt = False

```