# Spaces Restart mid LLM Fine Tuning

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspaces-restart-mid-llm-fine-tuning%2F98174%3E%5C
**Category:** Spaces\
**Created:** [July 18, 2024, 2:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspaces-restart-mid-llm-fine-tuning%2F98174 "2024-07-18T14:35:19Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![uncucan](https://avatars.discourse-cdn.com/v4/letter/u/ed8c4c/32.png) [@uncucan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Funcucan)%5C
**Post date:** [July 18, 2024, 2:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspaces-restart-mid-llm-fine-tuning%2F98174%2F1 "2024-07-18T14:35:19Z")

</div>

At first attempt I started a fine-tuning job for an 8B LLM and I was checking the logs regularly. My space was using 4xL4s with no persistent storage. Last time I checked the training was 96% done. 2 minutes later I opened my space to check again and saw that the space has restarted without any error logs or anything. I did a little bit of research and realized it might’ve happened because of some storage issue so I upgraded my storage to 150GB and restarted the training.  
After ~16 hours the same thing happened again. The space randomly just restarts without any error message or a log indicating why this has happened.

How am I supposed to debug?

---

<div class="post-metadata">

**Author:** ![chris-rannou](https://avatars.discourse-cdn.com/v4/letter/c/7feea3/32.png) [@chris-rannou](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fchris-rannou)%5C
**Post date:** [July 19, 2024, 10:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspaces-restart-mid-llm-fine-tuning%2F98174%2F2 "2024-07-19T10:59:38Z")

</div>

Hello uncucan,

It looks like this could be an out of m emory issue leading to a restart of the space.

We are currently working on exposing events related to a space so that this kind of errors are clearer.

In any case we cannot guarantee that a space will run without interruption for any arbitrary duration.

A good practice for such training cases is to regularly save checkpoints (in your persistent storage) to resume the training upon restarting.
