We download data from Hugging Face at our CI (GitHub Actions) like this: + git lfs install Updated Git hooks. Git LFS initialized. + git clone https://huggingface.co/datasets/certik/fastGPT Cloning into 'fastGPT'... fatal: unable to access 'https://huggingface.co/datasets/certik/fastGPT/': The requested URL returned error: 403 Quite often in the last month we started encountering “403 Forbidden” and “429 Too Many Requests” errors. We were getting similar errors when using curl over https. We do not want to overload your servers. I have a few questions to figure out how to move forward: Is it ok to use Hugging Face at our CI to download data? If so, what is the rate limit? How should we handle the 403/429 errors — should we retry with exponential delay iterations (1, 2, 4, 8, … seconds)? Should we use some authentication? Thanks for maintaining this service, it has been very useful to us.

SSL configuration errors and token configuration errors are common causes. Also, in Windows environments, it is easier to stabilize git-lfs as well as git by reinstalling them using the installer. The default ones are kind of old. [image] Connection Error when Accessing Dataset URL on Hugging Face 🤗Datasets I am currently experiencing an issue while trying to access my private dataset in my repo for example: (https://huggingface.co/dataset/Mustafa21/dataset) The error message I encountered is as follows: ionError(f"Couldn't reach {url} ({repr(head_error)})") ConnectionError: Couldn't reach https://huggingface.co/datasets/Mustafa21/dataset/resolve/main/dataset_infos.json (ConnectionError('Unauthorized for URL https://huggingface.co/datasets/Mustafa21/dataset/resolve/main/dataset_infos.json. Pleas… [image] Error 403! What to do about it? Beginners I can not save the model, it gives HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/repos/create - You don’t have the rights to create a model under this namespace how to fix?

Thanks @John6666. Note that the repository I am trying to download is public. I nevertheless created a Huggingface token and I can download it using curl and the token like this: curl -f -L -H "Authorization: Bearer ${HUGGINGFACE_TOKEN}" -o model.dat https://huggingface.co/datasets/certik/fastGPT/resolve/main/model_fastgpt_124M_v1.dat I tested that if I use invalid token that this download will fail. So I think it is working when I do it by hand in a terminal. However in order to use this at the CI, I first tried creating a secure variable at GitHub, but it doesn’t get exposed to PRs from forks. I could expose the token I created online, but this doesn’t seem like a good practice even if the token has no permissions, since it is still tied to my account and there might be some other consequences. Also, from Huggingface’s perspective – if there were some special privileges, like not getting download errors for people that use the token, then if I exposed the token publicly, anyone could use the same token, so it would defeat the measure. So that doesn’t seem like a solution.

Hmm… huggingface-cli login? [image] Command Line Interface (CLI) We’re on a journey to advance and democratize artificial intelligence through open source and open science.

The huggingface-cli still requires a token, so at our CI the token would have to be public, which I think is not a good idea. However, I found this page that shows how to download files: Download files from the Hub, maybe it doesn’t fail like a direct curl invocation.

We also tried the following curl options: --retry-all-errors --retry 10 --retry-delay 20, but unfortunately that still sometimes fails, here is an example: fatal: unable to access 'https://huggingface.co/datasets/certik/fastGPT/': The requested URL returned error: 403 opened 11:48PM - 25 Feb 25 UTC (UTC) [image] certik https://github.com/lfortran/lfortran/actions/runs/13531928535/job/37815936445?pr…=6418 ``` + git lfs install Updated Git hooks. Git LFS initialized. + git clone https://huggingface.co/datasets/certik/fastGPT Cloning into 'fastGPT'... fatal: unable to access 'https://huggingface.co/datasets/certik/fastGPT/': The requested URL returned error: 403 Error: Process completed with exit code 128. ``` Background: this `git clone https://huggingface.co/datasets/certik/fastGPT` was introduced in https://github.com/lfortran/lfortran/pull/6406 as a fix for https://github.com/lfortran/lfortran/issues/6318. The error is now much better, it fails at the point where it should fail. What remains to figure out is if we should be using huggingface at all, given that it is not very reliable for our CI. We can also ask at https://github.com/huggingface/huggingface_hub, I found some similar issues there. Maybe we need to authenticate with a token when downloading, we might be getting rate limited. That is over 3 minutes of trying every 20 seconds, and it did not succeed. So I think this is not some transient error, it is likely some rate limit. So I think we should not be downloading from Huggingface using curl at our CI.

For a moment I thought it might be a Windows git problem, but if it’s happening with curl, it’s a different error that’s unrelated… Hmm, I wonder if something is being blocked in the connection path… [image] Proxy Issues While Accessing Hugging Face Beginners Hello, I’m experiencing issues while trying to access Hugging Face from my institute’s network. Here are the details of my problem: Error Message ProxyError: (MaxRetryError(“HTTPSConnectionPool(host=‘huggingface.co’, port=443): Max retries exceeded with url: /api/whoami-v2 (Caused by ProxyError(‘Cannot connect to proxy.’, OSError(‘Tunnel connection failed: 403 Forbidden’)))”)) Steps Taken: Checked and unset the proxy variables. Changed the token key and tried to connect. Attempted to conne… [image] 403 Client Error: Forbidden for url: Models I’m kicking off the batch job from a sagemaker notebook instance, but the batch job runs in sagemaker. It still fails with 403. I tried it from a different aws account with the same results. More detail in this thread Error 403 when downloading model for Sagemaker batch inference - #6 by philschmid For Windows users Git - Install for Windows [image] Git Large File Storage Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like GitHub.com or GitHub Enterprise.

For public datasets in CI you shouldn’t need a token at all, and the 403/429 you’re seeing is the edge throttling the runner’s outbound IP, not your auth. A few things that moved the needle for me in GitHub Actions: Drop git-lfs for downloads and use the Hub’s file download path instead (hf download / huggingface_hub snapshot_download with allow_patterns). git clone pulls every LFS object; you usually want a subset. Set HF_HUB_ENABLE_HF_TRANSFER=1 — the accelerated downloader uses fewer, larger connections, which trips per-connection limits far less than many small curl/LFS requests. Back off on the actual signal: on 429, read Retry-After and sleep that long (plus jitter) rather than a flat --retry-delay 20. A fixed delay is why --retry 10 still intermittently fails. Narrow the scope with allow_patterns so you fetch 3 files, not 3000 — fewer requests, lower chance of tripping the limit. Cache the downloaded files between runs (actions/cache keyed on revision) so only changed shards re-download. If it must stay a single git clone for your workflow, then the shared-IP angle is the real fix: GitHub Actions runners come from a pool, so a busy egress IP is throttled globally and it looks random from your side.

Git clone often gives 403 or 429 errors

Beginners

John6666 March 6, 2025, 10:20pm 2

SSL configuration errors and token configuration errors are common causes. Also, in Windows environments, it is easier to stabilize git-lfs as well as git by reinstalling them using the installer. The default ones are kind of old.

Topic		Replies	Views
Help with DeepSeek-V3-0324 Model Download Intermediate	6	739	July 9, 2026
Rate limited with Pro subscription Beginners	2	173	July 31, 2025
403ed from huggingfacehub 🤗Hub	1	282	September 4, 2025
Error while downloading a repo : Max retries exceeded Beginners	0	1774	November 20, 2022
Private repo wget download not working why? Beginners	6	7081	April 20, 2025

Git clone often gives 403 or 429 errors

Related topics