# Time to talk about Katana

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftime-to-talk-about-katana%2F191961%3E%5C
**Category:** 🤗Transformers\
**Created:** [November 17, 2026, 2:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftime-to-talk-about-katana%2F191961 "2026-11-17T02:17:48Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![OpenBlade](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/openblade/32/64659_2.png) [@OpenBlade](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOpenBlade)%5C
**Post date:** [November 17, 2026, 2:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftime-to-talk-about-katana%2F191961%2F1 "2026-11-17T02:17:48Z")

</div>

Hey everyone! I’m an independent researcher based in the EU, and for the last 4 months or so, I have been working extremely hard on something a bit different.

While the current industry is pushing token counts and brute forcing logic, I took a structured approach, and it damn near drove me crazy. Anyway, here’s what I’ve built so far (which I also plan on open-source releasing):

Custom Stuff -

- Complete training pipeline (custom optimizer, custom dataloader with chunk logging and randomization, sequence preservation)
- Custom Architecture - Katana 1.1B uses cross-context attention (still WIP, but shows promise), custom tokenizer (65k vocab), gqa, tied embeddings, rope scaling to 1m, a native 4096 ctx window, gaussian activations, gaussian norm modulation, and much more.
- The dataset/corpus - Refined traces from fineweb, finewebedu (40%), code and math (60%), some cross-lingual capabilities (Spanish, German, etc)
- The reason for everything custom: This had to fit on my single rtx 3060 12gb, be fast enough to be viable, and also be large enough to handle fine tuning or tasks.

So all in all, Katana has 1B non embedding weights + 100M tied embeddings (the 65k vocab), with 36 total layers to allow it to really reason through and through. Katana has only been training for roughly 2 months now, but has managed to see almost 11B total tokens (out of 84B total planned).

I plan to release this in the following stages:

- Avalanche optimizer (99.99% Vram reduction and upto 10x larger batches/faster training) - Already released, check it out on github.
- Katana 1.0 Base (around 11B tok seen) - Planned for 1st of December 2026.
- iRun Studio Update for Katana Support - Planned for second week of December.

- Katana Instruct - Planned for around mid December.

Please keep in mind I’m a solo researcher/developer, and I do also make mistakes, my research and optimizer/information is all available for free on my websites. Please share your feedback as it all helps me, and please try out avalanche optimizer if you haven’t already.

(P.S. I am currently also working on Katana’s weight space editing and trajectories, and all that research will also be available for free to use).

Have a nice day, and thanks for reading!

Openblade: [www.openblade.org](http://www.openblade.org)  
iRun Studio: www.irun.studio

---

<div class="post-metadata">

**Author:** ![discourse\_ai\_spam](https://avatars.discourse-cdn.com/v4/letter/d/c68b51/32.png) [@discourse\_ai\_spam](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdiscourse_ai_spam)%5C
**Post date:** [November 17, 2026, 2:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftime-to-talk-about-katana%2F191961%2F2 "2026-11-17T02:17:51Z")

</div>

Unlisted on November 17, 2026, 2:17am UTC



---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [November 18, 2026, 12:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftime-to-talk-about-katana%2F191961%2F3 "2026-11-18T12:00:18Z")

</div>

Listed on November 18, 2026, 12:00pm UTC



---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 19, 2026, 10:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftime-to-talk-about-katana%2F191961%2F4 "2026-11-19T10:51:54Z")

</div>

For now, I tried the publicly available Avalanche optimizer on a small language model:

* * *

Thanks for sharing Katana and for releasing Avalanche separately. The single-RTX-3060, from-scratch constraint makes the combination of optimizer, model, and training pipeline interesting. Since you invited feedback and specifically asked people to try Avalanche, I started with something I could actually measure independently: a small, real autoregressive language-model run. The earlier [Avalanche release thread](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frelease-avalanche-ultra-low-vram-matrix-factorized-optimizer-for-pytorch-up-to-99-99-memory-reduction%2F191097) also asked for training-loss curves.

**The immediate result is encouraging, within a deliberately small scope.** The public optimizer in `mode="none"` completed all of my test runs, and both training and held-out loss decreased. Its persistent optimizer tensors occupied much less memory than PyTorch AdamW’s. The learning-rate comparison was mixed, which makes this a useful functional test rather than evidence that either optimizer has better general convergence.

This was a **1,457,856-parameter** causal, byte-level decoder trained on TinyShakespeare, on a Colab **NVIDIA L4** , with **PyTorch `2.14.1+cu130`**. I used the same initial weights, preselected training windows, and fixed held-out windows for four runs of **144 optimizer steps**. The measured held-out loss is natural-log loss _per predicted byte_; it is not Katana’s tokenizer-level perplexity.

| Optimizer | LR | Held-out loss, step 0 | Held-out loss, step 144 | Persistent optimizer tensor state |
| --- | --- | --- | --- | --- |
| Avalanche `mode="none"` | `3e-4` | 5.6627 | **2.5706** | **73,216 B** |
| Avalanche `mode="none"` | `1e-3` | 5.6627 | **2.5607** | **73,216 B** |
| AdamW | `3e-4` | 5.6627 | **2.5912** | 11,663,012 B |
| AdamW | `1e-3` | 5.6627 | **2.5230** | 11,663,012 B |

So the public `none` mode did more than pass an `optimizer.step()` smoke test: it moved held-out loss substantially in this toy LM. **AdamW used about 159.3 times as much persistent optimizer tensor state** here, or equivalently Avalanche used **about 99.37% less**. At `3e-4` the final Avalanche value was slightly lower; at `1e-3` the AdamW value was lower. Two learning rates, 144 steps, and one seed are not an optimizer leaderboard. Also, the update recipes differ: Avalanche `none` uses zero first-moment momentum; AdamW here used `beta1=0.9`.

I made one additional, **separate 24-step** GPU-memory check to investigate unequal baselines in the first notebook. Among the later, baseline-stable runs, the same-`1e-3` comparison showed **74.61 MiB for Avalanche versus 85.66 MiB for AdamW** in _PyTorch peak allocated memory_: **11.05 MiB** , or approximately **12.9% of AdamW’s peak**. The first/cold arm was not baseline-stable, so the full four-arm memory series did **not** pass the common-baseline check. The state reduction and allocated-peak reduction have different denominators; neither establishes a percentage reduction in total VRAM for Katana on a 3060.

**The released zero-momentum mode trains a small LM with much less optimizer state; large-model fit, sustained speed, and model quality are separate questions.** The next measurement depends on the goal:

| Goal | Smallest useful route |
| --- | --- |
| Check that Avalanche trains | Pin the public source, run a short real LM with a fixed held-out set, and share the curve. This is what I tested. |
| Compare memory fairly | Count persistent state; separately measure full-step allocator peak from equal live baselines. |
| Understand the **12 GB** training result | Profile the actual Katana architecture, precision, sequence length, loss head, checkpointing, and optimizer _together_. |
| Compare training efficiency | Use warmed-up end-to-end tokens/s **and** loss versus tokens, under stated batch/precision settings. |
| Check long-run continuity | If exact restart matters, compare uninterrupted training with a fresh-process save/reload path. |
| Evaluate Katana itself | When its weights/tokenizer are available, verify the base-model interface and a few controlled outputs before broad benchmarks. |

I have the CSV loss records and plots from these runs if a closer look at the trajectories would be useful. Here are the conditions and references.

> **The public source and the small training experiment**
>
> ### Exactly which code and environment?
> 
> I tested the public [OpenBladeAI/Avalanche repository](https://github.com/OpenBladeAI/Avalanche) at [commit `3cdf9fa76620ef144a09d80210d4f294e896d36b`](https://github.com/OpenBladeAI/Avalanche/tree/3cdf9fa76620ef144a09d80210d4f294e896d36b), importing the pinned [optimizer.py](https://github.com/OpenBladeAI/Avalanche/blob/3cdf9fa76620ef144a09d80210d4f294e896d36b/optimizer.py). The file’s SHA-256 was:
> 
> ```plaintext
> 1e672c26581e1ac8158a8be9f3779ceabe1965dfe4de0ee6db644ec7b08dbd2c
> 
> ```
> 
> The Colab test fetched that file, checked its hash, and used a normal PyTorch loop without a CUDA extension or GitHub API polling. The [Katana report](https://www.openblade.org/report) describes a project-specific `GPUFactorizedAdamV2` and custom trainer. **This test covers the pinned public implementation, not the complete trainer.**
> 
> The actual runtime reported **Python `3.13.16`** , **PyTorch `2.14.1+cu130`** , CUDA build 13.0, and an **NVIDIA L4**. I did not run this test on an RTX 3060, and I did not load the 1.12B Katana architecture. Recording the PyTorch build matters for reproducing CUDA allocation behavior; recording the Git revision matters for reproducing optimizer behavior.
> 
> ### Model, data and held-out contract
> 
> - Causal decoder-only next-byte model: **3 transformer layers** , width **192** , **4 attention heads** , **1,457,856 trainable parameters** , with standard embeddings, LayerNorm, GELU feed-forward layers, and [PyTorch scaled-dot-product attention](https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html).
> - Byte vocabulary **256** , context **128** , microbatch **8** ; FP32, dropout `0`, weight decay `0`, norm clipping `1.0`, no AMP scaler or separate precision master copy.
> - [TinyShakespeare raw input](https://github.com/karpathy/char-rnn/blob/master/data/tinyshakespeare/input.txt), treated as UTF-8 bytes, **1,115,394 bytes** ; SHA-256 `86c4e6aa9db7c042ec79f339dcb96d42b0075e16b8fc2e86bf0ca57e2dc565ed`.
> - Contiguous split: first **90%** of bytes for training, final **10%** held out. Each evaluation used the same four preselected batches of eight 128-byte targets: **4,096 predicted bytes** in total. This is a fixed sampled estimate, not a full-corpus held-out sweep or a document-level independence guarantee.
> 
> This is a cheap learning check, not the report’s code/math corpus. Its byte-level loss is not comparable numerically to Katana’s tokenizer-level result.
> 
> ### The four arms
> 
> ```plaintext
> Avalanche: mode="none", beta1=0.0, beta2=0.999
> AdamW: beta1=0.9, beta2=0.999, foreach=False, fused=False
> Both: FP32, weight_decay=0, grad clipping=1.0
> LRs: 3e-4 and 1e-3 for each optimizer
> Steps: 144 per arm; held-out eval at 0,24,48,72,96,120,144
> Seed: 20261119
> 
> ```
> 
> I copied the _same initialized weights_ into each arm. A dedicated RNG precomputed training-window positions; every arm saw that same ordered window list. Held-out positions were fixed too. All four runs returned finite losses. This controls the most obvious data-order and initialization differences, but it does **not** isolate the effect of factorization alone, because first-moment behavior differs; nor does it provide independently tuned optimal schedules for either algorithm.
> 
> For a quick reproduction: pin the commit, use a small public LM dataset, fix initial weights and sample order, compare `none` with an explicit baseline at two LRs, and log held-out loss plus state tensor bytes. Record dtype, shape, split, betas/LR, clipping, seed and evaluation cadence. A real backward pass is more informative than a synthetic constant-gradient step without becoming a major study.

> **Reading the short training curves without overclaiming**
>
> ### What changed during the run?
> 
> All four arms started at **5.6626557 nats per byte** on the same held-out windows. Each produced **144 per-step training losses** and **seven held-out checkpoints** , or **604 CSV records** overall. All values remained finite; the drop to final held-out losses between **2.5230 and 2.5912** is the main positive observation.
> 
> The comparison at _matched tested LR_ is worth showing explicitly:
> 
> | Learning rate | Avalanche step 144 | AdamW step 144 | Observation |
> | --- | --- | --- | --- |
> | `3e-4` | **2.5706** | 2.5912 | Avalanche lower by about 0.0206 |
> | `1e-3` | 2.5607 | **2.5230** | AdamW lower by about 0.0377 |
> 
> At `1e-3`, step-120 losses were slightly lower than the endpoints: Avalanche **2.5542** , AdamW **2.5160**. At `3e-4`, the lowest recorded checkpoint was step 144. These are repeated measurements on fixed windows, not independent test sets or a divergence diagnosis.
> 
> ### What does this comparison answer?
> 
> It answers whether this public zero-momentum recipe can train a small transformer with ordinary autograd, and how much persistent state it uses. It does **not** isolate the effect of factorization on convergence: `none` removes first-moment momentum but the AdamW control retains it; there are only **two LRs** , **one seed** , **144 steps** , and a toy literary-byte task rather than Katana’s architecture and corpus.
> 
> If convergence becomes the goal, a **practical recipe comparison** could allow equal optimizer-specific tuning budgets and compare loss at equal tokens or time. An **update-rule ablation** would instead match momentum and other choices to isolate factored statistics. Neither is required to recognize the functional result.
> 
> An extra seed, modestly longer curve, or small LR scan would test the early crossover if anyone wants to pursue it. That is optional; the release thread’s request for an independent loss curve is already addressed.
> 
> ### Why the loss unit matters
> 
> These numbers are the mean natural-log negative log-likelihood over **byte targets**. Katana uses a custom approximately **65k-token vocabulary** , a different text distribution, and a far larger context. Exponentiating my per-byte loss would yield _per-byte_ perplexity, not a metric directly comparable to Katana’s _per-token_ perplexity. The [Hugging Face perplexity guide](https://huggingface.co/docs/transformers/perplexity) emphasizes that tokenization and evaluation-window strategy are part of the metric.
> 
> For eventual cross-tokenizer evaluation, use the **same raw evaluation text** , specify the scoring window/stride and boundaries, and consider a carefully defined byte- or character-normalized NLL alongside task-specific correctness. For this experiment, the contiguous 90/10 split and fixed 4,096-byte held-out sample are enough for a local early-training sanity check, not a claim about code correctness or broad language modeling. That distinction protects the useful result from an accidental comparison it was never designed to make.

> **Persistent optimizer state versus whole-step GPU memory**
>
> The memory improvement is real for the measured component, but the word _memory_ needs a label. Optimizer state, model weights, gradients, backward activations, temporary update buffers, CUDA library workspaces, and total device consumption are not interchangeable. An optimizer can reduce its resident state dramatically while leaving many shared costs of the training step intact. This is particularly relevant to Katana’s **12 GB** constraint: saving the optimizer moments makes space for the other parts of the step, but it does not remove those parts.
> 
> ### The state census
> 
> After training began, I enumerated the tensors in `optimizer.state`, counted `numel() * element_size()`, and excluded references to the model’s existing `Parameter` objects. This last exclusion matters because the pinned implementation can keep parameter references in state entries. A reference to an existing weight is not a second independently allocated optimizer tensor.
> 
> ```python
> # Core counting rule, abbreviated from the actual experiment.
> parameter_ids = {id(p) for p in model.parameters()}
> seen = set()
> total = 0
> for state in optimizer.state.values():
> for value in state.values():
> if isinstance(value, torch.Tensor):
> if id(value) not in parameter_ids and id(value) not in seen:
> total += value.numel() * value.element_size()
> seen.add(id(value))
> 
> ```
> 
> The results were **73,216 B (0.0698 MiB)** for Avalanche `none` and **11,663,012 B (11.12 MiB)** for AdamW. The **11,589,796 B** difference is **159.3x** in state ratio or **99.37%** less auxiliary tensor payload. This excludes Python/allocator overhead, serialized checkpoints and CUDA libraries. A general counter should also handle distinct tensor views sharing storage.
> 
> For large 2D matrices, factorized second moments change the shape of a statistic from approximately `M*N` stored elements to row/column summaries scaling as `M+N`. That is the conceptual attraction shared by [Adafactor](https://proceedings.mlr.press/v80/shazeer18a.html) and Avalanche’s public `none` mode. Actual counts also depend on vectors, embedding shapes, tied/shared parameters, precision, and whether momentum or error-feedback residuals are enabled. I measured **none** , not Avalanche’s 4-bit, 8-bit, BF16 or other modes. Their extra state must be counted separately rather than inheriting this state ratio.
> 
> ### A different 24-step measurement
> 
> The first, 144-step notebook was good evidence for _learning_ and persistent tensor size, but its sequential GPU-memory baselines differed. I therefore did not use its four raw peaks as a fair allocator-peak ranking. I ran a **new 24-step** test with each arm in a separate Python function scope, tracking live allocated bytes before the arm, the model-loaded baseline, the peak over training and evaluation, and live bytes after cleanup.
> 
> | Execution order in 24-step run | Before arm, B | With model, B | Peak allocated, B | After cleanup, B |
> | --- | --- | --- | --- | --- |
> | Avalanche `3e-4`, first/cold | 8,925,184 | 14,762,496 | 78,234,624 | 25,964,544 |
> | Avalanche `1e-3` | **25,964,544** | **31,801,856** | **78,234,624** | **25,964,544** |
> | AdamW `3e-4` | **25,964,544** | **31,801,856** | **89,824,768** | **25,964,544** |
> | AdamW `1e-3` | **25,964,544** | **31,801,856** | **89,824,768** | **25,964,544** |
> 
> The first arm ended above the memory floor from which it began. **The later three arms** had identical before-arm floors, identical model baselines, and no further post-cleanup increase. The experiment-wide flag `comparison_baselines_clean` was therefore **false** , not a four-arm pass. Within that stable subset, however, the two `1e-3` arms have a comparable baseline and workload:
> 
> ```plaintext
> Avalanche: 78,234,624 B = 74.61 MiB peak allocated
> AdamW: 89,824,768 B = 85.66 MiB peak allocated
> Delta: 11,590,144 B = 11.053 MiB, about 12.9% of AdamW peak
> 
> ```
> 
> The peak delta exceeds the state delta by just **348 B** , consistent with state accounting for much of this toy-run difference, but **not** a per-allocation trace. PyTorch allocated peak is distinct from reserved memory or total VRAM; see its [CUDA memory guide](https://docs.pytorch.org/docs/2.14/torch_cuda_memory.html).
> 
> For a reader comparing memory percentages, I would keep three separate sentences rather than one spectacular number: **99.37% less persistent optimizer tensor state** in this tiny `none` experiment; **about 12.9% less PyTorch peak allocated memory** in the matched, baseline-stable `1e-3` subset; and **no independent measurement of Katana’s full RTX 3060 process memory** here. All three can be true simultaneously. The separate metric labels strengthen rather than weaken the concrete observation.

> **Why the first CUDA memory baseline moved**
>
> This is a measurement caveat, **not a problem I observed in Avalanche or Katana**. The first arm of the separate memory run began with **8,925,184 B** of live PyTorch allocations and, even after the model and temporary objects were cleaned up, left **25,964,544 B** live. That once-only increase was **17,039,360 B, exactly 16.25 MiB**. The next three arms started and ended at the same higher floor, with the same model-loaded baseline. It made sense to compare the later stable arms and explicitly mark the four-arm check as unclean.
> 
> There is a plausible _version-specific_ explanation. [PyTorch 2.14’s CUDA documentation](https://docs.pytorch.org/docs/2.14/notes/cuda.html#cublas-workspaces) describes a default cuBLAS workspace arrangement equivalent to:
> 
> ```plaintext
> CUBLAS_WORKSPACE_CONFIG=:4096:2:16:8
> One workspace: (4096 KiB * 2) + (16 KiB * 8) = 8.125 MiB
> Two workspaces: 16.25 MiB = 17,039,360 bytes
> 
> ```
> 
> The retained amount matches two such workspaces _numerically_. A [PyTorch Forums CUDA-memory discussion](https://discuss.pytorch.org/t/help-with-cuda-memory-allocation-during-forward-linear/190797) describes a related lazy-workspace behavior. That is a strong candidate for why the first ordinary forward/backward workload changed the baseline. But I did **not** trace these allocations to particular cuBLAS handles in this run, so I would call it a likely cold-start mechanism, not a proven root cause. The first notebook’s shared function scope could also retain references to earlier computation graphs; the follow-up isolated each arm in its own function. This reduces that possible confounder without proving the source of the earlier baseline drift.
> 
> The easy control, _if precise allocator peaks later become important_, is to run a small unmeasured CUDA warm-up using the same operations, then measure each arm from a matching pre-arm floor and model-loaded baseline. Record `memory_allocated`, the full forward/backward/optimizer/evaluation peak, and post-cleanup live bytes. Resetting a peak counter does not free surviving tensors; `empty_cache()` does not remove tensors still referenced by Python. If a baseline keeps moving, isolated subprocesses or PyTorch’s memory tracing are better discriminators than accepting the table as clean.
> 
> I would not demand this extra run for the present thread. The three-arm stable subset already yields a narrow observation, and the independent **held-out training curve** is the more direct answer to the request for Avalanche feedback. A different PyTorch/CUDA build could initialize library workspaces differently; that’s why I would keep the installed version attached to the 16.25 MiB observation.

> **Related methods: a map of comparisons, not a winner list**
>
> The closest research vocabulary I found for understanding Avalanche is **factored second-moment estimation**. In a large matrix, keeping statistics along rows and columns instead of one full statistic per element can save substantial auxiliary state. [Adafactor](https://proceedings.mlr.press/v80/shazeer18a.html) (Shazeer and Stern, 2018) is a useful conceptual neighbor: it discusses factored second moments, update control, and avoiding conventional first-moment momentum. That connection helps with _understanding the geometry of the state_, but it does not imply that Adafactor and the pinned public Avalanche use identical equations, clipping, initialization or schedules.
> 
> There are also adjacent routes with different design choices: [SM3](https://proceedings.neurips.cc/paper_files/paper/2019/hash/8f1fa0193ca2b5d2fa0695827d8270e9-Abstract.html) uses structured parameter covers for memory-efficient adaptive statistics; [CAME](https://aclanthology.org/2023.acl-long.243/) augments factored ideas with confidence-related mechanisms. [GaLore](https://proceedings.mlr.press/v235/zhao24s.html) reduces optimizer-related memory through low-rank gradient projection during full-parameter training, and [APOLLO](https://proceedings.mlsys.org/paper_files/paper/2025/hash/437bc4ccafd3fc6d4289bd10940be42b-Abstract-Conference.html) explores compressed adaptive scaling. These are **different mechanisms and evaluation setups** , not evidence that one particular implementation duplicates or outperforms another.
> 
> | Reference or mode | What to compare | What cannot be transferred automatically |
> | --- | --- | --- |
> | AdamW | Dense per-parameter moments; familiar learning baseline | Its state denominator and LR behavior differ from `none` |
> | Adafactor | Factored second moments, update clipping/scheduling | Exact formula, handling of vectors and learning dynamics |
> | Public Avalanche `none` | Row/column-style statistics without dense first moment | Not representative of all Avalanche modes or Katana’s private trainer |
> | Avalanche low-bit momentum modes | Packed velocity and any scales/residuals | The tiny `none` state result and its loss curve |
> | SM3, CAME | Alternative ways to retain reduced adaptive statistics | Their estimators and training conditions |
> | GaLore, APOLLO | Projection or compressed scaling during training | Memory-quality-throughput tradeoffs under a different update path |
> 
> The comparison axes are **state bytes, full-step peak, feasible batch, tokens/s, loss versus tokens, and restart behavior**. A state-light optimizer may fit larger batches but spend more time per update; activations may dominate elsewhere.
> 
> For low-bit momentum, a **4-bit packed vector** describes the payload, while scale tensors and optional residual/error-feedback buffers also take space. This is an accounting distinction, not a design objection. The project’s [mode documentation](https://github.com/OpenBladeAI/Avalanche) covers those choices; my test only covers `none`.
> 
> An optional **Adafactor** baseline could distinguish general factorization benefits from implementation choices, with its own documented schedule and precision. But **AdamW plus the observed small LM curve is enough for this first feedback**. These links are a map for future readers, not homework for the author.

> **Condition-based checks for a full single-GPU training system**
>
> The main Katana constraint is unusually concrete: **full-parameter training from scratch on one RTX 3060 with 12 GB** , not merely fitting a quantized inference model. I would keep that objective intact. A memory-efficient optimizer does not need to solve every memory or speed issue itself; it needs to free enough budget for the _rest of the designed training step_. Your [Katana report](https://www.openblade.org/report) already describes **512-token chunked cross-entropy** , **variable-density gradient checkpointing** , and **pinned-memory background prefetching**. Those are real parts of the reported design, not missing optimizations I am recommending from the outside.
> 
> If the goal is **state compression** , the simple census above is sufficient to start. The state footprint depends on the shapes of 2D matrices and smaller vectors, whether embeddings are tied, which modes retain momentum, and the precision of the statistics. A short state-by-layer table at one real update answers that question much more directly than a large end-to-end benchmark.
> 
> If the goal is **whether a full step fits inside 12 GB** , the relevant budget includes weights, gradients, any master copies, stored optimizer state, live activations, the head/loss computation, temporary optimizer update tensors, and library workspaces. In particular, materializing logits for every position over a roughly **65k** vocabulary can be costly. The report’s **512-token loss chunking** addresses precisely that risk; activation checkpointing exchanges compute for lower activation storage. One representative profile can label peaks at the head/loss, backward, and optimizer phases without treating the techniques already present as if they had been overlooked. The actual full-step maximum and whether offload is used answer a different question from a state-only percentage.
> 
> If the goal is **sustained pretraining throughput** , compare warmed-up end-to-end **tokens per second** at the intended sequence length, effective batch, precision, data path and accumulation schedule; pair it with loss versus training tokens. The tiny L4 experiment incidentally recorded about **13.5–13.8 ms per step for Avalanche** and **11.0–11.1 ms for AdamW**. Those order-dependent, tiny-FP32-model timings are **not** a meaningful 3060 throughput comparison or a refutation of any advantage obtained by fitting larger batches. Fit, throughput, and learning progress are separate axes. Lower optimizer-state overhead can improve throughput indirectly by changing a feasible batch configuration even when an isolated update is not faster.
> 
> If the goal is **reliable continuation of a long run** , a small fresh-process resume test is a useful optional control. Exact continuation may depend on model weights, optimizer statistics and step counters, scheduler, AMP scaler if present, RNG, custom dataloader position, and sample order. [PyTorch’s optimizer state-dict documentation](https://docs.pytorch.org/docs/2.14/generated/torch.optim.Optimizer.state_dict.html) describes the generic serialization boundary; a custom trainer may preserve extra state separately. A cheap test is to compare several uninterrupted steps with the same steps split by save, fresh-model/fresh-optimizer reload, and continued identical data. It should be evaluated against the intended tolerance for numerical determinism. This is **not** a claim that Katana currently mishandles checkpoints; I have not seen its internal save/restart path.
> 
> These are independent choices, not a to-do list. The [Hugging Face memory anatomy guide](https://huggingface.co/docs/transformers/model_memory_anatomy) provides useful budget terminology; Katana’s actual design determines which tradeoffs matter.

> **Evaluating Katana itself when a specific checkpoint is available**
>
> I would keep model capability separate from this optimizer feedback. The forum thread and [research report](https://www.openblade.org/report) describe the architecture, token stream, custom corpus and training observations, along with a staged release plan. I have **not independently loaded Katana weights** , so I cannot confirm those generations, the reported 3060 performance, or the model’s code/math competence from my small Avalanche run. A planned public date is also different from a verified downloadable revision.
> 
> When a public base checkpoint and tokenizer can actually be obtained, a useful low-cost path for outside readers would be:
> 
> 1. Pin the **model revision** , architecture/config, custom tokenizer and any required loader code. Confirm a tiny load, forward pass, tensor shapes and basic next-token generation before discussing capability.
> 2. Distinguish **base** from **instruct**. A base model may be better assessed by prompt-prefix continuation or candidate log-likelihood than by treating it as a chat assistant with an assumed chat template.
> 3. Check tokenizer IDs, special tokens and sequence conventions. A custom roughly **65k** tokenizer is an essential part of the interface; replacing it silently changes the experiment.
> 4. Start with a few fixed, short tasks and decoding settings. For code, separate plausible **surface syntax** , parser/compiler acceptance, and **executable functional correctness**. For math, distinguish a well-formed derivation from a checked answer. A tiny unit test or parser can supply an unambiguous signal where a visual impression cannot.
> 5. If checkpoints at different training stages become available, use a consistent small probe set to examine changes in structure and functional behavior over time. [Pythia’s training-dynamics work](https://proceedings.mlr.press/v202/biderman23a.html) is one useful precedent for this style of analysis, not a demand to publish every Katana checkpoint.
> 
> The report’s idea of early structural acquisition is particularly suited to this separation: forming recognizable signatures, tags, or code layouts may precede robust semantics. That is an interesting trajectory even if the first checkpoint does not yet solve complete programming tasks. A handful of reproducible prefixes, parser results and tiny functional tests could give a future reader a much clearer picture than a single impressive or disappointing generation. It remains a _small probe_, not a broad benchmark or a verdict on the research.
> 
> For comparisons with other 1B-class models, keep the **raw evaluation text, scoring convention, context window, and any task-specific tests** aligned. Token-level perplexity from different tokenizers and corpora is not a straightforward head-to-head ranking. If a larger evaluation later makes sense, [EleutherAI’s lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), [EvalPlus](https://github.com/evalplus/evalplus) for executable code checks, and the [HF model-card guidance](https://huggingface.co/docs/hub/en/model-cards) are reusable starting points, subject to the actual model interface and supported tasks.
> 
> One further boundary is context length. The thread mentions a **native 4,096-token** window and RoPE-related scaling toward a much larger number. A large scaling setting alone is not an observed long-context capability. If long-context behavior is important after release, a few increasing-length retrieval or continuation checks would help locate the validated region, without prejudging the design. The near-term path is simply to use the native, documented interface first.

So my concrete feedback for now is modest but positive: **the released, pinned Avalanche `none` implementation trained a small causal LM with decreasing held-out loss, using dramatically less persistent optimizer tensor state than AdamW in that experiment**. The extra GPU-peak observation is narrower and carefully baseline-conditioned. The links above are ways to extend the question depending on the objective, while preserving Katana’s original single-GPU, from-scratch goal. I hope the observed loss trajectory and explicit measurement conditions are useful to others trying Avalanche or, later, evaluating Katana checkpoints.
