Time to talk about Katana

Hey everyone! I’m an independent researcher based in the EU, and for the last 4 months or so, I have been working extremely hard on something a bit different.

While the current industry is pushing token counts and brute forcing logic, I took a structured approach, and it damn near drove me crazy. Anyway, here’s what I’ve built so far (which I also plan on open-source releasing):

Custom Stuff -

  • Complete training pipeline (custom optimizer, custom dataloader with chunk logging and randomization, sequence preservation)
  • Custom Architecture - Katana 1.1B uses cross-context attention (still WIP, but shows promise), custom tokenizer (65k vocab), gqa, tied embeddings, rope scaling to 1m, a native 4096 ctx window, gaussian activations, gaussian norm modulation, and much more.
  • The dataset/corpus - Refined traces from fineweb, finewebedu (40%), code and math (60%), some cross-lingual capabilities (Spanish, German, etc)
  • The reason for everything custom: This had to fit on my single rtx 3060 12gb, be fast enough to be viable, and also be large enough to handle fine tuning or tasks.

So all in all, Katana has 1B non embedding weights + 100M tied embeddings (the 65k vocab), with 36 total layers to allow it to really reason through and through. Katana has only been training for roughly 2 months now, but has managed to see almost 11B total tokens (out of 84B total planned).

I plan to release this in the following stages:

  • Avalanche optimizer (99.99% Vram reduction and upto 10x larger batches/faster training) - Already released, check it out on github.
  • Katana 1.0 Base (around 11B tok seen) - Planned for 1st of December 2026.
  • iRun Studio Update for Katana Support - Planned for second week of December.

- Katana Instruct - Planned for around mid December.

Please keep in mind I’m a solo researcher/developer, and I do also make mistakes, my research and optimizer/information is all available for free on my websites. Please share your feedback as it all helps me, and please try out avalanche optimizer if you haven’t already.

(P.S. I am currently also working on Katana’s weight space editing and trajectories, and all that research will also be available for free to use).

Have a nice day, and thanks for reading!

Openblade: www.openblade.org
iRun Studio: www.irun.studio

For now, I tried the publicly available Avalanche optimizer on a small language model:


Thanks for sharing Katana and for releasing Avalanche separately. The single-RTX-3060, from-scratch constraint makes the combination of optimizer, model, and training pipeline interesting. Since you invited feedback and specifically asked people to try Avalanche, I started with something I could actually measure independently: a small, real autoregressive language-model run. The earlier Avalanche release thread also asked for training-loss curves.

The immediate result is encouraging, within a deliberately small scope. The public optimizer in mode="none" completed all of my test runs, and both training and held-out loss decreased. Its persistent optimizer tensors occupied much less memory than PyTorch AdamW’s. The learning-rate comparison was mixed, which makes this a useful functional test rather than evidence that either optimizer has better general convergence.

This was a 1,457,856-parameter causal, byte-level decoder trained on TinyShakespeare, on a Colab NVIDIA L4, with PyTorch 2.14.1+cu130. I used the same initial weights, preselected training windows, and fixed held-out windows for four runs of 144 optimizer steps. The measured held-out loss is natural-log loss per predicted byte; it is not Katana’s tokenizer-level perplexity.

Optimizer LR Held-out loss, step 0 Held-out loss, step 144 Persistent optimizer tensor state
Avalanche mode="none" 3e-4 5.6627 2.5706 73,216 B
Avalanche mode="none" 1e-3 5.6627 2.5607 73,216 B
AdamW 3e-4 5.6627 2.5912 11,663,012 B
AdamW 1e-3 5.6627 2.5230 11,663,012 B

So the public none mode did more than pass an optimizer.step() smoke test: it moved held-out loss substantially in this toy LM. AdamW used about 159.3 times as much persistent optimizer tensor state here, or equivalently Avalanche used about 99.37% less. At 3e-4 the final Avalanche value was slightly lower; at 1e-3 the AdamW value was lower. Two learning rates, 144 steps, and one seed are not an optimizer leaderboard. Also, the update recipes differ: Avalanche none uses zero first-moment momentum; AdamW here used beta1=0.9.

I made one additional, separate 24-step GPU-memory check to investigate unequal baselines in the first notebook. Among the later, baseline-stable runs, the same-1e-3 comparison showed 74.61 MiB for Avalanche versus 85.66 MiB for AdamW in PyTorch peak allocated memory: 11.05 MiB, or approximately 12.9% of AdamW’s peak. The first/cold arm was not baseline-stable, so the full four-arm memory series did not pass the common-baseline check. The state reduction and allocated-peak reduction have different denominators; neither establishes a percentage reduction in total VRAM for Katana on a 3060.

The released zero-momentum mode trains a small LM with much less optimizer state; large-model fit, sustained speed, and model quality are separate questions. The next measurement depends on the goal:

Goal Smallest useful route
Check that Avalanche trains Pin the public source, run a short real LM with a fixed held-out set, and share the curve. This is what I tested.
Compare memory fairly Count persistent state; separately measure full-step allocator peak from equal live baselines.
Understand the 12 GB training result Profile the actual Katana architecture, precision, sequence length, loss head, checkpointing, and optimizer together.
Compare training efficiency Use warmed-up end-to-end tokens/s and loss versus tokens, under stated batch/precision settings.
Check long-run continuity If exact restart matters, compare uninterrupted training with a fresh-process save/reload path.
Evaluate Katana itself When its weights/tokenizer are available, verify the base-model interface and a few controlled outputs before broad benchmarks.

I have the CSV loss records and plots from these runs if a closer look at the trajectories would be useful. Here are the conditions and references.

The public source and the small training experiment

Exactly which code and environment?

I tested the public OpenBladeAI/Avalanche repository at commit 3cdf9fa76620ef144a09d80210d4f294e896d36b, importing the pinned optimizer.py. The file’s SHA-256 was:

1e672c26581e1ac8158a8be9f3779ceabe1965dfe4de0ee6db644ec7b08dbd2c

The Colab test fetched that file, checked its hash, and used a normal PyTorch loop without a CUDA extension or GitHub API polling. The Katana report describes a project-specific GPUFactorizedAdamV2 and custom trainer. This test covers the pinned public implementation, not the complete trainer.

The actual runtime reported Python 3.13.16, PyTorch 2.14.1+cu130, CUDA build 13.0, and an NVIDIA L4. I did not run this test on an RTX 3060, and I did not load the 1.12B Katana architecture. Recording the PyTorch build matters for reproducing CUDA allocation behavior; recording the Git revision matters for reproducing optimizer behavior.

Model, data and held-out contract

  • Causal decoder-only next-byte model: 3 transformer layers, width 192, 4 attention heads, 1,457,856 trainable parameters, with standard embeddings, LayerNorm, GELU feed-forward layers, and PyTorch scaled-dot-product attention.
  • Byte vocabulary 256, context 128, microbatch 8; FP32, dropout 0, weight decay 0, norm clipping 1.0, no AMP scaler or separate precision master copy.
  • TinyShakespeare raw input, treated as UTF-8 bytes, 1,115,394 bytes; SHA-256 86c4e6aa9db7c042ec79f339dcb96d42b0075e16b8fc2e86bf0ca57e2dc565ed.
  • Contiguous split: first 90% of bytes for training, final 10% held out. Each evaluation used the same four preselected batches of eight 128-byte targets: 4,096 predicted bytes in total. This is a fixed sampled estimate, not a full-corpus held-out sweep or a document-level independence guarantee.

This is a cheap learning check, not the report’s code/math corpus. Its byte-level loss is not comparable numerically to Katana’s tokenizer-level result.

The four arms

Avalanche: mode="none", beta1=0.0, beta2=0.999
AdamW:     beta1=0.9, beta2=0.999, foreach=False, fused=False
Both:      FP32, weight_decay=0, grad clipping=1.0
LRs:       3e-4 and 1e-3 for each optimizer
Steps:     144 per arm; held-out eval at 0,24,48,72,96,120,144
Seed:      20261119

I copied the same initialized weights into each arm. A dedicated RNG precomputed training-window positions; every arm saw that same ordered window list. Held-out positions were fixed too. All four runs returned finite losses. This controls the most obvious data-order and initialization differences, but it does not isolate the effect of factorization alone, because first-moment behavior differs; nor does it provide independently tuned optimal schedules for either algorithm.

For a quick reproduction: pin the commit, use a small public LM dataset, fix initial weights and sample order, compare none with an explicit baseline at two LRs, and log held-out loss plus state tensor bytes. Record dtype, shape, split, betas/LR, clipping, seed and evaluation cadence. A real backward pass is more informative than a synthetic constant-gradient step without becoming a major study.

Reading the short training curves without overclaiming

What changed during the run?

All four arms started at 5.6626557 nats per byte on the same held-out windows. Each produced 144 per-step training losses and seven held-out checkpoints, or 604 CSV records overall. All values remained finite; the drop to final held-out losses between 2.5230 and 2.5912 is the main positive observation.

The comparison at matched tested LR is worth showing explicitly:

Learning rate Avalanche step 144 AdamW step 144 Observation
3e-4 2.5706 2.5912 Avalanche lower by about 0.0206
1e-3 2.5607 2.5230 AdamW lower by about 0.0377

At 1e-3, step-120 losses were slightly lower than the endpoints: Avalanche 2.5542, AdamW 2.5160. At 3e-4, the lowest recorded checkpoint was step 144. These are repeated measurements on fixed windows, not independent test sets or a divergence diagnosis.

What does this comparison answer?

It answers whether this public zero-momentum recipe can train a small transformer with ordinary autograd, and how much persistent state it uses. It does not isolate the effect of factorization on convergence: none removes first-moment momentum but the AdamW control retains it; there are only two LRs, one seed, 144 steps, and a toy literary-byte task rather than Katana’s architecture and corpus.

If convergence becomes the goal, a practical recipe comparison could allow equal optimizer-specific tuning budgets and compare loss at equal tokens or time. An update-rule ablation would instead match momentum and other choices to isolate factored statistics. Neither is required to recognize the functional result.

An extra seed, modestly longer curve, or small LR scan would test the early crossover if anyone wants to pursue it. That is optional; the release thread’s request for an independent loss curve is already addressed.

Why the loss unit matters

These numbers are the mean natural-log negative log-likelihood over byte targets. Katana uses a custom approximately 65k-token vocabulary, a different text distribution, and a far larger context. Exponentiating my per-byte loss would yield per-byte perplexity, not a metric directly comparable to Katana’s per-token perplexity. The Hugging Face perplexity guide emphasizes that tokenization and evaluation-window strategy are part of the metric.

For eventual cross-tokenizer evaluation, use the same raw evaluation text, specify the scoring window/stride and boundaries, and consider a carefully defined byte- or character-normalized NLL alongside task-specific correctness. For this experiment, the contiguous 90/10 split and fixed 4,096-byte held-out sample are enough for a local early-training sanity check, not a claim about code correctness or broad language modeling. That distinction protects the useful result from an accidental comparison it was never designed to make.

Persistent optimizer state versus whole-step GPU memory

The memory improvement is real for the measured component, but the word memory needs a label. Optimizer state, model weights, gradients, backward activations, temporary update buffers, CUDA library workspaces, and total device consumption are not interchangeable. An optimizer can reduce its resident state dramatically while leaving many shared costs of the training step intact. This is particularly relevant to Katana’s 12 GB constraint: saving the optimizer moments makes space for the other parts of the step, but it does not remove those parts.

The state census

After training began, I enumerated the tensors in optimizer.state, counted numel() * element_size(), and excluded references to the model’s existing Parameter objects. This last exclusion matters because the pinned implementation can keep parameter references in state entries. A reference to an existing weight is not a second independently allocated optimizer tensor.

# Core counting rule, abbreviated from the actual experiment.
parameter_ids = {id(p) for p in model.parameters()}
seen = set()
total = 0
for state in optimizer.state.values():
    for value in state.values():
        if isinstance(value, torch.Tensor):
            if id(value) not in parameter_ids and id(value) not in seen:
                total += value.numel() * value.element_size()
                seen.add(id(value))

The results were 73,216 B (0.0698 MiB) for Avalanche none and 11,663,012 B (11.12 MiB) for AdamW. The 11,589,796 B difference is 159.3x in state ratio or 99.37% less auxiliary tensor payload. This excludes Python/allocator overhead, serialized checkpoints and CUDA libraries. A general counter should also handle distinct tensor views sharing storage.

For large 2D matrices, factorized second moments change the shape of a statistic from approximately M*N stored elements to row/column summaries scaling as M+N. That is the conceptual attraction shared by Adafactor and Avalanche’s public none mode. Actual counts also depend on vectors, embedding shapes, tied/shared parameters, precision, and whether momentum or error-feedback residuals are enabled. I measured none, not Avalanche’s 4-bit, 8-bit, BF16 or other modes. Their extra state must be counted separately rather than inheriting this state ratio.

A different 24-step measurement

The first, 144-step notebook was good evidence for learning and persistent tensor size, but its sequential GPU-memory baselines differed. I therefore did not use its four raw peaks as a fair allocator-peak ranking. I ran a new 24-step test with each arm in a separate Python function scope, tracking live allocated bytes before the arm, the model-loaded baseline, the peak over training and evaluation, and live bytes after cleanup.

Execution order in 24-step run Before arm, B With model, B Peak allocated, B After cleanup, B
Avalanche 3e-4, first/cold 8,925,184 14,762,496 78,234,624 25,964,544
Avalanche 1e-3 25,964,544 31,801,856 78,234,624 25,964,544
AdamW 3e-4 25,964,544 31,801,856 89,824,768 25,964,544
AdamW 1e-3 25,964,544 31,801,856 89,824,768 25,964,544

The first arm ended above the memory floor from which it began. The later three arms had identical before-arm floors, identical model baselines, and no further post-cleanup increase. The experiment-wide flag comparison_baselines_clean was therefore false, not a four-arm pass. Within that stable subset, however, the two 1e-3 arms have a comparable baseline and workload:

Avalanche: 78,234,624 B = 74.61 MiB peak allocated
AdamW:     89,824,768 B = 85.66 MiB peak allocated
Delta:     11,590,144 B = 11.053 MiB, about 12.9% of AdamW peak

The peak delta exceeds the state delta by just 348 B, consistent with state accounting for much of this toy-run difference, but not a per-allocation trace. PyTorch allocated peak is distinct from reserved memory or total VRAM; see its CUDA memory guide.

For a reader comparing memory percentages, I would keep three separate sentences rather than one spectacular number: 99.37% less persistent optimizer tensor state in this tiny none experiment; about 12.9% less PyTorch peak allocated memory in the matched, baseline-stable 1e-3 subset; and no independent measurement of Katana’s full RTX 3060 process memory here. All three can be true simultaneously. The separate metric labels strengthen rather than weaken the concrete observation.

Why the first CUDA memory baseline moved

This is a measurement caveat, not a problem I observed in Avalanche or Katana. The first arm of the separate memory run began with 8,925,184 B of live PyTorch allocations and, even after the model and temporary objects were cleaned up, left 25,964,544 B live. That once-only increase was 17,039,360 B, exactly 16.25 MiB. The next three arms started and ended at the same higher floor, with the same model-loaded baseline. It made sense to compare the later stable arms and explicitly mark the four-arm check as unclean.

There is a plausible version-specific explanation. PyTorch 2.14’s CUDA documentation describes a default cuBLAS workspace arrangement equivalent to:

CUBLAS_WORKSPACE_CONFIG=:4096:2:16:8
One workspace: (4096 KiB * 2) + (16 KiB * 8) = 8.125 MiB
Two workspaces: 16.25 MiB = 17,039,360 bytes

The retained amount matches two such workspaces numerically. A PyTorch Forums CUDA-memory discussion describes a related lazy-workspace behavior. That is a strong candidate for why the first ordinary forward/backward workload changed the baseline. But I did not trace these allocations to particular cuBLAS handles in this run, so I would call it a likely cold-start mechanism, not a proven root cause. The first notebook’s shared function scope could also retain references to earlier computation graphs; the follow-up isolated each arm in its own function. This reduces that possible confounder without proving the source of the earlier baseline drift.

The easy control, if precise allocator peaks later become important, is to run a small unmeasured CUDA warm-up using the same operations, then measure each arm from a matching pre-arm floor and model-loaded baseline. Record memory_allocated, the full forward/backward/optimizer/evaluation peak, and post-cleanup live bytes. Resetting a peak counter does not free surviving tensors; empty_cache() does not remove tensors still referenced by Python. If a baseline keeps moving, isolated subprocesses or PyTorch’s memory tracing are better discriminators than accepting the table as clean.

I would not demand this extra run for the present thread. The three-arm stable subset already yields a narrow observation, and the independent held-out training curve is the more direct answer to the request for Avalanche feedback. A different PyTorch/CUDA build could initialize library workspaces differently; that’s why I would keep the installed version attached to the 16.25 MiB observation.

Related methods: a map of comparisons, not a winner list

The closest research vocabulary I found for understanding Avalanche is factored second-moment estimation. In a large matrix, keeping statistics along rows and columns instead of one full statistic per element can save substantial auxiliary state. Adafactor (Shazeer and Stern, 2018) is a useful conceptual neighbor: it discusses factored second moments, update control, and avoiding conventional first-moment momentum. That connection helps with understanding the geometry of the state, but it does not imply that Adafactor and the pinned public Avalanche use identical equations, clipping, initialization or schedules.

There are also adjacent routes with different design choices: SM3 uses structured parameter covers for memory-efficient adaptive statistics; CAME augments factored ideas with confidence-related mechanisms. GaLore reduces optimizer-related memory through low-rank gradient projection during full-parameter training, and APOLLO explores compressed adaptive scaling. These are different mechanisms and evaluation setups, not evidence that one particular implementation duplicates or outperforms another.

Reference or mode What to compare What cannot be transferred automatically
AdamW Dense per-parameter moments; familiar learning baseline Its state denominator and LR behavior differ from none
Adafactor Factored second moments, update clipping/scheduling Exact formula, handling of vectors and learning dynamics
Public Avalanche none Row/column-style statistics without dense first moment Not representative of all Avalanche modes or Katana’s private trainer
Avalanche low-bit momentum modes Packed velocity and any scales/residuals The tiny none state result and its loss curve
SM3, CAME Alternative ways to retain reduced adaptive statistics Their estimators and training conditions
GaLore, APOLLO Projection or compressed scaling during training Memory-quality-throughput tradeoffs under a different update path

The comparison axes are state bytes, full-step peak, feasible batch, tokens/s, loss versus tokens, and restart behavior. A state-light optimizer may fit larger batches but spend more time per update; activations may dominate elsewhere.

For low-bit momentum, a 4-bit packed vector describes the payload, while scale tensors and optional residual/error-feedback buffers also take space. This is an accounting distinction, not a design objection. The project’s mode documentation covers those choices; my test only covers none.

An optional Adafactor baseline could distinguish general factorization benefits from implementation choices, with its own documented schedule and precision. But AdamW plus the observed small LM curve is enough for this first feedback. These links are a map for future readers, not homework for the author.

Condition-based checks for a full single-GPU training system

The main Katana constraint is unusually concrete: full-parameter training from scratch on one RTX 3060 with 12 GB, not merely fitting a quantized inference model. I would keep that objective intact. A memory-efficient optimizer does not need to solve every memory or speed issue itself; it needs to free enough budget for the rest of the designed training step. Your Katana report already describes 512-token chunked cross-entropy, variable-density gradient checkpointing, and pinned-memory background prefetching. Those are real parts of the reported design, not missing optimizations I am recommending from the outside.

If the goal is state compression, the simple census above is sufficient to start. The state footprint depends on the shapes of 2D matrices and smaller vectors, whether embeddings are tied, which modes retain momentum, and the precision of the statistics. A short state-by-layer table at one real update answers that question much more directly than a large end-to-end benchmark.

If the goal is whether a full step fits inside 12 GB, the relevant budget includes weights, gradients, any master copies, stored optimizer state, live activations, the head/loss computation, temporary optimizer update tensors, and library workspaces. In particular, materializing logits for every position over a roughly 65k vocabulary can be costly. The report’s 512-token loss chunking addresses precisely that risk; activation checkpointing exchanges compute for lower activation storage. One representative profile can label peaks at the head/loss, backward, and optimizer phases without treating the techniques already present as if they had been overlooked. The actual full-step maximum and whether offload is used answer a different question from a state-only percentage.

If the goal is sustained pretraining throughput, compare warmed-up end-to-end tokens per second at the intended sequence length, effective batch, precision, data path and accumulation schedule; pair it with loss versus training tokens. The tiny L4 experiment incidentally recorded about 13.5–13.8 ms per step for Avalanche and 11.0–11.1 ms for AdamW. Those order-dependent, tiny-FP32-model timings are not a meaningful 3060 throughput comparison or a refutation of any advantage obtained by fitting larger batches. Fit, throughput, and learning progress are separate axes. Lower optimizer-state overhead can improve throughput indirectly by changing a feasible batch configuration even when an isolated update is not faster.

If the goal is reliable continuation of a long run, a small fresh-process resume test is a useful optional control. Exact continuation may depend on model weights, optimizer statistics and step counters, scheduler, AMP scaler if present, RNG, custom dataloader position, and sample order. PyTorch’s optimizer state-dict documentation describes the generic serialization boundary; a custom trainer may preserve extra state separately. A cheap test is to compare several uninterrupted steps with the same steps split by save, fresh-model/fresh-optimizer reload, and continued identical data. It should be evaluated against the intended tolerance for numerical determinism. This is not a claim that Katana currently mishandles checkpoints; I have not seen its internal save/restart path.

These are independent choices, not a to-do list. The Hugging Face memory anatomy guide provides useful budget terminology; Katana’s actual design determines which tradeoffs matter.

Evaluating Katana itself when a specific checkpoint is available

I would keep model capability separate from this optimizer feedback. The forum thread and research report describe the architecture, token stream, custom corpus and training observations, along with a staged release plan. I have not independently loaded Katana weights, so I cannot confirm those generations, the reported 3060 performance, or the model’s code/math competence from my small Avalanche run. A planned public date is also different from a verified downloadable revision.

When a public base checkpoint and tokenizer can actually be obtained, a useful low-cost path for outside readers would be:

  1. Pin the model revision, architecture/config, custom tokenizer and any required loader code. Confirm a tiny load, forward pass, tensor shapes and basic next-token generation before discussing capability.
  2. Distinguish base from instruct. A base model may be better assessed by prompt-prefix continuation or candidate log-likelihood than by treating it as a chat assistant with an assumed chat template.
  3. Check tokenizer IDs, special tokens and sequence conventions. A custom roughly 65k tokenizer is an essential part of the interface; replacing it silently changes the experiment.
  4. Start with a few fixed, short tasks and decoding settings. For code, separate plausible surface syntax, parser/compiler acceptance, and executable functional correctness. For math, distinguish a well-formed derivation from a checked answer. A tiny unit test or parser can supply an unambiguous signal where a visual impression cannot.
  5. If checkpoints at different training stages become available, use a consistent small probe set to examine changes in structure and functional behavior over time. Pythia’s training-dynamics work is one useful precedent for this style of analysis, not a demand to publish every Katana checkpoint.

The report’s idea of early structural acquisition is particularly suited to this separation: forming recognizable signatures, tags, or code layouts may precede robust semantics. That is an interesting trajectory even if the first checkpoint does not yet solve complete programming tasks. A handful of reproducible prefixes, parser results and tiny functional tests could give a future reader a much clearer picture than a single impressive or disappointing generation. It remains a small probe, not a broad benchmark or a verdict on the research.

For comparisons with other 1B-class models, keep the raw evaluation text, scoring convention, context window, and any task-specific tests aligned. Token-level perplexity from different tokenizers and corpora is not a straightforward head-to-head ranking. If a larger evaluation later makes sense, EleutherAI’s lm-evaluation-harness, EvalPlus for executable code checks, and the HF model-card guidance are reusable starting points, subject to the actual model interface and supported tasks.

One further boundary is context length. The thread mentions a native 4,096-token window and RoPE-related scaling toward a much larger number. A large scaling setting alone is not an observed long-context capability. If long-context behavior is important after release, a few increasing-length retrieval or continuation checks would help locate the validated region, without prejudging the design. The near-term path is simply to use the native, documented interface first.

So my concrete feedback for now is modest but positive: the released, pinned Avalanche none implementation trained a small causal LM with decreasing held-out loss, using dramatically less persistent optimizer tensor state than AdamW in that experiment. The extra GPU-peak observation is narrower and carefully baseline-conditioned. The links above are ways to extend the question depending on the objective, while preserving Katana’s original single-GPU, from-scratch goal. I hope the observed loss trajectory and explicit measurement conditions are useful to others trying Avalanche or, later, evaluating Katana checkpoints.