lithinnon/Merousis-26B-A4B

Merousis 26B A4B

I put this merge up for my own setup because none of the existing G4 finetunes could get most of my preferences across.

Some available finetunes do work pretty well but come with drawbacks over a long campaign, like losing track of the story, items, or locations across the room or map. Other models track world state and items reliably, but their prose is flat, boring, and uninteresting.

So I picked models that score high according to my preferences but have trade-offs between them. With that said, merging them came to mind. However, merging DARE or SLERP across the models did not work well here. Those techniques tend to mess up Gemma 4's MoE routers and could trigger prompt leaks. Instead, I inspected the layers and weights to figure out how each part influences the strengths of the finetune, and compositely merged the effective layers from each model.

My issues with some finetunes out there are mostly personal preferences. Take Boulesis v2.1: in long sessions it struggles with consistency. At times it loses track of concrete state over time, like item locations or counter tracking on simple math (even though it got complex ones right), and got stuck in reasoning loops easier and longer than other finetunes. I think the culprit is Pantheon, but it would perform better when reasoning was disabled compared to others. It also leaked internal prompts from tags (XML tags), though the HeadTune variant apparently fixed this issue. Maybe the StyleTune V2 LM Head doesn't like working with a finetune model. However, it has superior grasp over complex physical intuition and environmental nuances, avoids a lot of story clichés, and writes great psychological prose. G4 is probably on a cliff of overfitting, so putting Pantheon in it caused downgrades in several aspects.

As for MeroMero, it is the closest to the stock model in many aspects. It can still get stuck in a reasoning loop like other finetunes, but it pretty much behaves like the stock model, which isn't as severe. Because stock was merged back in, its ability to track details is superior to Boulesis v2.1, though it couldn't follow complex real-world reasoning like physics, social dynamics, and ambient details as well. However, MeroMero mostly dominated in terms of spatial tracking, simple math, and instruction following.

With that said, I started working on it. Most Boulesis layers are the same as stock or Heretic: only 96 tensors to be exact, which are mostly Q/K and some V/O from the LoRA (all documented in their model card). I used Heretic as the base model. Starting with Boulesis and its Pantheon layers: the Pantheon composite merge is already well-made, but I only wanted its reasoning intelligence and deduction ability, so I pulled in its middle layers (9-24), where most of the Pantheon reasoning lives. With that done, the Boulesis part was complete.

On to MeroMero: fortunately, the raw SFT model is available. By going through layer and weight differences, it came down to 265 tensors. All Value projections and Output projections were pulled in, which mostly represent their roleplay prose and reasoning data. Lastly, all MoE experts and MLPs from all layers were pulled in with a catch: they were merged with Heretic, though not 50/50. If pulled in entirely, it might suffer from reasoning loops like Boulesis. If pulled too low, the low active parameter count might ignore it.

As for other weights and layers, leaving them to Heretic is fine since stock/Heretic is more than enough. The stock model was already good; I just pulled in the parts of the finetunes that satisfy my preferences. Lastly, the LM Head is a merge of HeadTune and the stock one. Why? Well, I like it. Nothing significant.

For GGUF quantization, I provided an imatrix (Merousis-26B-A4B.imatrix.gguf) which is a blend of mradermacher's imatrix and the one used in MeroMero GGUF. The first reduces quantization loss on general knowledge including reasoning and deductive logic, while the latter aids the roleplay and creative writing aspect.