Insufficient context length
A context length of 32K is reasonable when we are talking about models no larger than 8B, but when we get to a size of 24B, the context length of 32K is too short and affects usability. Your work is very excellent, will you consider extending the context length to version 2.5 next? Like 128K or more？
According to the config, max_position_embeddings is 128000, indicating support for a 128K context window. However, unexpected outputs or performance degradation might occur at extreme context lengths.
Yes, we plan to extend it for the 2.5 release. Please see this one as an early checkpoint. :)
Please train for coding, add benchmarks also for it the next version you are cooking. Thankyou.
Please train for coding, add benchmarks also for it the next version you are cooking. Thankyou.
Coding is pretty vast. Somehow the tech has to come together for domain-specific computing.
My idea: MoE with pluggable experts: need linux shell + terminal + networking + kernel expertise? Link-in the UnixGuru.moe module.
Something something, adapters + meta-router blah blah i have no idea how...