Kye Gomez (swarms) (@KyeGomezB): Update:

Update: I never expected this post to blow up at all. This is extremely interesting. And, I did not make this post for attention; I made it simply to share my thoughts and hypothesis on how Claude Mythos could theoretically work. This is still a research project in its very early stages, and the model architecture will continue to improve over time. I, alongside some researchers, am training a 3B and possibly even a 5B version on the FineWeb-Edu dataset on @huggingface. We will document all of our findings and release the trained model as open source. After the base 3B model is pretrained, we will further fine-tune it with GRPO and a series of high-quality RL datasets. A small training run conducted by a community member in our Discord shows that OpenMythos reaches its best validation in 2.67× fewer steps than nanoGPT, which is extremely interesting. Again, this is still theoretical and simply an implementation of my personal hypothesis and other researchers’ ideas in the reasoning space of what we believe Mythos could function like. All assistance and contributions are extremely welcome, as this is an open effort to experiment with and observe the performance of looped transformer models. The training script is in the GitHub repository, and I will share the links below. I will also release a detailed blog post on OpenMythos in the coming days. I am sick, by the way, and have been for the past week, so please pardon the delay in responding to questions or feedback. Onwards we go!