Alright the prelim went well. We're at about 3.4 billion tokens learned.

Model very stable as the prelim tests showed. The structure is not collapsing and the model is in fact learning useful pathways of information. So far it hasn't established legitimate ground-form informational segmentation pathology, the model requires many more tokens for this.

Model learns at around 100k tokens per second scaled to the RTX 6000 PRO card available on colab, so the bf16 training is scaling nicely to the benefits from the blackwell hardware.

Each is packed with an fp8 variation for inference as well, which is substantially faster for inference, but they aren't very smart yet. They ARE available. The results show the correctly aligned fp8 variants are roughly the same accuracy on inference, but training they essentially collapse the attention in less than a thousand steps.

There is a full pretrain lineup ready. We're looking at maybe 80 billion tokens or so, and with that the chat AMOE cluster will be ran after each major finetune. We'll be drawing the conversation out of the model each major train, and with each train we'll determine if the model even needs to be finetuned with chat to allow the model to behave.

AMOE arm training planning begins already, as each layer can have direct integrated AMOE arms into the core for couple/decouple purposes. The structure itself will be learning chat through finetuning arms with chat directly onto the trunk. Based on the responses, we will continue pretraining the trunk until the construction of the AMOE chat conforms to the behavior.

So far it's showing the model likely does not require the chat to be baked into the core in finetune state. Prelim shows the chat is likely corrupting the core of the data compendium by learning it atop the core system. The AMOE could very well handle the first-class behavior, but there's no guarantees for this just yet.

wikitext 103 and fineweb main + extended are the preliminary.

These two are targeting;

  [  done  ] warmup_wikitext    wikitext-103   0.300/0.30B
  [  done  ] fineweb_main       fineweb-edu    3.000/3.00B
  [planned ] fineweb_extended   fineweb-edu    0.000/12.00B

So roughly 30 hours until fineweb's larger structure is trained. After this our first chat arms will be trained, which ought to let us have a conversation with her. See how well she took to the information.

I can say with a solid underlined statement: The model DID WORK.

The preliminary chat amoe arm trained atop responded. The conversation was shallow but the conversation wasn't simple noise or chaos, the responses were confidently wrong, and effectively related to the questions and answers.

The structure in the huggingface space DOES NOT HAVE this chat arm yet, but I plan to have this fully functional and operational by tomorrow evening. Each arm needs to be tailored to the model because the model is brand new, the deviations from the core to the pretrain are still high, so the structure needs to be aligned with the AMOE arm correctly.

This is a sample from the arm based on 21.9k steps, roughly 11 billion tokens trained I think.

The core was not trained with chat, so she requires an anchored mixture of experts leg or a lora trained atop. She's quite compliant, so if you wish to train a lora she'll listen.

For now she's a next byte prediction auto-complete model, a pretrained trunk capable of AMOE expansion directly.

Once the pretrain concludes I'll prepare 2 chat arms.

After an anealment of around 2b tokens of various behavioral attunements, concepts, conversational learning, and so on; the stage 1 variant will have 2 core chat arms for direct huggingface space conversation.

Pre anealment chat arm, and post anealment chat arm. This will allow a structured fusion between the two behaviors and the necessary outcomes, creating a unique and interpretable duality between pure pretraining with a chat module, and post pretraining scattered conceptual finetuning with a chat arm. Post will essentially understand more about conversation, while the pre will have less knowledge and more a crash course in utilization.

I'll name them accordingly when the time comes, but they will both be Beatrix variants.

After this process we will begin forming candidate arm extensions for behavior. Concepts like wikipedia recall, toolchain utilization, mathematics, coding basics, and more. Very small pieces of the information for recall with the KV cache.

How effective they will be is another story. The tests and utilizations of the outcomes will show which arms are to be integrated into the larger form on pretrain. In other words, which arms will intentionally have their finetuning directly tied to the core and become post-train guarantees that don't decouple.

Stay tuned my friends. She's just getting started.

I have a list of upcoming prototype arms.

• Deterministic chat - Can we speak to the pretraining directly with an AMOE arm?
• Retokenization arm - Can we retokenize and cluster the bytes into BPE?
• Arbitration arm - Can we teach a small arbitration arm to communicate with another system?
• Mathematics arm - Multiple mathematics formats within an arm cluster.

This model will run on CPU. I ran it on windows 10 with my 12 core, roughly 10 bytes/s give or take, a fair prediction ratio for cpu.