Wildroot

Queries may use an external AI service. Details

Wildroot Browser

TodayAISimilar posts

Similar posts

@lea-linux.org

"Most modern transformer-based LLMs use the same broad structure: tokenization, embeddings, positional encoding, stacked transformer layers (each with multi-head attention and a feed-forward network), residual streams, layer normalization, and next-token prediction."

Most modern transformer-based LLMs use the same broad structure: tokenization, embeddings, positional encoding, stacked transformer layers (each with multi-head attention and a feed-forward network), residual streams, layer normalization, and next-token prediction. What changes between models is: The trained weights themselves, learned from different training data at different scales. The configuration: number of layers, vocabulary size, head count, parameter count, MoE or dense. The post-training: instruction tuning, learning from human feedback, safety controls applied on top of the base model. Tiny explainer: weights Weights are the learned numbers inside the model. Training changes those numbers until the model predicts text well. The 2023-2025 “modern transformer” stack converged on a common set of choices across many serious frontier and open-weight models, even though different teams arrived at them independently. Pre-norm placement. RMSNorm. RoPE. SwiGLU. Grouped-Query Atten
Info

2 posts close in meaning, closest first

  1. @dzaladan.eurosky.social

    I'm a naturally curious person, so I like to explore any kind of topic. This thread here gives a precise explanation about how LLMs work. In a nutshell, confirms what has been said already: it's controlled randomization of preexisting data. So... math, not chemistry.

    Same collectionAI