Idea workflow

Temporarily reduce the learning-rate multiplier only for Q/K projections in the upper transformer layers during the first 100 warmup steps, then restore the matched baseline multiplier; this may improve 3k-step decoder pretraining sample efficiency by delaying premature upper-layer attention specialization while preserving architecture, data, objective, and the final optimizer state.

The local Titan snapshot can support the proposed fixed 100-step upper-layer Q/K LR damping mostly through existing config machinery: lr_scale_regex_initial, lr_scale_regex_initial_steps, and LRScaledSchedulersContainer. The smallest safe path is to generate a downstream candidate from scripts/runs/tiny.yaml, keep the matched control unchanged, damp only selected upper-layer attention.wq/wk weights by 0.25 for steps 1-100, then restore the normal multiplier. The only unresolved local scientific choice is the odd-depth upper-layer boundary for the 9-layer tiny override.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

14048b8a98717d7037a6733b093ababc8560dc8f

Branch

auto-research/qk-warmup-manual-recovery

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/implementation/action_11f7627f44ae4395

Configuration

/persistent/.local/state/george-auto-research/worktrees/implementation/action_11f7627f44ae4395/scripts/runs/auto_research_qk_warmup_control.yaml

Config SHA-256 (recorded, unverified)

3fd647964bd228ba51129e2b77fb26fa240c1da678d9c855a4053fd334a947ef

Actual config SHA-256

3fd647964bd228ba51129e2b77fb26fa240c1da678d9c855a4053fd334a947ef

Resolved config SHA-256 (recorded, unverified)

3fd647964bd228ba51129e2b77fb26fa240c1da678d9c855a4053fd334a947ef

No training launch command was recorded. The provenance above identifies the exact revision and configuration evidence that is available.

Fix pipeline-parallel MoE router EMA updates so only retained registered MoE layers are updated after splitting, and cache references match the live post-parallelization parameters. The repair must preserve PP1 behavior and enable the matched PP3/DP2 3k control and candidate runs; reject it if the exact PP3 control still asserts or if it changes non-PP router EMA behavior.

Feasible as a local code repair: pipeline splitting currently prunes registered layers but leaves TransformerMoE._moe_layers stale, while EMA reference caching walks only live registered routers after parallelization. The smallest fix is to refresh the MoE layer cache on each PP stage chunk after pruning, so update_ema and running stats visit only retained registered layers and cached references match post-parallelization parameters. No scientific choice is missing for the 1B implementation. The exact 17B PP3/DP2 8-GPU request is not feasible from this snapshot because the baseline is MTP-wrapped, MTP has no pipelining registry, and PP3*DP2 cannot exactly fit eight ranks.

Add one learnable RMSNorm to the complete output of every attention branch after output projection and gating, and to every dense, routed-expert, and shared-expert MLP branch after its down projection, immediately before each branch is added to the residual stream. Retain the existing input Pre-RMSNorms, QK norms, embedding path, final norm, attention pattern, and residual topology. This isolated output-normalization component from Peri-LN should lower equal-token held-out decoder pretraining loss and reduce depth-wise activation growth and gradient spikes without changing model width. Falsify the sample-efficiency claim if the candidate does not beat the matched control beyond calibrated local run noise at step 3000, or if any apparent loss gain depends on a changed schedule, data order, token count, or added embedding normalization.

Local 1B implementation is feasible with a small architecture flag and focused tests. The supplied historical commit 9f13144d implements a different residual-norm switch: it reuses existing input norms on sublayer outputs and can remove PreNorm, so it should not be ported directly. The faithful diff adds separate learnable output RMSNorms after attention output projection/gating, dense FFN down projection, and the complete routed/shared MoE branch outputs immediately before residual addition, while preserving existing PreNorms, QK norms, embedding path, final norm, attention pattern, and residual topology. No unresolved local scientific choice blocks implementation.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_d22956657b444feb

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

479f0b199890728563f86a74f03b3615b5498683

Branch

auto-research/idea_50ea033e59f842f7/local-dp4-candidate_ebffb3c1117440605e50

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/local-dp4-candidate_ebffb3c1117440605e50

Configuration

/persistent/.local/state/george-auto-research/run_configs/local-dp4-candidate_ebffb3c1117440605e50.local.yaml

Config SHA-256 (recorded, unverified)

d3cfa4db4d54835154547a73ec9a2618224b7a19ebe517d44568b9119dd282d9

Actual config SHA-256

d3cfa4db4d54835154547a73ec9a2618224b7a19ebe517d44568b9119dd282d9

Resolved config SHA-256 (recorded, unverified)

ee2cbab8da41ebc334920b46b46e4e84584eeffe905d28424dff229002a602aa

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/local-dp4-candidate_ebffb3c1117440605e50.local.yaml

In every dense, routed-expert, and shared-expert SwiGLU FFN, keep the SiLU gate branch unchanged and replace only the linear multiplicative branch z with the source one-sided token-adaptive mixture sum_k sigmoid(u_k^T x)*sigma_k(z), sharing the existing linear projections across activations. Use the exact paper-selected Type-II activation dictionary and sigmoid gating, initialize the gate vectors with N(0,0.02^2), and do not use bi-sided, quadratic, hard-routed, or learnable-input-independent variants. This added token-adaptive nonlinearity should lower equal-token held-out loss at approximately unchanged memory and modest runtime cost. Reject the claim if final loss does not beat the matched SwiGLU control beyond calibrated local noise, if gates collapse to a token-independent mixture without a loss benefit, or if measured throughput makes the gain unfavorable.

Local implementation is structurally possible in Titan but not exactly specifiable from the supplied evidence because the required Type-II activation dictionary is missing. The relevant code paths are dense Transformer FeedForward, TransformerMoE FeedForward for dense/shared experts, GroupedRoutedExperts for routed experts, and the MoE/MTP parallelizers. The supplied local-history patch 5215b002 only changes shared-expert gate dimensionality and does not implement this MoA mechanism.

At every decoder Pre-RMSNorm, compute the RMS-normalized direction once, then replace the one shared affine scale with separately learned gamma_Q, gamma_K, and gamma_V vectors before the query, key, and value projections, plus separate gamma_gate and gamma_up vectors before the two SwiGLU input projections in dense, routed-expert, and shared-expert MLPs. Initialize every branch-specific vector from the control shared gamma so the initial function is identical, and leave the existing QK output norms and all other architecture and optimizer behavior unchanged. The source argues that these O(d) vectors change optimization preconditioning rather than expressivity; the falsifiable prediction is lower equal-token held-out loss. Reject the claim if step-3000 loss does not beat the matched control beyond calibrated local noise or if the gain disappears when all non-scale-vector settings are matched.

The mechanism is mostly implementable in the target tree: current decoder PreNorms already centralize the single affine RMSNorm before attention and MLP branches, and MoE routed/shared experts have identifiable gate/up projection sites. The smallest diff is a new branch-specific norm path plus wiring through dense, MoE, FP8 routed, shared-expert, TP, MTP, and activation-checkpoint call sites. The main unresolved local scientific choice is Titan's existing attention gate projection, which is fed by the same PreNorm but is not covered by the source's Q/K/V-only branch definition.

Replacing RoPE with the full PoPE v3 mechanism - softplus query/key magnitudes, one scalar frequency per head dimension, and a learned per-layer/per-head/per-frequency phase delta zero-initialized and clamped to [-2*pi, 0] - will decouple content and position and lower held-out next-token loss/AUC under the fixed Titan TPP30 protocol.

The ICML 2026 camera-ready primary paper evaluates matched decoder-only language models on OpenWebText from 124M to 774M parameters. Reported RoPE-to-PoPE perplexity is 21.55 to 21.33 at 124M, 18.88 to 18.55 at 253M, and 15.85 to 15.45 at 774M; the 774M downstream average is 51.80 to 52.46. The paper's ablations show both softplus magnitudes and learned phase delta are needed for the full result.

Replace each decoder RMSNorm output covered by the source mechanism with rank-16 elementwise GatedNorm: for y=RMSNorm(x), compute y_gate=sigmoid(W_up(SiLU(W_down(y)))) and return y_gate*y. Preserve the baseline gated-attention mechanism, attention pattern, residual topology, and all non-normalization behavior. Use the source rank 16 and bounded sigmoid gate, and match the control total parameter count by reducing only candidate FFN capacity by the minimum required amount. The explicit learned rescaling should lower equal-token held-out decoder pretraining loss and reduce residual-stream outlier concentration. Reject the claim if final loss does not beat the parameter-matched control beyond calibrated local noise, if training becomes unstable, or if the result requires a non-sigmoid or tensorwise gate.

The local 1B implementation is feasible without production code changes: the target snapshot already contains rank-configurable elementwise sigmoid low-rank GatedRMSNorm and MoE config wiring. The candidate should be generated downstream from scripts/runs/tiny.yaml, not checked in under scripts/runs, with residual norms switched to gated_rmsnorm rank 16 and dense FFN intermediate_size reduced from 2048 to 1845. Keep gated attention, attention windows, QK norms, loss masks, EOS/data settings, optimizer, seed, and fixed equal-token protocol unchanged. The supplied gated-XSA history is unrelated provenance for attention gating and does not implement this normalization mechanism.

Implement the source anGPT architecture as one explicit compound candidate: use its dimension-derived fixed approximate normalization factors, LERP-style residual updates, and bounded directional weight parameterization instead of nGPT-style exact per-step hypersphere normalization. For this first architecture test, retain the Titan control optimizer, weight decay, 100-step warmup, and 900-step cooldown so schedule and regularization do not confound the architecture comparison. The compact-space geometry should lower equal-token held-out loss and control residual-norm growth with small runtime overhead. Reject this architecture claim if final loss does not beat the matched GPT-plus-QK-norm control beyond calibrated local noise, if norms escape the source bounds, or if improvement appears only after also removing warmup or weight decay.

Repository wiring can host an architecture candidate in TransformerMoE, and the matched local config should be generated from scripts/runs/tiny.yaml with fixed 3000 steps, 100 warmup, and 900 cooldown while retaining Muon weight decay. Inspection found no existing anGPT implementation and no complete source equations in the supplied local evidence, so exact implementation is blocked rather than safely inferable. The 17B path has verified configs, including scripts/runs/large.yaml, but the checked-in large baseline is MTPTransformerMoE and should not be used directly for a clean architecture-only claim.

Changing the pretraining objective to add MIR, an auxiliary next-token loss on randomly masked inputs, should improve held-out base-model loss at matched training tokens because input corruption regularizes repeated-data memorization and encourages more robust autoregressive representations without changing inference architecture.

The source proposes masked-input regularization for autoregressive decoder-only pretraining: add an auxiliary next-token prediction loss where inputs are randomly masked, aiming to improve data-constrained repeated-data training without changing architecture or inference. It also proposes SoftQ, a scaling law for finite-data repeated-epoch regimes, but the clean Titan candidate is the MIR objective intervention.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_273bf95a8c1842c7

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

7b79c5c37ee1453d085c599412b000638c52d9b1

Branch

auto-research/idea_1278bd21deec473e/probe-dp4-candidate-fullac_6e16c91a699b51295e18

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/probe-dp4-candidate-fullac_6e16c91a699b51295e18

Configuration

/persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate-fullac_6e16c91a699b51295e18.local.yaml

Config SHA-256 (recorded, unverified)

c751c6e82aeab43693d159e7bade86eba2070a1b68fdd7911b320c3584a1fd0a

Actual config SHA-256

c751c6e82aeab43693d159e7bade86eba2070a1b68fdd7911b320c3584a1fd0a

Resolved config SHA-256 (recorded, unverified)

4a87d478567e19d6a45336ffa1a27595176327a6903b36f5deed05354a46d7f4

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate-fullac_6e16c91a699b51295e18.local.yaml

Changing eligible Transformer matrix updates from decoupled weight decay to Muon Hyperball should improve held-out base-model loss under matched tokens or matched measured FLOPs because fixed Frobenius weight and update norms directly control angular learning rate instead of relying on implicit weight-decay equilibrium.

Hyperball replaces decoupled weight decay's implicit matrix-norm equilibrium with an explicit Frobenius-sphere optimizer constraint. For eligible Transformer matrices, it fixes matrix norms and update norms while leaving embeddings and norm-like parameters on Adam. The source reports that MuonH maintains larger token-equivalent speedups than MuonW at 1.2B scale and improves LR transfer across model shapes.

Changing eligible Muon matrix updates to apply one post-polar bidirectional normalization step should improve held-out decoder pretraining loss under matched tokens because it reduces row/column imbalance amplified by practical polar iterations without adding optimizer state.

Muon+ addresses a claimed post-polar imbalanced-update problem in Muon: practical polar/Newton-Schulz orthogonalization can amplify row and column imbalance in the update matrix. The proposed fix adds one normalization step after polar orthogonalization, without extra optimizer state. The supplied evidence reports lower validation perplexity than Muon across GPT and LLaMA pretraining runs on FineWeb, with direction, polar-function, competing-optimizer, overtraining, and limited multi-seed ablations.

Changing eligible decoder matrix updates from standard Muon to Newton-Muon right-preconditioned orthogonalized updates should improve held-out base-model loss under matched pretraining tokens because input-second-moment inverse preconditioning corrects Muon's neglected right-side curvature before spectral normalization.

Newton-Muon modifies Muon by right-preconditioning each eligible matrix gradient with a ridge-stabilized inverse estimate of the layer input second moment before taking the matrix sign or polar direction. The source reports faster target-validation-loss attainment on reproduced Modded-NanoGPT FineWeb speedrun settings, with smaller or marginal gains on more optimized records.

Changing eligible Muon matrix updates to NorMuon neuron-wise normalized orthogonalized updates should improve held-out decoder pretraining loss under matched tokens because row-wise adaptive second-moment scaling balances per-neuron update norms while preserving Muon's matrix conditioning benefit.

NorMuon combines Muon's matrix update orthogonalization with neuron-level adaptive learning rates. The source argues that Muon fixes update condition numbers but leaves high variance in per-neuron update norms, and that maintaining row-wise second-moment statistics followed by row-wise normalization balances neuron utilization. Reported decoder pretraining experiments compare AdamW, Muon, Dion, and NorMuon across multiple model scales.

Changing causal softmax attention to Muon-trained Parallax attention should improve held-out base-model loss under matched tokens or matched measured FLOPs because the learned rho projector supplies a local-linear KV-covariance correction that improves associative-memory prediction beyond the local-constant estimator of softmax attention.

Parallax modifies attention from a local-constant softmax estimator toward a parameterized local-linear estimator. It adds a learned rho/R projector that probes KV covariance and contributes a correction branch to the standard attention output. The supplied evidence reports 0.6B and 1.7B pretraining gains under Muon, with parameter-matched and compute-matched controls, and shows the advantage weakens under AdamW.

Changing eligible Muon matrix updates to variance-adapt momentum before Newton-Schulz orthogonalization should improve held-out decoder pretraining loss under matched tokens because per-coordinate variance or noise-to-signal normalization conditions the momentum matrix before spectral whitening.

The source proposes Muon-NSR and Muon-VS, optimizer variants that normalize Muon momentum using variance or noise-to-signal information before Newton-Schulz orthogonalization. Reported decoder-only GPT-2 and LLaMA pretraining experiments compare against AdamW, Muon, and AdaMuon and claim lower validation loss and faster convergence under matched training protocols.

Changing eligible decoder matrix optimization to Conda column-normalized Adam should improve held-out base-model loss under matched pretraining tokens because projecting updates into an orthogonal subspace before column-wise second-moment normalization combines Muon-like spectral conditioning with Adam-like coordinate adaptivity.

Conda modifies Adam for matrix parameters by projecting momentum into an orthogonal subspace and maintaining column-wise second moments from projected gradients. The source reports decoder-only LLaMA and GPT-2 pretraining gains over AdamW, Muon, SOAP, and Adafactor, with claimed step and wall-clock convergence speedups.

Changing eligible matrix-parameter Muon+Moonlight updates to include a dynamic per-layer trust ratio (real-update-direction Frobenius-norm denominator, coupled weight decay, r_l,0=1 calibration) should improve held-out decoder pretraining loss under matched tokens because it restores time-varying, layer-adaptive update magnitude control that a static shape-only scaling factor cannot provide, while avoiding the clip-saturation and weight-decay-runaway failure modes of naive LAMB-Muon hybrids.

Proposes OrScale-LM, a LARS/LAMB-style dynamic per-layer trust ratio applied on top of Muon+Moonlight: r_l,t = ||W_l||_F / (||lambda*W_l + s_l*Q_l||_F + eps), where Q_l is the Newton-Schulz orthogonalized Muon direction, s_l is the Moonlight shape factor (0.2*sqrt(max(m,n))), and a one-time per-layer calibration anchors r_l,0=1 exactly at init. Weight decay is coupled into the trust-ratio-scaled step rather than applied separately, which the authors show avoids a runaway-norm failure mode. The paper also analyzes and rejects two naive LAMB-Muon hybrids (raw-momentum denominator and post-orthogonalized-norm denominator) via clip-saturation diagnostics. Empirically tested on GPT-2-style dense decoder-only transformers pretrained on FineWeb-Edu at 125M/399M/545M/1.1B, beating Muon+Moonlight at 3 of 4 scales and AdamW at all 4 scales; also tested (irrelevantly) on CIFAR-10/DavidNet vision.

Changing the pretraining optimizer from AdamW/Muon to Sophia (gradient-EMA divided by clipped, periodically re-estimated diagonal-Hessian EMA) should improve held-out decoder base-model loss at matched training tokens or matched FLOPs because the lightweight Hessian preconditioner adapts step size to per-coordinate curvature while clipping bounds worst-case updates against Hessian non-convexity/rapid change, reducing steps needed to reach a target loss versus first-order Adam.

Sophia is a second-order optimizer for LM pretraining that maintains an EMA of gradients (momentum) divided by an EMA of a light-weight diagonal Hessian estimate (Hutchinson or Gauss-Newton-Bartlett), re-estimated only every ~10 steps, followed by element-wise clipping to bound worst-case update size. On GPT-2 (125M-770M, OpenWebText) and GPT NeoX (1.5B/6.6B, Pile), Sophia reaches matched validation perplexity with about half the optimizer steps, total compute, and wall-clock time versus tuned AdamW, with peak LR and other hyperparameters separately grid-searched per method and per budget to satisfy a stated fair-comparison criterion (avoiding conflating same-hyperparameter transfer across budgets with true speedup).

Replacing AdamW (or the current Muon-eligible matrix optimizer) with SOAP -- Adam-style first/second-moment updates computed in the periodically-refreshed eigenbasis of Shampoo's Kronecker-factored (L=GG^T, R=G^T G) preconditioner -- should improve held-out decoder base-model loss at matched training tokens or matched optimizer steps because continuously updating second-moment statistics in a slowly-rotating, curvature-aligned coordinate system captures more useful preconditioning information than isotropic per-coordinate Adam while avoiding Shampoo's need for frequent expensive eigendecompositions.

The paper shows Shampoo (with 1/2 power) is equivalent to running Adafactor in the eigenbasis of Shampoo's Kronecker-factored preconditioner, and proposes SOAP: maintain Shampoo's L=sum GG^T and R=sum G^T G accumulators per 2D layer, periodically (every f steps) extract eigenvectors Q_L, Q_R via power iteration + QR, rotate the gradient into that basis, run standard Adam first/second-moment updates in the rotated (slowly-changing) coordinate system every step, then rotate the update back. Very large layers (first/last embedding) fix Q to identity, recovering plain Adam for those. On 360M and 660M decoder LMs at 2M and 256K token batch sizes with chinchilla-optimal (20x) and longer (100x) token budgets, SOAP reduces training iterations by >40% and wall-clock time by >35% vs tuned AdamW, and ~20% vs tuned DistributedShampoo, at matched final train loss (measured via scaling-law fits across fractional-data runs). A frequency ablation shows SOAP degrades more gracefully than Shampoo as preconditioning frequency f is reduced.

Changing eligible decoder matrix-parameter updates to Nora row-orthogonal momentum projection plus row-wise normalized updates should improve held-out base-model loss under matched pretraining tokens or matched measured FLOPs because it preserves scale-invariant angular learning while approximating Muon-like row-block preconditioning at lower update cost.

Nora is a matrix optimizer for autoregressive LLaMA-style pretraining that projects momentum row-wise onto the orthogonal complement of each weight row, row-normalizes the projected direction, and updates with a width-scaled learning rate. The source claims this preserves scale-invariant angular learning, approximates Transformer row-block preconditioning, and is faster than heavier matrix optimizers.

Changing eligible decoder matrix optimization from additive Muon or AdamW updates to Pion spectrum-preserving orthogonal equivalence updates should improve held-out base-model loss under matched pretraining tokens or matched measured FLOPs because weights can rotate their input/output geometry while preserving singular spectra and stabilizing norm growth.

Pion is an optimizer that updates weight matrices by orthogonal left/right transformations, preserving singular values during training. The source's final variant combines RMS-controlled scaling, Lie-algebra first- and second-order momentum, alternating-side updates, and a second-order matrix-exponential approximation. It reports 60M design studies, muP transfer checks, and 1.3B LLaMA pretraining on C4 where Pion is comparable to Muon in validation loss and better than AdamW, plus out-of-scope fine-tuning/RL evidence.

Source claim to test: The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates.

The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates. However, like all first-order methods, Muon can become trapped in sharp local minima. In this work, we present MONA, an optimizer that bridges Muon's orthogonalization framework with curvature-aware acceleration. MONA adds an acceleration term directly into Muon's gradient processing pipeline. This term is calculated from the exponential moving average of gradient differences. We provide a detailed convergence analysis for MONA, showing that the acceleration term enables escape from sharp minima while preserving Muon's spectral-norm regularization. Empirically, MONA achieves better convergence and downstream task performance compared to both Muon and AdamW across three scales of Mixture-of-Experts pretraining, spanning from 1B to 68B parameters, with the largest model trained on 1 trillion tokens. Furthermore, we conduct supervised fine-tuning on the MOE-68B-A3B model and evaluate it on general capability, mathematical reasoning, and code generation benchmarks, where MONA achieves SOTA performance.

Changing decoder Transformer residual joins to dynamic hyper-connections with expansion rate 4 should improve held-out base-model loss under matched pretraining tokens because learned depth-connection weights can mix Pre-Norm-like long-range feature preservation with Post-Norm-like decay and reduce representation collapse without severe gradient loss.

Hyper-connections replace fixed residual joins with trainable static and dynamic depth-connection weights over expanded hidden streams. The source reports decoder LM pretraining experiments on dense OLMo-style models and OLMoE, with DHC variants improving validation loss over residual baselines at 500B tokens and ablations indicating expansion rate, dynamic connections, tanh, and trainable connection parameters matter.

Changing eligible Muon matrix-block updates to LITE flat-direction damping and learning-rate amplification should improve held-out decoder pretraining loss at matched tokens or matched measured FLOPs because the optimizer increases progress along flat, loss-reducing directions while preserving sharper directions more conservatively.

LITE modifies adaptive matrix optimizers such as Muon and SOAP by identifying sharp versus flat directions and increasing damping or learning-rate amplification in flat directions. The source reports decoder-only pretraining experiments across dense and MoE models, datasets, schedules, and 130M-1.3B scales, with Muon-LITE and SOAP-LITE improving training dynamics over their baselines.

Changing scheduled AdamW/WSD training to ScheduleFree+ should improve held-out decoder pretraining loss under matched tokens because Polyak/adaptive schedule-free iterate averaging can avoid horizon-specific decay choices while stabilizing large-batch training with inner momentum and delayed averaging.

ScheduleFree+ is a schedule-free, learning-rate-free optimizer recipe for LLM pretraining. It keeps separate gradient-evaluation and output iterate sequences, uses adaptive Polyak or inverse-gradient-norm step sizing, adds inner momentum for large-batch stability, delays iterate averaging during warmup, and uses long-run averaging weights. The source claims better loss than WSD or tuned linear-decay schedules, especially for long-duration training.

Changing causal attention to apply head-wise RMSNorm after value aggregation should improve held-out decoder pretraining loss under matched tokens because it reduces position-wise variance discrepancy that drives attention sinks and early-layer representation outliers.

The source argues that causal value aggregation creates position-wise variance discrepancy that is amplified by early FFN super neurons, producing attention sinks. It proposes head-wise RMSNorm after attention value aggregation while retaining standard softmax attention, and reports faster convergence in 152M Llama-2-style scratch pretraining on OpenWebText compared with a baseline and sigmoid-attention variant.

Changing eligible matrix-parameter updates from plain Muon-style spectral whitening to Zeta's coordinate-whitening-then-spectral-whitening update should improve held-out decoder pretraining loss under matched tokens or matched measured FLOPs because coordinate scale homogenization conditions the momentum matrix before Newton-Schulz orthogonalization.

Zeta is a matrix-aware optimizer that first whitens coordinate-wise scale heterogeneity in momentum matrices and then applies Muon-like Newton-Schulz spectral whitening. The source argues that coordinate whitening improves the conditioning required for reliable orthogonalization and reports language-model pretraining comparisons against AdamW, Muon, and AdaMuon across GPT2-Large, Qwen3 dense models up to 8B, and Qwen3-MoE.

Changing a decoder MoE by reallocating part of the sparse expert parameter budget to hashed n-gram Engram conditional memory should improve held-out base-model loss under matched tokens and active FLOPs because local static dependencies can be served by O(1) lookup, freeing attention and MoE depth for compositional computation.

Engram adds a large hashed n-gram embedding memory to a decoder MoE as a second sparsity axis. Local compressed token n-grams deterministically address sparse memory rows, and context-aware gating/fusion injects the retrieved representation into selected layers. The source reports that reallocating about 20%-25% of sparse parameter budget from MoE experts to Engram improves validation loss in allocation sweeps and that a 27B Engram model beats an iso-parameter and iso-FLOPs MoE baseline on downstream benchmarks.

Changing the matrix-parameter optimizer from AdamW/LAMB to MERIT's max-norm plus element-wise trust-ratio update should improve held-out decoder pretraining loss at matched tokens and batch size because directly bounding the max attention-logit-driving weight magnitudes reduces attention entropy collapse and yields more robust local update scaling than LAMB's l2-norm-based, weight-wise trust ratio.

Proposes MERIT, an Adam/LAMB-family optimizer for large-batch causal-LM pretraining. It replaces LAMB's l2-norm layer-wise trust ratio with a max-norm-based trust ratio (to directly bound max attention logit growth in attention Q/K weights) plus element-wise trust ratios and element-wise update clipping (magnitude <=1) for finer local scaling. Tested on GPT-2 (125M/355M/770M) and Llama (130M/350M) trained from scratch on OpenWebText/C4 at Chinchilla-optimal token budgets, comparing MERIT against AdamW, LAMB, Lion, and Sophia-G with grid-searched LR and fixed weight decay 0.1. Reports lower validation loss/perplexity and better LAMBADA/WikiText zero-shot scores, and shows MERIT sustains a 6k batch size on GPT-2 Medium without degradation versus the 480-batch AdamW baseline. Includes a nonconvex convergence proof and open-source code.

Changing causal dense attention to a gated associative successor-memory path should improve held-out decoder pretraining loss under matched tokens because bounded content-addressed retrieval can recover useful long-context successor evidence without forcing every token pair through quadratic attention.

PCAF replaces dense causal attention with a local parametric path plus an in-context associative successor cache. It writes local causal records into hash buckets, retrieves a bounded candidate set for each query, converts retrieved records into a sparse next-token cache distribution, and mixes that distribution with the parametric model using a learned gate. The source reports lower autoregressive validation perplexity and higher throughput than dense and local attention baselines at 303M parameters on WikiText-103 and PG-19, with smaller ablations supporting the cache and gate.

Changing diagonal recurrent decoder layers to PoST geometric log-decay spectra with position-adaptive scaling should improve held-out base-model loss under matched tokens because better-conditioned decay channels preserve long-range memory without wasting channels at early positions.

PoST modifies diagonal linear recurrent layers by enforcing geometrically structured log-decay spectra and scaling those spectra by position so channels cover the effective dependency range at each timestep. The source reports equal-hyperparameter and equal-compute language-model pretraining comparisons on FineWeb-Edu for Mamba-2, RWKV-7, and Gated DeltaNet at about 180M parameters, plus Mamba-2 at about 440M, with improvements described on zero-shot benchmarks and long-context retrieval.

Changing decoder causal attention to MEA with separate cross-head linear combinations over K and V plus head-level GroupNorm should improve held-out base-model loss under matched pretraining tokens because explicit inter-head communication can share key/value evidence across heads while normalization prevents the cross-head transform from degenerating during training.

Multi-head Explicit Attention adds learnable cross-head linear combinations for key and value vectors plus head-level GroupNorm/RMSNorm. The supplied evidence reports decoder LM from-scratch pretraining comparisons at a LLaMA3.2-1B-style scale, with MEA showing lower fitted test loss and greater tolerance for large learning rates than standard attention and related ablations.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_9bd5f867cb7746c9

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

016830f35cb93c95f9c16395d88c0a2c6b0e3702

Branch

auto-research/idea_90da09ea90ff46c5/probe-dp4-candidate_c060f443d83e26be45a7

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/probe-dp4-candidate_c060f443d83e26be45a7

Configuration

/persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_c060f443d83e26be45a7.local.yaml

Config SHA-256 (recorded, unverified)

379f18f5a28044fd916642e1c493d73fdc91173b19a4f63b2c5a63cd25d59137

Actual config SHA-256

379f18f5a28044fd916642e1c493d73fdc91173b19a4f63b2c5a63cd25d59137

Resolved config SHA-256 (recorded, unverified)

29e4cc38cb5f91d5051ca411b1cb4f09c830b0d3063d53aaa3a2f642fdfccef1

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_c060f443d83e26be45a7.local.yaml

Changing the decoder block/input format to State-Prediction Separation, where each real token is paired with a predict token that carries next-token loss under an SPS attention mask, should improve held-out base-model loss at matched training FLOPs because separating state maintenance from prediction may reduce interference between representation storage and next-token readout.

State-Prediction Separation interleaves each input token with a predict token sharing the same RoPE position, computes next-token loss only on predict positions, and uses a custom causal attention pattern so input tokens maintain state while predict tokens perform readout. The source reports validation-loss and downstream gains at GPT-2 scales but identifies the key confound: the method costs about twice the training compute per real token, so the local test must be judged at matched FLOPs.

Changing AdamW/Sophia/Shampoo preconditioned updates to cross-fitted numerator/preconditioner estimates plus variance-corrected inverse preconditioning should improve held-out decoder pretraining loss under matched tokens or matched measured FLOPs because it reduces finite-sample gradient-preconditioner coupling and nonlinear inverse-estimator bias.

The source argues that stochastic preconditioned optimizer updates are biased because the gradient and preconditioner are estimated from the same minibatch and because inverting a noisy preconditioner is nonlinear. It proposes cross-fitted microbatch preconditioning plus a delta-method inverse correction, reporting lower held-out pretraining loss for AdamW, Sophia-G, and Shampoo on Qwen2.5-0.5B FineWeb-Edu runs.

Changing decoder causal attention from full attention to NSA hierarchical trainable sparse attention should improve held-out base-model loss or quality-throughput tradeoff under matched pretraining tokens because compressed global context, learned blockwise selection, and a local sliding window preserve salient long-range information while reducing attention compute.

NSA replaces full attention with a trainable hierarchical sparse pattern that combines compressed global tokens, blockwise selected KV blocks, and a sliding local window, paired with hardware-aligned Triton kernels. The source reports a 27B-total / 3B-active MoE pretrained for 270B tokens where NSA has a smoother/lower pretraining loss curve than Full Attention and better long-context benchmark averages, while reducing long-context attention latency.

Changing decoder causal attention from full attention to end-to-end learned hierarchical landmark sparse attention should improve held-out base-model loss or equal-compute long-context quality under matched tokens or measured FLOPs because LM-loss-trained chunk retrieval can select useful KV blocks more accurately than nonparametric sparse chunk selection while bounding attention compute.

HiLS attention replaces dense causal attention with chunk-wise sparse retrieval learned through the language-modeling loss. Each chunk has a learnable landmark-derived summary; queries score chunk summaries, retrieve top-k chunks, attend within each retrieved chunk and a local window, then fuse chunk outputs using retrieval scores. The supplied evidence reports comparable perplexity to full attention, stronger long-context retrieval, a 1.4B from-scratch run on 300B tokens, and 7B continued-pretraining conversion results.

Changing eligible decoder matrix optimization from Adam/Muon-style updates to SCALE column-normalized SGD with last-layer-only momentum should improve held-out base-model loss or memory-adjusted training efficiency under matched tokens or matched measured FLOPs because column-wise gradient normalization supplies the essential adaptive scaling while avoiding most optimizer state.

The source proposes SCALE, a minimalist memory-efficient optimizer for LLM pretraining. Matrix parameters use column-wise gradient normalization instead of Adam moments, first-order momentum is applied only to the output layer, and vector parameters remain on Adam. Reported C4 LLaMA experiments from 60M to 1B compare against Adam, Muon, GaLore, Fira, APOLLO, SWAN, and Adam Stable-SPAM, with a 7B 100B-token stability run.

Changing MoE optimizer and initialization scaling to Complete-muE transfer rules should improve held-out decoder pretraining loss under matched tokens or matched measured FLOPs because dense-calibrated hyperparameters transfer more reliably across active width, activated experts, total experts, and per-expert workload.

Complete-muE proposes a compositional hyperparameter-transfer rule for dense FFN, Dense MoE, and sparse MoE Transformer blocks. It combines active-width muP, normalized router scale, and expert-side SDE reasoning so dense-tuned initialization and AdamW settings transfer across activated experts, total experts, granularity, shared experts, width, batch, and duration changes.

Changing matrix-preconditioned optimizer hyperparameters to muP-scaled learning rates plus 1/width independent weight decay should improve held-out decoder pretraining loss at matched tokens or matched measured FLOPs because optimizer optima transfer more reliably across width and depth instead of drifting with scale.

The source argues that matrix-preconditioned optimizers only retain apparent speedups over AdamW when their learning rate and independent weight decay are transferred with scale-aware rules. It reports that muP-derived learning-rate scaling plus 1/width independent weight decay gives Muon, SOAP, and Shampoo stable near-1.4x compute-efficiency gains on dense Llama-style language models from 190M to 1.4B, while incorrect scaling causes the gains to vanish.

Source claim to test: Large language models are typically deployed as monolithic systems, requiring the full model even when applications need only a narrow subset of capabilities, e.g., code, math, or domain-specific knowledge.

Large language models are typically deployed as monolithic systems, requiring the full model even when applications need only a narrow subset of capabilities, e.g., code, math, or domain-specific knowledge. Mixture-of-Experts (MoEs) seemingly offer a potential alternative by activating only a subset of experts per input, but in practice, restricting inference to a subset of experts for a given domain leads to severe performance degradation. This limits their practicality in memory-constrained settings, especially as models grow larger and sparser. We introduce EMO, an MoE designed for modularity-the independent use and composition of expert subsets-without requiring human-defined priors. Our key idea is to encourage tokens from similar domains to rely on similar experts. Since tokens within a document often share a domain, EMO restricts them to select experts from a shared pool, while allowing different documents to use different pools. This simple constraint enables coherent expert groupings to emerge during pretraining using document boundaries alone. We pretrain a 1B-active, 14B-total EMO on 1T tokens. As a full model, it matches standard MoE performance. Crucially, it enables selective expert use: retaining only 25% (12.5%) of experts incurs just a 1% (3%) absolute drop, whereas standard MoEs break under the same setting. We further find that expert subsets in EMO specialize at semantic levels (e.g., domains such as math or code), in contrast to the low-level syntactic specialization observed in standard MoEs. Altogether, our results demonstrate a path toward modular, memory-efficient deployment of large, sparse models and open new opportunities for composable architectures.

Add the exact paper GRAPE-AP path-integral attention-logit bias while retaining the current Titan rotary bilinear attention term. For each attention head, form positional probes from token-local features with a learned linear projection followed by RMS normalization, compare fixed commuting rotations of the probes at relative offsets, pass the scaled similarity through log-sigmoid to obtain bounded non-positive edge potentials, and causally sum those edges from each key position to the current query position. This source-faithful, content-dependent forgetting path should improve equal-token held-out decoder pretraining loss and sample efficiency by suppressing irrelevant earlier keys while preserving global attention, row-wise path composition, and cacheable causal semantics. Gate 1 must implement GRAPE-AP without KV-shift and must not reuse the unsupported multiplicative GRAPE-M mechanism.

The local 1B GRAPE-AP candidate is feasible with a small attention-path change plus focused tests. The exact mechanism should live in the existing Transformer and TransformerMoE attention modules, use flex attention score_mod rather than dense bias tensors, and wire new probe parameters through the existing MoE and MTP parallelizers. The 17B baseline is also feasible only with config changes: the checked-in cuDNN MTP MoE config needs a matched flex candidate/control pair because cuDNN has no current additive score boundary. No unresolved local scientific choice blocks implementation; acceptance rests on equation/reference tests, disabled equivalence, streaming/cache consistency, AC recomputation, and real TP/FSDP gradient coverage.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_96218a5b87454a46

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

ced6630f3568bfe501aae3a72720a7c0c688b3ed

Branch

auto-research/idea_da01d43462214529/smoke_0450a7111cdbb932a26f

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/smoke_0450a7111cdbb932a26f

Configuration

/persistent/.local/state/george-auto-research/run_configs/smoke_0450a7111cdbb932a26f.smoke.yaml

Config SHA-256 (recorded, unverified)

9e93d47fd702ead0006435026bbc6e678f71ac477c10d3d953757603d211198c

Actual config SHA-256

9e93d47fd702ead0006435026bbc6e678f71ac477c10d3d953757603d211198c

Resolved config SHA-256 (recorded, unverified)

a5f0fa9471a185752377651254a09e6b8304fb277251b09d64c4ad08901373b2

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/smoke_0450a7111cdbb932a26f.smoke.yaml

Add only Forgetting Attention to each causal attention layer while retaining the Titan baseline RoPE, QK norms, gated-attention output, global and sliding-window masks, and MLP. For each token and head, compute the source scalar data-dependent sigmoid forget gate, accumulate its log values along the sequence, and add the resulting query-key decay difference to the unnormalized causal softmax logits before softmax; initialize the data-dependent gate projection to zero as in the paper. The learned content-dependent decay should lower equal-token held-out loss by suppressing irrelevant history without reducing global retrieval capacity. Reject the claim if final loss does not beat the matched RoPE control beyond calibrated local noise, if learned gates become data-independent without benefit, or if a kernel-correct implementation cannot sustain the local throughput target.

The local 1B Titan implementation is feasible without a new model wrapper. The smallest faithful change is an optional per-head forget-gate projection inside the existing dense and MoE Attention classes, feeding a float32 cumulative log-forget tensor into FlexBackend as an additive query-minus-key score bias before softmax. This preserves RoPE, QK norms, global/SWA/document masks, existing gated-attention output, MLP, optimizer ownership, and loss semantics. Supplied history shows related FP8 gated-XSA work, but it is an output-gate projection change rather than this attention-logit forget mechanism, so it should not be reused as the mechanism. Prior FoX implementation failures are addressed by reversible activation fields, a source-faithful zero-logit gate instead of a neutralized variant, a cloned Flex score_mod read for backward, and focused tests including distributed TP/FSDP gradients.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

79d54d799dfab9ec07ad43c7a225c1d99a420ee6

Branch

external-observation

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/local-pp3dp2-candidate_798e2501474121f461d5

Configuration

/persistent/.local/state/george-auto-research/run_configs/fox-pp3dp2-control-seed43.full.yaml

Config SHA-256 (recorded, unverified)

b737304c5109c650f3d286e643f53e9016bf1cc7850d36fcf6dbf43a2925e6a3

Actual config SHA-256

b737304c5109c650f3d286e643f53e9016bf1cc7850d36fcf6dbf43a2925e6a3

Resolved config SHA-256 (recorded, unverified)

b737304c5109c650f3d286e643f53e9016bf1cc7850d36fcf6dbf43a2925e6a3

Seed

43

external observed titan-pretrain --config <FoX seed-43 control config>

Changing the last third of decoder attention layers to layer-specific Bank of Values token-id value lookups should improve held-out base-model loss under matched pretraining tokens because deep-layer value vectors benefit from preserving context-free token identity while queries and keys still carry contextual routing information.

Bank of Values replaces standard residual-stream value projections in the last third of decoder attention layers with layer-specific token-id lookup tables scaled by a learnable coefficient. The source argues and ablates that deep-layer values prefer context-free original token information, while queries and keys still use contextual residual features. Reported 135M and 780M decoder-only experiments show validation-loss improvements over standard attention, with component ablations for source, coefficient, sharing, and target-layer choices.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/implementation/action_5caefcbb202d4e61

Configuration

scripts/runs/auto_research_bov_control.yaml

Seed

43

No training launch command was recorded. The provenance above identifies the exact revision and configuration evidence that is available.

Changing decoder-only Transformer blocks from uniform hidden width to an x-shaped layerwise width schedule with wider early and late layers, narrower middle layers, and parameter-free residual coordinate copying should improve held-out base-model loss at matched training tokens or matched measured FLOPs because capacity is allocated to depth regions that use residual-stream dimensions more effectively while the middle layers avoid representation collapse.

The source proposes a decoder-only Transformer with nonuniform hidden width across depth: early and late layers are wider, middle layers are narrowed, and inactive residual-stream coordinates bypass narrow layers through parameter-free copying, with zero padding only when no prior coordinate exists. The default recipe uses an x-shaped schedule with bottleneck index around 0.75L and bottleneck dimension around 0.3d, rounded to compatible dimensions. Reported experiments compare parameter-matched constant-width baselines across 200M-2B dense LMs and a 3B total/1B active MoE on DCLM with cl100k_base, muP-aware AdamW, SwiGLU, bf16, and power LR decay. The x shape outperforms other variable-width shapes, copying features beats zero padding or learned projection for expansion, and the final x-shaped models reportedly improve language-modeling loss while reducing attention/KV-related cost.

Changing each decoder attention block to apply a causal residual depthwise k=3 convolution to the projected Q, K, and V tensors before attention should improve held-out base-model loss under matched pretraining tokens because local neighboring-token context is injected into attention inputs before global aggregation without materially changing parameter count.

The source studies lightweight causal depthwise Conv1D modules inside decoder Transformer blocks. Its placement ablation over 17 locations finds the best reported loss/perplexity when convolution is applied to the projected Q, K, and V representations before attention. Follow-up ablations favor a residual k=3 depthwise convolution without extra normalization, activation, a second module, or multi-branch reparameterization.

Add identity-initialized causal dynamic depthwise short convolutions at two architectural locations in each decoder block: directly after the key and value projections, and on the attention-output and MLP-output residual branches immediately before each rejoins the main residual stream. Local input-dependent filtering at both memory formation and residual writing should improve held-out pretraining loss and sample efficiency without replacing global attention.

Implementation audit requested repair: Current scientific review is identifiable: scientific_review.review_id=review_9494e7cbae2d4b03 and scientific_review_sha256=369e02f92e1428fbd5b48b0eaa20c258eef8ae35be5e1d5cd3cfe8a9d97033ea.; Host verification is not clean: host_verification.verified is false and format_normalization failed on tests/titan/test_dynamic_shortconv.py with PT018, so this candidate is not ready before a GPU run.; Activation is not valid at the required boundary. The current diff adds the typed dynamic_shortconv_v1 config field and rank=16 default, but the supplied parsed activation/off switch still uses flat model.init_args.config.dynamic_shortconv_enabled and flat apply/rank/kernel overrides rather than job_config.model.init_args.config.dynamic_shortconv_v1.enabled=false for control and the typed bundle for candidate.; Repository alignment fails for the MTP distributed boundary. The normative repository evidence requires ShortConv TP/FSDP handling in poolside/titan/parallelisms/parallelize_transformer_mtp_moe.py for MTP blocks, but the current unified_diff does not modify that file.; The MoE attention tensor-parallel path remains semantically unsafe. Attention.k_shortconv and Attention.v_shortconv are constructed with global config.n_kv_heads * head_dim, while attention_tensor_parallel=true makes wk/wv produce local K/V feature shards before the ShortConv call; the module only slices weights inside forward and does not prove the parameter geometry is consistently sharded with the local K/V layout.; The TP plans add proj_in/proj_out entries and manually replicate static_w, but there is no focused distributed test in the current diff asserting exact placements and finite gradients for every k_shortconv, v_shortconv, attn_write_shortconv, mlp_write_shortconv proj_in, proj_out, and static_w parameter. This would not reject plausible but wrong sharding variants.; Compute accounting is not satisfied before launch. The downstream protocol requires implementation exposure of extra forward timing, but the current unified_diff does not add timing exposure or a mechanism to report distributed communication changes.; Coverage is still incomplete for the exact launch requirements: tests cover causal equation, K/V-only behavior, residual-write source ordering, identity recovery, typed MTP config presence, and resolved-sublayer checkpoint cache behavior, but not the actual typed activation/off-switch config path used by the runner, the required MTP parallelizer behavior, or distributed placement under the target topology.; Current review is identifiable: scientific_review.review_id=review_c4ddc421a1874f72 and scientific_review_sha256=522e111e540624808e9b8861a396919f14105a2bb69cd5f13f2fa6a16a8d5641.; Host verification is not clean: host_verification.verified is false and the touched_tests check failed, so this candidate is not ready before a GPU run.; Activation is not valid against the repository requirement. The current diff wires flat dynamic_shortconv_* ModelArgs and TransformerMoEArgs fields plus a derived property, and parsed activation uses model.init_args.config.dynamic_shortconv_enabled, while the normative activation requires the typed job_config.model.init_args.config.dynamic_shortconv_v1 bundle with enabled=false as the off-switch.; The implementation uses rank=15 in DynamicShortConvV1Config defaults, assertions, model config fields, tests, and parsed activation. Repository evidence requires rank=16, so this is an invented activation value.; Repository alignment fails for required MTP propagation. The diff changes ModelArgs inheritance surfaces but does not implement or test the required typed dynamic_shortconv_v1 bundle in both TransformerMoE base config and mtp_config for MTPTransformerMoE.; Repository alignment fails for distributed placement. The current diff does not modify poolside/titan/parallelisms/parallelize_transformer_moe.py or poolside/titan/parallelisms/parallelize_transformer_mtp_moe.py, and adds no

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

03bd60411c194302545a359c8c481ee56ed838bd

Branch

preserved/shortconv-fae7dad

Preserved worktree

/persistent/.local/state/george-auto-research/preserved_worktrees/shortconv-fae7dad

Configuration

/persistent/.local/state/george-auto-research/run_configs/shortconv-v1-full-pp2dp4-seed44-r1-0717.local.yaml

Config SHA-256 (recorded, unverified)

ada7d078d3ab5748f3d29987f3cd862b157d2bf0e2bc37d1e4487b6b30e85af4

Actual config SHA-256

ada7d078d3ab5748f3d29987f3cd862b157d2bf0e2bc37d1e4487b6b30e85af4

Resolved config SHA-256 (recorded, unverified)

56e0cf76583107ce1e7098745881f6ff89f31e70a9c0cb59683b5e9d7aa97831

Seed

44

bash /persistent/.local/state/george-auto-research/launch_manifests/shortconv-v1-final-run-one-seed44-0717.sh shortconv-v1-full-pp2dp4-seed44-r1-0717

Apply the official Thinking Machines unit-Stiefel Manifold Muon update to one audited class of eligible 2D hidden weights. Transpose W and G iff W.shape[0] < W.shape[1]; at eta=0 initialize W with msign(W); initialize Lambda=-0.25*(W.T@G+G.T@W), compute A=msign(G+2*W@Lambda) and H=W.T@A+A.T@W, then retract with new_W=msign(W-eta*A) and transpose back iff required. The unit-Stiefel constraint may improve held-out decoder pretraining loss or stability over standard Muon at matched tokens, while initialization and forward/gradient-scale consequences are measured rather than canceled by an invented fan-scaled radius.

The request is locally feasible as a narrow optimizer-only change. The target tree already has the needed Muon owner-rank P2P gather/scatter, replica broadcast, parameter grouping, search_param_name matching, optimizer metrics, and MoE/MTP wrapper wiring. The smallest diff is to add a literal official unit-Stiefel Manifold Muon update, split exactly layers.0.feed_forward.w1.weight into its own Muon group when the regex is non-null, run the replacement full-logical update on the existing owner path, and activate through generated candidate/control configs derived from scripts/runs/tiny.yaml. No scientific choice blocks the local 1B implementation; the main risks are performance from tensor-valued early stopping and scale compatibility that must be tested before 17B launch.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_88962dcaaedf4b0c

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

ef0e285c7253f02cc1df7c90ae5a7e216b8a2712

Branch

auto-research/idea_80448b61b5ec4507/probe-dp4-candidate_47a203c910e9b52f9d44

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/probe-dp4-candidate_47a203c910e9b52f9d44

Configuration

/persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_47a203c910e9b52f9d44.local.yaml

Config SHA-256 (recorded, unverified)

ce44be7cff62d17680ab4adf2a37564d02d150023122f78f03653a37a89aa4d9

Actual config SHA-256

ce44be7cff62d17680ab4adf2a37564d02d150023122f78f03653a37a89aa4d9

Resolved config SHA-256 (recorded, unverified)

55f59471af71699036b39f2973f66fe78b2de3396e9be2e80652a21266517a65

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_47a203c910e9b52f9d44.local.yaml

Adding StableMask's parameter-free, head-wise gamma=0.5 future-position pseudo logits before softmax and zeroing future probabilities after softmax will let real-token attention mass fall below one, reduce attention sinks, encode absolute position without leakage, and lower held-out next-token loss/AUC under the fixed Titan TPP30 protocol.

The primary paper evaluates 71M through 1.4B causal decoder LMs and reports gains across datasets and positional encodings. In its 1.4B OpenLLaMA/RoPE Pile experiment, perplexity is 15.4 to 15.0 at 5B tokens and 10.7 to 10.4 at 25B tokens. The method is parameter-free and the paper describes integration with optimized attention.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_25def381d850468b

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

98d22f5c81ff61650a9f7f5bd539abd96461d27f

Branch

auto-research/idea_390098622868482f/probe-dp4-candidate_d02a68c08381af813a9d

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/probe-dp4-candidate_d02a68c08381af813a9d

Configuration

/persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_d02a68c08381af813a9d.local.yaml

Config SHA-256 (recorded, unverified)

998b5e0171d282d36e14e557765550f21a05fd2327c118eae4ae1a1f8498dff4

Actual config SHA-256

998b5e0171d282d36e14e557765550f21a05fd2327c118eae4ae1a1f8498dff4

Resolved config SHA-256 (recorded, unverified)

5cd1adbdcb37d72e1546195aec4e6433e762daed602d365c0d6985f05e16ac7a

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_d02a68c08381af813a9d.local.yaml

For each attention layer n >= 2, replacing only its value tensor V_n with 0.5 * V_1 + 0.5 * V_n, while retaining that layer's own Q, K, attention matrix, output projection, and all ordinary hidden-state residuals, will preserve early token information and lower held-out next-token loss/AUC under the fixed Titan TPP30 protocol without a material systems regression.

The v5 primary paper tests Llama-like pre-norm, SwiGLU, RoPE decoder LMs on SlimPajama from 82M to 468M parameters at 20B tokens and a 1.6B model at 200B tokens. It reports equivalent validation loss with 16.11% fewer parameters or 20.3% fewer training tokens. At 468M, the reported WikiText perplexity is 24.8 for Transformer and 23.8 for fixed Identity-ResFormer.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_89e7f3b061ed4fb9

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

1a99e336d0011431ea9a5ca0cdbd9977b5cf2e2c

Branch

auto-research/idea_c82794cd571f4819/probe-dp4-candidate_d9e7a06283f56e4bd236

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/probe-dp4-candidate_d9e7a06283f56e4bd236

Configuration

/persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_d9e7a06283f56e4bd236.local.yaml

Config SHA-256 (recorded, unverified)

d2438165ce97f77b7f4235658473ccfe4ce8becd6a91a236b79453fb39cafb38

Actual config SHA-256

d2438165ce97f77b7f4235658473ccfe4ce8becd6a91a236b79453fb39cafb38

Resolved config SHA-256 (recorded, unverified)

5b2c5247ac849cbd3dbe6e171a92f11839921914492272436dbdf8e423b37685

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_d9e7a06283f56e4bd236.local.yaml

Changing momentum-based optimizer updates to an elementwise cautious variant that applies only update coordinates aligned with the current gradient should improve held-out decoder pretraining loss at matched tokens because it suppresses momentum-driven overshoot and oscillatory coordinates while preserving the base optimizer's main hyperparameters.

The source proposes a cautious variant of momentum-based optimizers such as AdamW and Lion: suppress update coordinates when the proposed momentum update and current gradient are not aligned. It claims this preserves useful stability properties and improves LLM pretraining perplexity in 100M LLaMA C4 runs and FineWeb-Edu scaling comparisons, with minimal hyperparameter changes.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_60a5a690ca6e410c

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

caa6e0f4300f8eac0b88e3754d9cfd4532e199fa

Branch

auto-research/idea_720cfdb19bfc4041/probe-dp4-candidate_e64ea0c60ca321c56c6b

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/probe-dp4-candidate_e64ea0c60ca321c56c6b

Configuration

/persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_e64ea0c60ca321c56c6b.local.yaml

Config SHA-256 (recorded, unverified)

fee71d876b6d6579a15e1a06eb968f2ed19324552b14fd34522599eafed98ec3

Actual config SHA-256

fee71d876b6d6579a15e1a06eb968f2ed19324552b14fd34522599eafed98ec3

Resolved config SHA-256 (recorded, unverified)

51c78e6513e086cd950f6655bcf7fd70090085edb503b6dab7105592ce377221

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_e64ea0c60ca321c56c6b.local.yaml

Changing the pretraining learning-rate schedule from fixed-horizon cosine/WSD to a warmup plus shifted inverse-square-root base with final linear decay should improve held-out base-model loss at matched tokens and optimizer steps because the horizon-independent base schedule avoids suboptimal peak-rate and stable-phase choices when the effective training horizon changes.

WSqD replaces WSD's constant stable learning-rate phase with a shifted inverse-square-root base schedule followed by a final linear cooldown. The claimed benefit is that the base schedule is horizon-independent, so the peak learning rate need not be retuned when training is extended. The source reports language-model pretraining experiments on SlimPajama where WSqD matches or outperforms tuned WSD and other schedules across multiple horizons while reusing one peak learning rate.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

f381a509a71eab4bab1853e236964c52b9a0f846

Branch

auto-research/idea_e17af23e8a484933/local_b717ec317f1d4a7bd5e9_recovery_nograph

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/local_b717ec317f1d4a7bd5e9_recovery_nograph

Configuration

/persistent/.local/state/george-auto-research/run_configs/local_b717ec317f1d4a7bd5e9_recovery_dp4_fullac.local.yaml

Config SHA-256 (recorded, unverified)

ae629fadf6961fb921c46a50dc85b18af4ca3c4bc21bc30eec8f3feb1a51fc44

Actual config SHA-256

ae629fadf6961fb921c46a50dc85b18af4ca3c4bc21bc30eec8f3feb1a51fc44

Resolved config SHA-256 (recorded, unverified)

8a35de507e06b8ef748b5c6c5e60cfbcedb8736ecb40f5a66b4ea2fec86495ba

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/local_b717ec317f1d4a7bd5e9_recovery_dp4_fullac.local.yaml

Changing matrix-parameter optimization from AdamW or unscaled Muon to weight-decayed, update-RMS-matched Muon should improve held-out decoder pretraining loss under matched tokens or matched measured FLOPs because orthogonalized matrix updates provide better spectral-norm-conditioned descent while the added scaling and weight decay stabilize large-scale RMS growth.

The source claims Muon scales to large decoder-only LLM pretraining when two recipe changes are added: AdamW-style weight decay to control weight and activation RMS growth, and per-matrix update scaling/RMS matching so Muon updates resemble AdamW update magnitudes. It reports dense scaling-law evidence versus tuned AdamW and a large MoE model trained with the resulting optimizer recipe.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_915f27bf5bba45ac

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

f5b95625e729ea067bc6685f742bf5b53a774b33

Branch

auto-research/idea_2507927fd7424bc4/probe-dp4-candidate_80999ab4e48e7676b9ef

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/probe-dp4-candidate_80999ab4e48e7676b9ef

Configuration

/persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_80999ab4e48e7676b9ef.local.yaml

Config SHA-256 (recorded, unverified)

dd8cbbe59e3d56e15aeec493e642f83e77096709a766c670e18d4042677e9cbb

Actual config SHA-256

dd8cbbe59e3d56e15aeec493e642f83e77096709a766c670e18d4042677e9cbb

Resolved config SHA-256 (recorded, unverified)

81fb201229ba3912fc9850d60a2693b4a470b4e66f66b5fa94f0104df8492f3c

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_80999ab4e48e7676b9ef.local.yaml

Changing uniform per-layer MLP intermediate width to a cosine front-loaded taper at fixed total parameters and active FLOPs should improve held-out decoder pretraining loss under matched tokens because earlier MLPs write more novel residual information while later MLPs are more redundant and can use less capacity.

The source proposes Tapered Language Models: replace a constant MLP intermediate dimension with a monotone per-layer width schedule under a fixed total parameter budget. The reported best default is a cosine taper with start/end MLP widths at 1.5/0.5 times the uniform width, rounded to efficient multiples while preserving total MLP parameters. Other dimensions and training hyperparameters are held fixed. The claimed mechanism is that early MLPs add more novel residual information while later MLPs increasingly reinforce existing residual content, so capacity should be front-loaded across depth.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_48234c89910b4b42

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

cb2457dd57a37527fc6d84e95fdd0eafebf6fbbd

Branch

auto-research/idea_cc4eda14e35f47fa/probe-dp4-candidate_314013bcdd62017902e1

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/probe-dp4-candidate_314013bcdd62017902e1

Configuration

/persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_314013bcdd62017902e1.local.yaml

Config SHA-256 (recorded, unverified)

dbf6336c5890dfe8c6f0746ba04ec21270b4a0047c17fa8454071720f7fabadc

Actual config SHA-256

dbf6336c5890dfe8c6f0746ba04ec21270b4a0047c17fa8454071720f7fabadc

Resolved config SHA-256 (recorded, unverified)

fa30896fbae272fd45bd945523bf2675c2ff66024eee6f390759abd009283d1c

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_314013bcdd62017902e1.local.yaml

Changing MoE load-balancing loss from micro-batch-local frequency estimates to global-batch frequency estimates should improve held-out decoder pretraining loss at matched tokens because it relaxes per-sequence uniform routing pressure and permits domain-specialized experts while preserving corpus-level load balance.

The source argues that standard MoE load-balancing loss is often computed within tiny micro-batches, effectively forcing each sequence to spread tokens uniformly across experts and suppressing specialization. It proposes computing the expert selection frequency term over the global batch via synchronization across micro-batches, preserving corpus-level balance while allowing sequence or domain-level routing concentration.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_b0d79214316e445a

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

2a17ae471879b4261ec94d27d897928951f11402

Branch

auto-research/idea_fa55e0cc92d34f43/probe-dp4-candidate_4bd40a476607d711d3ce

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/probe-dp4-candidate_4bd40a476607d711d3ce

Configuration

/persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_4bd40a476607d711d3ce.local.yaml

Config SHA-256 (recorded, unverified)

f6d8ef544ef59d3cec5a9d47cf917eb654b35495a0fb31bf115a4e703895dbe3

Actual config SHA-256

f6d8ef544ef59d3cec5a9d47cf917eb654b35495a0fb31bf115a4e703895dbe3

Resolved config SHA-256 (recorded, unverified)

0e85f444bd17a45dd8859e62f60241705f69dda6d90ab1de50cef2fa7777127a

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_4bd40a476607d711d3ce.local.yaml

Changing optimizer assignment so only VO attention weights and FFN matrices use Muon while other parameters retain the baseline optimizer should improve held-out decoder pretraining loss at matched tokens because Muon's orthogonalized matrix updates better fit associative-memory outer-product learning on heavy-tailed token distributions.

The source argues that Muon's advantage over Adam in LLM pretraining comes primarily from transformer associative-memory parameters: attention value/output projections and FFN matrices. It reports FineWeb NanoGPT ablations that apply Muon to individual transformer components and combined subsets, finding VO and FFN blocks are the main contributors, and supports the claim with a one-layer associative-memory analysis under heavy-tailed class frequencies.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_f2ee2bfd1b4f4963

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

4832b2c2d6858b859e24f90000ae1c5a08ef9c12

Branch

auto-research/idea_de12bc7d887e43a8/local-pp3dp2-candidate_b6e3c2379498d795cac6

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/local-pp3dp2-candidate_b6e3c2379498d795cac6

Configuration

/persistent/.local/state/george-auto-research/run_configs/local-pp3dp2-candidate_b6e3c2379498d795cac6.local.yaml

Config SHA-256 (recorded, unverified)

fd9d8bef70f96bc0c5e20cf75b1e47c66caa7274050b5e4a44bfb1416a9ce93f

Actual config SHA-256

fd9d8bef70f96bc0c5e20cf75b1e47c66caa7274050b5e4a44bfb1416a9ce93f

Resolved config SHA-256 (recorded, unverified)

a71773ccd73d6c592c11e2a61a152189d1ade29ab1969ea81f6d419b7027b07c

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/local-pp3dp2-candidate_b6e3c2379498d795cac6.local.yaml

Changing fixed RoPE frequencies to shared learnable per-frequency log scalars should improve held-out base-model loss at matched pretraining tokens and approximately matched FLOPs because the model can preserve useful high-frequency positional bands while suppressing bands better used for content-invariant matching.

LeRoPE replaces fixed RoPE frequency bands with shared learned log-space scalars alpha_m, initialized at zero so the model starts exactly as standard RoPE. The source reports decoder-only LM pretraining ladders where LeRoPE beats standard RoPE and partial RoPE, with the largest reported model requiring about 3.4% more RoPE compute to match LeRoPE loss. The method uses one shared frequency vector across layers and heads, omits weight decay on those parameters, and clips their gradients independently.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_9a63e405f5b94801

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

8f926c68fc4bee349b35d9a39de086e9ef4dd090

Branch

auto-research/idea_e9e639e8151d4f93/local-dp4-candidate_85350819372fec6fb2ef

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/local-dp4-candidate_85350819372fec6fb2ef

Configuration

/persistent/.local/state/george-auto-research/run_configs/local-dp4-candidate_85350819372fec6fb2ef.local.yaml

Config SHA-256 (recorded, unverified)

5f118eee28c6baae0b2794c082a14b55091186baf09e44679ead74d16c6194b4

Actual config SHA-256

5f118eee28c6baae0b2794c082a14b55091186baf09e44679ead74d16c6194b4

Resolved config SHA-256 (recorded, unverified)

a74d05390ef4466830dee96f2c6cf3009d84cfe01a15ab5bba777121097f01c7

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/local-dp4-candidate_85350819372fec6fb2ef.local.yaml

Changing AdamW training to add PACE's clipped per-coordinate pullback from live weights toward the EMA iterate should improve held-out decoder pretraining loss under matched tokens because the returned averaged model is explicitly optimized to stay closer to the low-oscillation iterate-average estimator while retaining AdamW preconditioning.

PACE is a lightweight AdamW wrapper for iterate-averaged language models. It keeps an EMA of the weights and adds a clipped, per-coordinate pullback from the live weights toward the EMA, with gain scaled by the Adam second-moment preconditioner, EMA power, learning rate, pullback strength, and update frequency. The source reports GPT-2 124M FineWeb pretraining improvements over AdamW and EMA baselines across schedules, plus 1-2B fine-tuning evidence.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_3207ed0deb8f4d14

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

f2ef1d9305f01f6fadbba737fa9db0a1202cfd05

Branch

auto-research/idea_1b09901ac1144189/probe-dp4-candidate_b855920271e261d42c4b

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/probe-dp4-candidate_b855920271e261d42c4b

Configuration

/persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_b855920271e261d42c4b.local.yaml

Config SHA-256 (recorded, unverified)

f4131a6965931ae2cf3e2f0a57e5f40453b294b9200e92f860f870501c0c82d9

Actual config SHA-256

f4131a6965931ae2cf3e2f0a57e5f40453b294b9200e92f860f870501c0c82d9

Resolved config SHA-256 (recorded, unverified)

7f3349b1c146df658d2b9461df1b60b5daaefed073b6ad8ba7f5b32970bc319b

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_b855920271e261d42c4b.local.yaml

Changing the pretraining optimizer from scheduled AdamW or Muon to AMUSE, which combines Muon matrix updates with schedule-free iterate averaging and a time-varying gradient-evaluation interpolation, should improve held-out base-model loss at matched pretraining tokens and optimizer steps because early gradients are evaluated near the fast Muon trajectory while later training shifts toward the averaged sequence to suppress high-curvature oscillations.

AMUSE modifies Muon optimization by evaluating gradients at an interpolation between the fast Muon sequence and an averaged sequence, with a time-varying beta_t that starts closer to the fast trajectory and gradually shifts toward the stable average. The supplied record reports decoder-only Llama-style pretraining on FineWeb-100B at 124M, 720M, and 1B parameters, with tied embeddings, SwiGLU, RMSNorm, RoPE, sequence length 512, GPT-2 tokenizer, and bfloat16 training. AMUSE fixes Muon momentum at 0.95, uses AdamW-style handling for non-Muon parameters with beta_2 fixed to 0.999, and sweeps beta_1 in {0.4, 0.6} and rho in {0.6, 0.8}. The claimed mechanism is that Muon accelerates low-curvature bulk progress but can amplify dominant-direction oscillations, while schedule-free averaging stabilizes late training without requiring an explicit LR decay schedule.

Inspect the recorded revision and configuration evidence. The command below is provenance, not an automatic relaunch.

Implementation action

action_982519ded7ca403c

Base commit

14048b8a98717d7037a6733b093ababc8560dc8f

Candidate commit

a5a776894322e210c0235bb9ec66a9782d5f381e

Branch

auto-research/idea_d01bc6b550634654/probe-dp4-candidate_96c4b1703a57d16801dc

Preserved worktree

/persistent/.local/state/george-auto-research/worktrees/probe-dp4-candidate_96c4b1703a57d16801dc

Configuration

/persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_96c4b1703a57d16801dc.local.yaml

Config SHA-256 (recorded, unverified)

92725958c8af665fb0f645e7fab2db9d7afde685db4d31e2903ea0fad3e02c1d

Actual config SHA-256

92725958c8af665fb0f645e7fab2db9d7afde685db4d31e2903ea0fad3e02c1d

Resolved config SHA-256 (recorded, unverified)

c8125e45785bd3b86bc96072904e2b3a6161fdc21439e1c645d9b60baf749276

Seed

42

titan-pretrain --config /persistent/.local/state/george-auto-research/run_configs/probe-dp4-candidate_96c4b1703a57d16801dc.local.yaml