Orthogonal Junk: Gradient-Orthogonality Data Selection for Continual Pre-Training on Low-Quality Data

Abstract

Continual pre-training on low-quality social media content causes ``brain rot''---measurable degradation in LLM capabilities that persists after instruction tuning. We investigate whether gradient-based data selection can mitigate this degradation by selecting training samples whose gradients are orthogonal to capability-preserving anchors. We propose Orthogonal Junk, a three-stage pipeline that computes anchor gradients from diverse benchmarks, scores candidates by gradient orthogonality, and uses pool-weighted sampling to balance selection quality with data diversity. Experiments on Llama-3.2-1B-Instruct reveal mixed results: Orthogonal Junk provides modest improvement on long-context understanding (RULER: +2.89pp vs random selection) but unexpectedly degrades reasoning (ARC-Challenge: $-$5.12pp). A simple perplexity baseline outperforms our method on RULER (+8.05pp). Analysis reveals that data repetition---an artifact of subset selection---is a dominant confound, with repetition rate affecting reasoning more than selection quality. These findings suggest that gradient orthogonality at the LM head may not capture the full dynamics of capability preservation.