Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Models

arXiv:2509.14255v2 Announce Type: replace-cross
Abstract: Mixture-of-Experts (MoE) models improve efficiency through sparse activation, but their learned gating functions provide limited insight into routing decisions. This work introduces the Semantic Resonance Architecture (SRA), which routes tokens to experts via cosine similarity between token representations and learnable semantic anchors, making every routing decision directly traceable to anchor-token similarity scores. We evaluate SRA on WikiText-103 across 17 configurations. In a controlled multi-seed comparison (3 seeds x 4 configurations, 256 experts, $D_ff=256$), cosine routing achieves competitive perplexity with standard linear routing ($12.57 pm 0.03$ vs $12.45 pm 0.03$ for $K=1 to 4$; $12.52 pm 0.02$ vs $12.57 pm 0.02$ for $K=2 to 4$). The training recipe — not the routing function — drives specialization quality, while cosine routing provides inherent inspectability. We introduce a bandpass routing loss — a floor-and-ceiling corridor on expert utilization — that reduces dead experts from 30-45% to 0-6% and transfers to both routing types. Routing-space evaluation shows cosine routing provides significantly better word-level subtoken coherence in deeper layers ($p < 0.001$), with 44-54% of expert specialization being syntactic rather than semantic. Extended analysis reveals cosine routing maintains more stable router saturation and tighter per-expert vocabulary distributions — structural advantages from the bounded cosine similarity range. An inference-time $k$-sweep shows that $k=5$ yields a free 0.08-0.16 perplexity gain over $k=4$. Cross-dataset validation on OpenWebText confirms generalization: cosine routing achieves comparable perplexity (44.88 vs 45.44), the bandpass loss eliminates dead experts, and specialization patterns are preserved.