I am a Master’s student in the School of Computing at KAIST, advised by Prof. Noseong Park. My research broadly studies how architectural inductive biases and training dynamics shape internal representations and functional specialization in neural networks. My current interests include:
- Mechanistic interpretability of LLMs
- Recurrent neural dynamics for efficient sequence modeling and adaptive reasoning
- Feedback-driven neural adaptation
Building on my undergraduate background in neuroscience, I am also drawn to brain-inspired inductive biases as universal learning principles that can be shared by both brain and machine.
News
-
Nov 2026 Linz, Austria
Invited Talk Giving an invited talk, Quotient Space Convergence for Looped Reasoning, at the workshop Learning-Based Methods in Scientific Computing at RICAM (Nov 17). -
Jul 2026 – Jan 2027 Tübingen, Germany
Research Internship Joining the Computational Applied Mathematics & AI Lab (CAMAIL) at the Max Planck Institute for Intelligent Systems as a research intern, advised by T. Konstantin Rusch. -
Jul 2026 Seoul, South Korea
ICML Conference Attending ICML 2026 (Jul 6–11) to present two works, Q-Delta and STAR.
-
Under Review · 2027
Looped Reasoning Finishes Earlier Than You Think
Sumin Park, T. Konstantin Rusch
Gauge-equivariant quotient recurrent reasoner (QRR) that decouples rotational symmetry motion from reasoning-relevant quotient dynamics, enabling stable and faster convergence
Looped Reasoning Quotient Dyanmics LLMs
Looped reasoning with dynamic depth can be viewed as relaxation toward solution attractors in a learned dynamical landscape. In practice, however, latent trajectories often oscillate, cycle, or drift rather than cleanly converge to stable fixed points. How these non-stationary dynamics contribute to reasoning and constrain convergence remains unclear. We show that slowly decaying oscillatory modes can impose their convergence rate on coupled latent dynamics, creating a bottleneck. Yet damping these modes indiscriminately can disrupt the reasoning computation they support. This tension motivates an architectural approach: introducing rotational degrees of freedom along which symmetry-generated rotational motion leaves the reasoning-relevant quotient dynamics unchanged by construction. We propose QRR (Quotient Recurrent Reasoner), a gauge-equivariant recurrent architecture with an exact per-token phase-rotation symmetry. Motion along these symmetry directions leaves the quotient representation unchanged and is dynamically decoupled from reasoning-relevant dynamics. Empirically, QRR achieves stable and faster convergence to solution roots in quotient space, without forming a dominant slow oscillatory bottleneck, while matching or surpassing recent recurrent reasoning models on Sudoku, Maze, and Mini-ARC.
-
ICML · 2026
Q-Delta: Beyond Key–Value Associative State Evolution
Sumin Park, Seojin Kim, Noseong Park
Query-aware delta rule for linear attention that uses mixed key–query prediction errors, enabling richer, jointly corrective state evolution dynamics
Linear Attention SSMs LLMs
Linear attention reformulates sequence modeling as recurrent state evolution, enabling efficient linear-time inference. Under the key–value associative paradigm, existing approaches restrict the role of the query to the readout operation, decoupling it from state evolution. We show that query-conditioned state readout induces a structured value prediction over accumulated memory that complements key-based retrieval. Based on this insight, we propose Q-Delta, a query-aware delta rule that integrates mixed key–query prediction errors into state evolution, enabling jointly corrective dynamics while preserving delta-rule efficiency. We establish stability guarantees for the resulting dynamics and derive a hardware-efficient chunkwise-parallel formulation with a custom Triton implementation. Empirical results demonstrate stable optimization, competitive throughput, and consistent improvements over strong baselines on language modeling and long-context retrieval tasks. Code is available at https://github.com/psmiz/Q-Delta.
-
ICML · 2026
STAR: Rethinking MoE Routing as Structure-Aware Subspace Learning
Sumin Park, Noseong Park
Input-aware MoE routing based on incremental subspace learning for evolving input representation
MoE Representation LLMs
Mixture-of-Experts (MoE) scales model capacity efficiently by selectively routing inputs to a specialized subset of experts. However, input-expert specialization, the core motivation of MoE, critically depends on whether the router is actually aware of input structure. In practice, MoE routing is typically implemented as a shallow linear projection with limited awareness of input representation, which often leads to unstable routing. We propose STAR, a Structure-Aware Routing that rethinks MoE routing as a subspace learning problem by augmenting standard learnable routing with an evolving principal subspace that tracks dominant input structure via Generalized Hebbian Algorithm (GHA). By aligning routing decisions directly with input structure, STAR enables stable expert specialization. We evaluate STAR on controlled synthetic setup and large-scale language and vision tasks, where it consistently improves routing quality and downstream performance over strong MoE baselines. Moreover, optional test-time subspace updates further enhance routing robustness and generalization under input distribution shifts. Code is available at https://github.com/psmiz/STAR.
-
AAAI · 2026
How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts
Sumin Park, Noseong Park
Adaptive MoE expansion mechanism based on gradient-guided semantic drift signals
Adaptive MoE Expert Expansion
Finding the optimal configuration of Sparse Mixture-ofExperts (SMoE) that maximizes semantic differentiation among experts is essential for exploiting the full potential of MoE architectures. However, existing SMoE frameworks either heavily rely on hyperparameter tuning or overlook the importance of diversifying semantic roles across experts when adapting the expert pool size. We propose Mixture-of-Experts for Adaptive Semantic Specialization (MASS), a semanticaware MoE framework for adaptive expert expansion and dynamic routing. MASS introduces two key advancements: (i) a gradient-based semantic drift detector that prompts targeted expert expansion when the existing expert pool lacks capacity to capture the full semantic diversity of the data, and (ii) an integration of adaptive routing strategy that dynamically adjusts expert usage based on token-level routing confidence mass. We first demonstrate that MASS reliably converges to the point of optimal balance between cost-performance trade-off with notably improved semantic specialization in a highly controlled synthetic setup. Further empirical results on real-world datasets across language and vision domains show that MASS consistently outperforms a range of strong MoE baselines, demonstrating its domain robustness and enhanced expert specialization.
-
ICML · 2024
PANDA: Expanded Width-Aware Message Passing Beyond Rewiring
Jeongwhan Choi, Sumin Park, Hyowon Wi, Sung-Bae Cho, Noseong Park
Expanded width-aware message passing for GNNs to address the over-squashing problem
GNNs Message-Passing
Recent research in the field of graph neural network (GNN) has identified a critical issue known as "over-squashing," resulting from the bottleneck phenomenon in graph structures, which impedes the propagation of long-range information. Prior works have proposed a variety of graph rewiring concepts that aim at optimizing the spatial or spectral properties of graphs to promote the signal propagation. However, such approaches inevitably deteriorate the original graph topology, which may lead to a distortion of information flow. To address this, we introduce an expanded width-aware (PANDA) message passing, a new message passing paradigm where nodes with high centrality, a potential source of over-squashing, are selectively expanded in width to encapsulate the growing influx of signals from distant nodes. Experimental results show that our method outperforms existing rewiring methods, suggesting that selectively expanding the hidden state of nodes can be a compelling alternative to graph rewiring for addressing the over-squashing.