Related articles

expert replication

Creation of additional expert copies in MoE systems, which can compete with token-related buffers for limited GPU memory.

• LRMoE Introduces Demand-Side Load Re-Calibration for Efficient MoE Training on Memory-Constrained Nodes