Deep Learning Phenomena
Deep learning has transformed AI, but its inner workings remain complex and mysterious. Unlocking these mysteries is crucial for building safer, more effective AI systems. By studying phenomena—observable patterns revealed through controlled experiments—we gain insights into deep learning's complexity, much like scientists uncover natural laws through careful observation. This platform is a hub for researchers to explore, share, and discuss key deep learning phenomena. Through well-documented, reproducible experiments, we aim to spark curiosity, bridge theory and practice, and drive collective progress in the field. Join us to discover open questions and advance AI together.
- CondensationDuring the nonlinear training of neural networks, neurons in the same layer tend to condense into groups with similar outputs
Learn more
- Double Descent随着模型规模、训练时长或样本数量的增加,测试误差呈现先下降、后上升、再下降的两段式U型曲线
Learn more
- Frequency Principle/Spectral BiasDeep neural networks tend to learn from low to high frequency
Learn more
- GrokkingA model initially memorizes training data but, after prolonged optimization, suddenly transitions to learning a more generalizable underlying structure.
Learn more
- Scaling LawsTransformer performance follows predictable power-law relationships with model size, data scale, and training compute.
Learn more