Theory-Trained Deep Neural Networks: Insights, Techniques, and Future Directions

Deep neural networks have achieved remarkable success across numerous domains, yet a comprehensive theoretical understanding of their training dynamics and generalization remains a fundamental challenge. Theory-Trained Deep Neural Networks (TT-DNNs) represent an emerging paradigm that integrates rigorous theoretical insights into the design and training of neural architectures. This survey provides a systematic overview of TT-DNNs, categorizing key approaches based on optimization theory, statistical learning theory, approximation theory, and information theory. We discuss theory-informed training paradigms that improve convergence, robustness, and interpretability, and highlight notable applications across computer vision, natural language processing, scientific computing, and healthcare. Furthermore, we identify open challenges and future directions to bridge the gap between theory and practice. Our aim is to offer a comprehensive resource that fosters deeper understanding and innovation at the intersection of theory and deep learning practice.