Yaoqing Yang

*Please note: this CyLab seminar is open only to partners and Carnegie Mellon University faculty, students, and staff.

Speaker:
Yaoqing Yang
Assistant Professor
Department of Computer Science - Dartmouth College

Talk Title:
Weight Matrix Diagnostics for Improved Neural Network Training and Compression

Abstract:
This talk will introduce several useful metrics derived from studying the heavy-tail phenomenon in neural network weight matrices. I will begin by motivating these metrics through random matrix theory and discussing their connection to recent studies on feature learning. I will then demonstrate how these metrics can be applied to various neural network applications, including layer-wise pruning of large language models, tuning training and fine-tuning learning rates, training scientific machine learning models with limited data, adjusting the architectural hyperparameters of LoRA networks, and model selection on Hugging Face Transformers without access to training or testing data. In a recent study, we show that these theory-driven metrics can be scaled to prune large language models with up to 65 billion parameters, outperforming some of the latest pruning methods.

Bio:
My current research focuses on diagnosing and mitigating failures in machine learning models. For example, I analyze shape and geometric features in high-dimensional spaces, such as loss landscapes, weight matrix spectral densities, and decision boundaries, to provide actionable insights for addressing common failure modes in these models. I also apply these techniques to applications such as 3D point clouds and graphs. My research draws inspiration from statistical learning and information theory.

Upcoming Events