Beidi Chen Home Page

Beidi Chen Home Page
Beidi
Chen
Email ·
beidic@andrew.cmu.edu
Check
InfiniAI Lab @ CMU
for latest update.
I am starting as an Assistant Professor of Electrical and Computer Engineering at Carnegie Mellon University in Fall 2023.
My lab is recruiting for 23' (
Application Link
)! I am looking for students and interns who are excited to tackle efficiency problems in ML together from an algorithm, modeling, or system/hardware perspective.
Ph.D., master's, undergraduate, and visiting students are welcome to reach out!
I am currently a visiting researcher at Meta/Facebook AI Research (
FAIR
). Previously, I was a postdoc researcher at Stanford working with Dr.
Chris Ré
. I received my Ph.D. in Computer Science from Rice University under the supervision of Dr.
Anshumali Shrivastava
in 2020. I received my B.S. from University of California, Berkeley in 2015. My mentors were Dr.
Sara Alspaugh
, Dr.
Kaifei Chen
and my advisor was Dr.
Randy Katz
. My research focuses on large-scale machine learning. Specifically, I design and optimize randomized algorithms (algorithm-hardware co-design) to accelerate large machine learning systems for real-world problems.
NEWS:
We will present three papers:
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer
, and
Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions
in NeurIPS 2023.
We will present
Fast Algorithms for a New Relaxation of Optimal Transport
in COLT 2023.
We will present three papers:
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
Oral
),
FlexGen: High-throughput Generative Inference of Large Language Models with a Single GPU
Oral
), and
CocktailSGD: Fine-tuning Foundation Models over 500Mbps Networks
in ICML 2023.
We will present two papers:
Decentralized Training of Foundation Models in Heterogeneous Environments (
Oral
and
Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees
in NeurIPS 2022.
Our paper:
Monarch: Expressive Structured Matrices for Efficient and Accurate Training
got
Outstanding Paper Runner Up
in ICML 2022.
Our recent work
Monarch
got
Outstanding Poster
Award at the
EfficientML Bay Area meetup
Pixelated Butterfly
is accepted by ICLR 2022 (
Spotlight
)!
Research
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang,
Beidi Chen
. "
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
". NeurIPS 2023.
Yuandong Tian, Yiping Wang,
Beidi Chen
, Simon Shaolei Du. "
Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformer
". NeurIPS 2023.
Stefano Massaroli, Michael Poli, Daniel Y Fu, Hermann Kumbong, David W. Romero, Rom Nishijima Parnichkun, Aman Timalsina, Quinn McIntyre,
Beidi Chen
, Atri Rudra, Ce Zhang, Christopher Ré, Stefano Ermon, Yoshua Bengio. "
Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions
". NeurIPS 2023.
Moses Charikar,
Beidi Chen
, Christopher Ré, Erik Waingarten. "
Fast Algorithms for a New Relaxation of Optimal Transport
". Conference on Learning Theory (COLT) 2023.
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré,
Beidi Chen
. "
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
". ICML 2023. (
Oral
).
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Daniel Y. Fu, Zhiqiang Xie,
Beidi Chen
, Clark Barrett, Joseph E. Gonzalez, Percy Liang, Christopher Ré, Ion Stoica, Ce Zhang. "
High-throughput Generative Inference of Large Language Models with a Single GPU
". ICML 2023. (
Oral
). [
code
Jue Wang, Yucheng Lu, Binhang Yuan,
Beidi Chen
, Percy Liang, Christopher De Sa, Christopher Ré, Ce Zhang. "
CocktailSGD: Fine-tuning Foundation Models over 500Mbps Networks
". ICML 2023.
Binhang Yuan, Yongjun He, Jared Quincy Davis, Tianyi Zhang, Tri Dao,
Beidi Chen
, Percy Liang, Christopher Ré, Ce Zhang. "
Decentralized Training of Foundation Models in Heterogeneous Environments
". NeurIPS 2022. (
Oral
).
Jue Wang, Binhang Yuan, Luka Rimanic, Yongjun He, Tri Dao,
Beidi Chen
, Christopher Ré, Ce Zhang. "
Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees
". NeurIPS 2022.
Tri Dao,
Beidi Chen
, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, Christopher Ré. "
Monarch: Expressive Structured Matrices for Efficient and Accurate Training
". ICML 2022. (
Long Talk, Outstanding Paper Runner Up
). [
code
][
poster
Beidi Chen
*, Tri Dao*, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, Christopher Ré. "
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
". In Proceedings of International Conference on Learning Representations, ICLR 2022 (
Spotlight
). [
video
][
code
Zichang Liu, Zhaozhuo Xu, Alan Ji, Junyan Zhang, Jonathan Li,
Beidi Chen
, Anshumali Shrivastava. "
HALOS: Hashing Large Output Space for Cheap Inference
". In Proceedings of the 5rd Conference on Machine Learning and Systems, MLSys 2022.
Beidi Chen
*, Tri Dao*, Eric Winsor, Zhao Song, Atri Rudra, Christopher Ré. "
Scatterbrain: Unifying Sparse and Low-rank Attention Approximation
". In Neural Information Processing Systems, NeurIPS 2021. [
code
Beidi Chen
, Zichang Liu, Binghui Peng, Zhaozhuo Xu, Jonathan Lingjie Li, Tri Dao, Zhao Song, Anshumali Shrivastava, Christopher Re. "
MONGOOSE: A Learnable LSH Framework for Efficient Neural Network Training
". In Proceedings of International Conference on Learning Representations, ICLR 2021. (
Oral
) [
video
][
code
Zhaozhuo Xu,
Beidi Chen
, Chaojian Li, Weiyang Liu, Le Song, Yingyan Lin, and Anshumali Shrivastava. "
Locality Sensitive Teaching
". In Neural Information Processing Systems, NeurIPS 2021.
Shabnam Daghaghi, Nicholas Meisburger, Mengnan Zhao,
Beidi Chen
, Tharun Medini, and Anshumali Shrivastava. "
A Tale of Two Efficient and Informative Negative Sampling Distributions
". In Proceedings of International Conference on Machine Learning, ICML 2021. (
Long Talk
Tharun Medini,
Beidi Chen
, Anshumali Shrivastava. "
SOLAR: Sparse Orthogonal Learned and Random Embeddings
". In Proceedings of International Conference on Learning Representations, ICLR 2021.
Beidi Chen
, Weiyang Liu, Animesh Garg, Zhiding Yu, Anshumali Shrivastava, Jan Kautz, Anima Anandkumar. "
Angular Visual Hardness
". In Proceedings of the 37th International Conference on Machine Learning, ICML 2020. [
Video
Beidi Chen
, Tharun Medini, James Farwell, Sameh Gobriel, Charlie Tai, Anshumali Shrivastava. "
SLIDE : In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems
". In Proceedings of the 3rd Conference on Machine Learning and Systems, MLSys 2020.
Beidi Chen
, Yingchen Xu, and Anshumali Shrivastava. "
LGD: Fast and Accurate Stochastic Gradient Estimation
". In Neural Information Processing Systems, NeurIPS 2019. (LSH-Sampling Breaks the Computational Chicken-and-Egg Loop in Adaptive Stochastic Gradient Estimation in ICLR 2018 Workshop)
Beidi Chen
, M. Sadegh Riazi, Anshumali Shrivastava, DanWallach, Farinaz Koushanfar. "
Sub-linear Privacy-preserving Search with Untrusted Server and Semi-honest Parties
". Manuscript.
Beidi Chen
, Anshumali Shrivastava. "
Revisiting Winner Take All (WTA) Hashing for Sparse Datasets
". In Proceedings of the 34th Conference in Uncertainty in Artificial Intelligence, UAI 2018.
Beidi Chen
, Anshumali Shrivastava, Rebecca C. Steorts. "
Unique Entity Estimation with Application to the Syrian Conflict
". The Annals of Applied Statistics 12.2 (2018). (Also Won IISA 2018
Best Student Paper
in Applied Statistics with this paper.)
Kaifei Chen, Siyuan He,
Beidi Chen
, John Kolb, Randy H. Katz, David E. Culler. "
BearLoc: A Composable Distributed Framework for Indoor Localization Systems
". In Proceedings of the 2015 Workshop on IoT challenges in Mobile and Industrial Systems, IoT-Sys@MobiSys 2015, pages 7-12, May. 2015. Florence, Italy.
S. Alspaugh,
Beidi Chen
, Jessica Lin, Archana Ganapathi, Marti Hearst, and Randy Katz. "
Analyzing Log Analysis: An Empirical Study of User Log Mining
". In Proceedings of the 28th Large Installation System Administration Conference, LISA 2014. (
Best Student Paper
Experience
Research Intern
Microsoft Research, Redmond, WA
May 2019 - Present
Machine Learning Scientist Intern
Nvidia, Santa Clara, CA
Feb 2019 - May 2019
Applied Scientist Intern
A9.com, Palo Alto, CA
May 2018 - Dec 2018
Applied Scientist Intern
Amazon Web Services, Inc., Palo Alto, CA
March 2017 - June 2017
Software Engineering Intern
Apple Inc., Cupertino, CA
Feb 2015 - Aug 2015
Research and Development Intern
VMware Inc., Palo Alto, CA
May 2014 - Aug 2014
Software Engineering Intern
Broadcom Corporation, Sunnyvale, CA
May 2013 - Aug 2013
Education
Stanford University
Postdoc
Computer Science
August 2020 - Present
Rice University
Doctor of Philosophy
Computer Science
August 2015 - May 2020
University of California, Berkeley
Bachelor of Science
Electrical Engineering and Computer Science
August 2011 - May 2015
Wuhan Foreign Languages School
High School
Sep 2008 - June 2011
Honors and Awards
ICML 2022 Outstanding Paper Runner Up
MIT EECS Rising Stars 2021
UIUC EECS Rising Stars 2019
IISA 2018 Best Student Paper Award
Ken Kennedy Institute for Information Technology 2017 Fellowship
USENIX LISA 2014 Best Student Paper Award
ANITA BORG INSTITUTE GHC Scholarship 2014
Qualcomm Undergraduate Experiences in Science & Technology Scholar 2013
Invited Talks
LightOn AI Meetup
[webpage]
VMware ML
TWIML AI Podcast
[video]
Microsoft Machine Translation Group
Microsoft Rearch Talks
[video]
Record Linkage workshop at CIMAT
[webpage]
IISA2018 Conference
JSM 2018 Topic-Contributed Session
Activity and Service
Teaching
TA in Comp530 (Database System Implementation)
Rice University, Spring 2018
TA in Comp382 (Reasoning about Algorithms)
Rice University, Fall 2017, 2019
TA in Comp330 (Tools and Models For Data Science)
Rice University, Fall 2016, 2018
TA in EE122 (Introduction to Communication Networks)
UC Berkeley, Spring 2013, 2014
TA in EE40 (Introduction to Microelectronic Circuits)
UC Berkeley, Fall 2014
Service
Reviewer for Science Advances, NeurIPS, ICML, ICLR, AISTATS, AAAI, UAI, AISTATS
Activity
Stanford CS Undergraduate Mentoring Program
Women in Science and Engineering at Stanford
Member of CSters at Rice University
Member of Association of Women in EECS (AWE) at UC Berkeley