DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles

Peter Karkus, Boris Ivanovic, Shie Mannor, Marco Pavone

Research output: Contribution to journalConference articlepeer-review

7 Scopus citations


Autonomous vehicle (AV) stacks are typically built in a modular fashion, with explicit components performing detection, tracking, prediction, planning, control, etc. While modularity improves reusability, interpretability, and generalizability, it also suffers from compounding errors, information bottlenecks, and integration challenges. To overcome these challenges, a prominent approach is to convert the AV stack into an end-to-end neural network and train it with data. While such approaches have achieved impressive results, they typically lack interpretability and reusability, and they eschew principled analytical components, such as planning and control, in favor of deep neural networks. To enable the joint optimization of AV stacks while retaining modularity, we present DiffStack, a differentiable and modular stack for prediction, planning, and control. Crucially, our model-based planning and control algorithms leverage recent advancements in differentiable optimization to produce gradients, enabling optimization of upstream components, such as prediction, via backpropagation through planning and control. Our results on the nuScenes dataset indicate that end-to-end training with DiffStack yields substantial improvements in open-loop and closed-loop planning metrics by, e.g., learning to make fewer prediction errors that would affect planning. Beyond these immediate benefits, DiffStack opens up new opportunities for fully data-driven yet modular and interpretable AV architectures. Project website: https://sites.google.com/view/diffstack.

Original languageEnglish
Pages (from-to)2170-2180
Number of pages11
JournalProceedings of Machine Learning Research
StatePublished - 2023
Event6th Conference on Robot Learning, CoRL 2022 - Auckland, New Zealand
Duration: 14 Dec 202218 Dec 2022


  • Autonomous Driving
  • Control
  • Differentiable Algorithms
  • Planning

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability


Dive into the research topics of 'DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles'. Together they form a unique fingerprint.

Cite this