Machine learning evolved as an attempt to make computers – imagined as perfect machines – understand the imperfect and noisy data produced in the real world. But large computer systems are far from perfect, and modern software systems produce data as imperfect as any real-world system. We discuss how we build tools that help understand, and prevent, failures in large software systems, using methodologies ranging from Bayesian modeling over Causal Analysis to Transformer-based neural networks.