Machine learning models should not live in a vacuum. When a predictive model is exposed to the user to enrich a scientific or industrial workflow, incorporating human feedback is essential for the model to improve over time, overcome distribution shifts, and learn novel phenomena. We address the engineering challenges of building interactive active learning systems on a practical example of large-scale video analysis.