Computer vision capabilities have rapidly been advancing and are becoming an important component to incident and disaster response. However, most commercial classifiers are insufficient to support disaster response at scale while potential end-users have a high regret for errors or misleading classifications. In response, the Low Altitude Disaster Imagery (LADI) dataset was developed. It is one of few aerial imagery datasets that includes disaster-specific labels (e.g. flooding, damage, etc.) and includes millions of labeled images and key video frames with various disasters. LADI is freely and publicly available; it is currently hosted on AWS. Using LADI, we’ve prototyped a workflow to classify images during a disaster. The workflow consists of an ensemble of image classifiers to be used in the pipeline, and algorithms to automatically determine whether human review is required.