Endolysosomal compartments maintain cellular fitness by clearing from cells dysfunctional organelles and proteins. Modulation of their activity offers therapeutic opportunities. Quantification of cargo delivery to and/or accumulation within endolysosomes is instrumental to characterize lysosome-driven pathways at the molecular level and to monitor consequences of genetic or environmental modifications. Here we introduce LysoQuant, a deep learning approach for segmentation and classification of fluorescence images capturing cargo delivery within endolysosomes for clearance. LysoQuant is trained for unbiased and rapid recognition with human-level accuracy and the pipeline informs on a series of quantitative parameters such as endolysosome number, size, shape, position within cells and occupancy, which report on activity of lysosome-driven pathways. In our selected examples, LysoQuant successfully determines the magnitude of mechanistically distinct catabolic pathways that ensure lysosomal clearance of a model organelle, the endoplasmic reticulum (ER), and of a model protein, polymerogenic ATZ. It does so with accuracy and velocity compatible with high throughput analyses.