Over the past 20 years, we have developed arrays of custom-fabricated silicon and InP Geiger-mode avalanche photodiode arrays, CMOS readout circuits to digitally count or time stamp single-photon detection events, and techniques to integrate these two components to make back-illuminated solid-state image sensors for lidar, optical communications, and passive imaging. Starting with 4 × 4 arrays, we have recently demonstrated 256 × 256 arrays, and are working to scale to megapixel-class imagers. In this paper, we review this progress and discuss key technical challenges to scaling to large format.