Abstract: The method of combining multi-scale segmentation and deep learning is used to extract the seismic damage information of buildings from three-dimensional images of oblique photography after an earthquake. In this way, the comprehensive damage information of the roof and wall of the building is obtained. Taking the 2017 M_S7.0 Jiuzhaigou earthquake as an example, firstly, based on the three-dimensional image of the roof and wall of the building, the multi-scale segmentation of samples are divided into three categories: intact building surface, damaged building surface, other ground objects and background. Secondly, 211 sample set with 100×100 pixel are selected to train the neural convolutional network model. The trained model is used to extract the seismic damage information of buildings in Qianguqing scenic spot and Zhangzha primary school. Finally, the accuracy of the extracted results is compared with the visual interpretation results. The results show that the extraction accuracy of damaged building surface is 65.5% and 71.1% respectively, the overall classification accuracy is 82.1% and 84.1% respectively, and the Kappa coefficient is 68.7% and 64.9% respectively. The result indicates that this method has certain advantages in seismic damage extraction of buildings from three-dimensional images of oblique photography.