Abstract: Aiming at the problems in image semantic segmentation such as detail information loss, fuzzy & rough edges of segmentation categories, a residual attention module is designed based on encoder-and-decoder combined with residual modules and attention mechanism. The attention mechanism strengthens the connectivity among feature-map channels to improve the fineness of semantic segmentation. For multi-scale object recognition, a joint pyramid up-sampling module is designed based on pyramid models. It uses different scale feature maps generated during encoding processes to extract semantic information and increases the recognition ability on model scenes. Finally, the proposed scheme is verified by experiments on the VOC2012 and Cityscape data sets. Comparing with FCN-8s、SegNet、Deeplab-v2、PSPNet, the highest mean Intersection over Union (mIoU) and mean Pixel Accuracy(mPA) increased by 15.9% and 3.57% for VOC 2012, 17.8% and 13.3% for Cityscape data set, respectively. The image semantic segmentation effect has been significantly improved.