In recent years, convolutional neural networks have been successfully applied to single image super-resolution (SISR) tasks, making breakthrough progress both in accuracy and speed. In this work, an improved dual-scale residual network (IDSRN), achieving promising reconstruction performance without sacrificing too much calculations, is proposed for SISR. The proposed network extracts features through two independent parallel branches: dual-scale feature extraction branch and texture attention branch. The improved dual-scale residual block (IDSRB) combined with active weighted mapping strategy constitutes the dual-scale feature extraction branch, which aims to capture dual-scale features of the image. As regards the texture attention branch, an encoder–decoder network employing symmetric full convolutional-deconvolution structure acts as a feature selector to enhance the high-frequency details. The integration of two branches reaches the goal of capturing dual-scale features with high-frequency information. Comparative experiments and extensive studies indicate that the proposed IDSRN can catch up with the state-of-the-art approaches in terms of accuracy and efficiency.