Building detection from remote sensing images has significant importance in updating the maps, urban monitoring, and a wide range of other applications. The high spatial resolution images are an important data source for geospatial information extraction. These images provide extraordinary facilities for feature extraction like buildings and spatial analysis in urban areas. However this task suffers from some problems due to spectral complexity in the image scene. Since high resolution images contain a lot of details about the scene such as; non-homogeneity of the roof of the buildings, sloping and flatness, it can create various spectral properties among other issues. Also, due to the use of similar materials, some buildings can’t be completely separated from the streets and parking. To overcome these issues, the use of neighborhood and height information is essential. Accordingly, the major part of this research is the use of spatial features of adjacent pixels in a multispectral image and elevation data to increase the accuracy of building classification. In this regard, on the one hand, with the expansion of the feature space in a quasi-deep method, the goal was to train the classification algorithm in higher level and more comprehensive information. But all the features available to distinguish between buildings and non-buildings are not useful. On the other hand, due to the large amount of input data and the increased computing time and memory required, to reduce the processing cost, it is necessary to perform the feature selection operation. Despite many efforts that have been made over the past decades to develop the methods for automatic building detection from these images, high-performance methods are still unavailable because of the uncertainties like optimal feature selection. Therefore, in this study, with a view to improving the automatic detection of the building from remote sensing data, a new hybrid approach is proposed to select the optimal features of the large dataset in a reasonable time. The proposed method of this research firstly extracts high-level features for optimal building detection by using quasi-deep texture structures. Then, it selects the optimal features based on the integration of the developed AdaBoost algorithm (Confidence Based AdaBoost) with the optimized support vector machines by particle swarm optimization (CB-SVMpso), and performs the binary classification of the building and background. The experiments were performed on the standard data set of Vaihingen in Germany and then the results of the proposed method were compared with efficient methods of machine learning. Also, a comparison was made between the quasi-deep feature sets with the traditional method of GLCM textures. In experiments, in order to purify the final results, no pre-processing and post-processing steps have been interfered. The experimental results showed that on average, the highest overall accuracy and kappa coefficient obtained from the proposed method were 93.25% and 83.06%, respectively, and in comparison to conventional methods, the accuracy of kappa coefficient has increased by 7.27%, as well as the computational time reduction by half, indicating the reliability and efficiency of the proposed method in detecting the majority of buildings.