The energy consumption of buildings has risen steadily in recent years. It is vital for the managers and owners of the building to manage the electric energy demand of the buildings. Forecasting electric energy consumption of the buildings will bring great profits, which is influenced by many factors that make it very difficult to provide an advanced forecasting. Recently, deep learning techniques are widely adopted to solve this problem. Deep neural network offers an excellent capability in handling complex non-linear relationships and competence in exploring regular patterns and uncertainties of consumption behaviors at the building level. In this paper, we propose a deep convolutional neural network based on ResNet for hour-ahead building load forecasting. In addition, we design a branch that integrates the temperature per hour into the forecasting branch. To enhance the learning capability of the model, an innovative feature fusion is presented. At last, sufficient ablation studies are conducted on the point forecasting, probabilistic forecasting, fusion method, and computation efficiency. The results show that the proposed model has the state-of-the-art performance, which reflects a promising prospect in application of the electricity market.