Conventional joint operation of integrated electricity and heating systems faces severe challenges, including non-convex models and computation complexity. Additionally, there are adverse impacts from the uncertainties of renewable distributed generators, as well as electrical and thermal loads. This paper proposes an optimal joint operation method of integrated electricity and heating systems based on multi-agent deep reinforcement learning (DRL) method. Firstly, a new hydraulic-thermal flow algorithm that is compatible with DRL training environment is developed. Then, a stochastic distributed optimization model is formulated with multiple agents to minimize network power losses while avoiding operation constraint violations under the spatial and temporal uncertainties. Last, a multi-agent deep deterministic policy gradient is adopted combined with offline neural network training via exploration in a virtual environment and online optimization of joint operation. A numerical case study indicates the effectiveness of the proposed method and solution robustness against spatial and temporal uncertainties.