Power system optimal dispatch with transient security constraints is commonly represented as transient security-constrained optimal power flow (TSC-OPF). Deep reinforcement learning (DRL)-based TSC-OPF trains efficient decision-making agents that are adaptable to various scenarios and provide solution results quickly. However, due to the high dimensionality of the state space and action spaces, as well as the non-smoothness of dynamic constraints, existing DRL-based TSC-OPF solution methods face a significant challenge of the sparse reward problem. To address this issue, a fast-converging DRL method for optimal dispatch of large-scale power systems under transient security constraints is proposed in this paper. The Markov decision process (MDP) modeling of TSC-OPF is improved by reducing the observation space and smoothing the reward design, thus facilitating agent training. An improved deep deterministic policy gradient algorithm with curriculum learning, parallel exploration, and ensemble decision-making (DDPG-CL-PE-ED) is introduced to drastically enhance the efficiency of agent training and the accuracy of decision-making. The effectiveness, efficiency, and accuracy of the proposed method are demonstrated through experiments in the IEEE 39-bus system and a practical 710-bus regional power grid. The source code of the proposed method is made public on GitHub.