Large-scale voltage collapse incidences, which result in power outages over large regions and extensive economic losses, are presently common occurrences worldwide. To avoid voltage collapse and operate more safely and reliably, it is necessary to analyze the voltage security operation region (VSOR) of power systems, which has become a topic of increasing interest lately. In this paper, a novel improved particle swarm optimization and recursive least square (IPSO-RLS) hybrid algorithm is proposed to determine the VSOR of a power system. Also, stability analysis on the proposed algorithm is carried out by analyzing the errors and convergence accuracy of the obtained results. Firstly, the voltage stability and VSOR-surface of a power system are analyzed in this paper. Secondly, the two algorithms, namely IPSO and RLS algorithms, are studied individually. Based on this understanding, a novel IPSO-RLS hybrid algorithm is proposed to optimize the active and reactive power, and the voltage allowed to identify the VSOR-surface accurately. Finally, the proposed algorithm is validated by using a simulation case study on three wind farm regions of actual Hami Power Grid of China in DIgSILENT/PowerFactory software. The error and accuracy of the obtained simulation results are analyzed and compared with those of the particle swarm optimization (PSO), IPSO and IPSO-RLS hybrid algorithms.