The occurrence of low-frequency electromechanical oscillations is a major problem in the effective operation of power systems. The scrutiny of these oscillations provides substantial information about power system stability and security. In this paper, a new method is introduced based on a combination of synchrosqueezed wavelet transform and the stochastic subspace identification (SSI) algorithm to investigate the low-frequency electromechanical oscillations of large-scale power systems. Then, the estimated modes of the power system are used for the design of the power system stabilizer and the flexible alternating current transmission system (FACTS) device. In this optimization problem, the control parameters are set using a hybrid approach composed of the Prony and residual methods and the modified fruit fly optimization algorithm. The proposed mode estimation method and the controller design are simulated in MATLAB using two test case systems, namely IEEE 2-area 4-generator and New England-New York 68-bus 16-generator systems. The simulation results demonstrate the high performance of the proposed method in estimation of local and inter-area modes, and indicate the improvements in oscillation damping and power system stability.