In recent years, with increasing amounts of renewable energy sources connecting to power networks, sub-/super-synchronous oscillations (SSOs) have occurred more frequently. Due to the time-variant nature of SSO magnitudes and frequencies, as well as the mutual interferences among SSO modes with close frequencies, the accurate parameter estimation of SSO has become a particularly challenging topic. To solve this issue, this paper proposes an improved spectrum analysis method by improving the window function and a spectrum correction method to achieve higher precision. First, by aiming at the sidelobe characteristics of the window function as evaluation criteria, a combined cosine function is optimized using a genetic algorithm (GA). Furthermore, the obtained window function is self-convolved to extend its excellent characteristics, which have better performance in reducing mutual interference from other SSO modes. Subsequently, a new form of interpolated all-phase fast Fourier transform (IpApFFT) using the optimized window function is proposed to estimate the parameters of SSO. This method allows for phase-unbiased estimation while maintaining algorithmic simplicity and expedience. The performance of the proposed method is demonstrated under various conditions, compared with other estimation methods. Simulation results validate the effectiveness and superiority of the proposed method.