In order to reduce the risk of commutation failure (CF) in the AC/DC hybrid power system, the quantitative analysis on CF is required for on-line assessment and optimal control. This paper presents an accurate and reliable method to quantify the commutation security based on the trajectory due to the complexity of the high-voltage direct current (HVDC) model. Firstly, the characteristics of the extinction angle trajectory are analyzed under both commutation success and failure conditions. The commutation security margin index (CSMI) is then proposed for the HVDC systems. Moreover, a search strategy for parameter limits is put forward based on the sensitivity analysis of CSMI to accelerate the search speed with a guaranteed accuracy level. A modified IEEE 39-bus power system and an actual large-scale power system with 46 generators and 821 buses are utilized to verify the validity and robustness of the proposed index and strategy.