The hybrid cascaded high-voltage direct current (HVDC) transmission system has various operation modes, and some operation modes are having sharply increasing requirements for protection rapidity, while the traditional pilot differential protection (PDP) has poor rapidity, and even refuses to operate when faults occur on the DC line. Therefore, a novel pilot protection scheme based on traveling wave characteristics is proposed. First, the adaptability of the traditional PDP applied in engineering is analyzed for different operation modes. Then, the expressions of the forward traveling wave (FTW) and backward traveling wave (BTW) on the rectifier side and the inverter side are derived for different fault locations. From the theoretical derivation, the difference between the BTW and FTW on the rectifier side is less than zero, and the same is true on the inverter side. However, in the event of an external fault of DC line, the difference between the BTW and FTW at near-fault terminal protection installation point is greater than zero. Therefore, by summing over the product of the difference between BTW and FTW of the rectifier side and that of the inverter side, the fault identification criterion is constructed. The simulation results show that the proposed pilot protection scheme can quickly and reliably identify the short-circuit faults of DC line in different operation modes.