A new physical explanation of the fringes in the high-order harmonic spectra generated by ellipticity-modulated infrared pulses is proposed, which is different from those in previous works. In this work we find that the fringes are due to the spectral interference effect induced by the two outgoing pulses with a delay, which are generated by the wave plates for ellipticity control. The visibility of interference fringes observed experimentally increases with the increase of the asymmetry of the