Hybrid in-band on-channel (IBOC) broadcasting system as a scheme of digital audio radio could transmit analog FM and digital audio simultaneously. In IBOC systems, broadcasters transmit signals within the allocated channel bands, thus the bandwidth assigned for digital audio transmission is very challenging. BICM-ID can improve BER performance over Rayleigh fading channels without bandwidth expansion, thus it is used in FM IBOC and has been verified it can yield a better coding gain. However, conventional IBOC systems apply QPSK as channel modulation scheme, which can not take full advantage of constellation space compared with duobinary modulation. In this paper, various modulation schemes with the same encode rate and the same bandwidth efficiency are compared and analyzed. Simulation results show that advanced modulation scheme can effectively improve the performance of BICM-ID FM IBOC. When BER is below 10
, 32APSK modulation scheme can achieve more than 1dB coding gains over Rayleigh fading channels.