Hidden Markov model (HMM)-based synthesized voices are intelligible but not natural especially under limited-data conditions due to over-smoothed speech spectra. Improving naturalness is a critical problem of HMM-based speech synthesis. One solution is to use voice conversion techniques to convert over-smoothed spectra to natural spectra. Although conventional conversion methods transform speech spectra to natural ones to improve naturalness, they cause unexpected distortions in the intelligibility of synthesized speech. The aim of the study is to improve naturalness without reducing the intelligibility of synthesized speech by employing our novel asymmetric bilinear model (ABM) to separate the intelligibility and naturalness of synthesized speech. In the study, our ABM was implemented on the modulation spectrum domain of Mel-cepstral coefficient (MCC) sequences to enhance the fine structure of spectral parameter trajectory generated from HMMs. Subjective evaluations carried out on English data confirmed that the achieved naturalness of the method using the ABM involving singular value decomposition (SVD) was competitive with other methods under large-data conditions and outperformed other methods under limited-data conditions. Moreover, modified rhyme test (MRT) showed that the intelligibility of synthesized speech was well preserved with our method.