In this paper, β-cyclodextrin was successfully modified by maleic anhydride under certain experimental conditions. A novel water-soluble nanocomposite (AAMC-S1) consisting of nano-SiO₂ was synthesized by free-radical polymerization using acrylamide (AM), 2-acrylamido-2-methyl propane sulfonic acid (AMPS) and modified β-cyclodextrin (MAH-β-CD) as raw materials. The characterization of AAMC-S1 was measured by Fourier transform infrared spectroscopy (FT-IR), ¹H nuclear magnetic resonance spectroscopy (¹H NMR), thermal degradation (TGA) and scanning electron microscopy (SEM). In the performance evaluation experiments, the AAMC-S1 nanocomposite demonstrated desirable properties in terms of its thickening ability, temperature resistance, salt tolerance and shear resistance compared to those of the pure polymer AM/AMPS/MAH-β-CD (denoted AAMC-S0). Desirable properties could be attributed to the introduction of inorganic nano-SiO₂, which endowed the AAMC-S1 structure with rigidity and made the network structure denser and stronger. Based on the indoor displacement test, the AAMC-S1 nanocomposite gave a resistance factor (RF) of 22.31 and a residual resistance factor (RFF) of 5.58, compared with 17.71 and 2.73 for AAMC-S0. The enhanced oil recovery test showed that AAMC-S1 could remarkably enhance 13.25% of oil recovery while AAMC-S0 could enhance 7.12% of oil recovery. These behaviors made this novel nanocomposite promising in enhancing oil recovery.