To improve the flame retardancy of flame retardants with the assistance of the synergistic effect between an organic flame retardant (vinyltrimethoxysilane grafted by 9,10-dihydro-9-oxa-10-phosphaphenanthrene-10-oxide, DOPO-VTS) and inorganic flame retardant (zinc hydroxystannate nanoparticle, ZHS), a novel organic–inorganic hybrid flame retardant, ZHS nanoparticle coated by DOPO-VTS (DOPO-VTS–ZHS), was successfully prepared and well characterized. Subsequently, DOPO-VTS, ZHS and as-prepared DOPO-VTS–ZHS were separately incorporated into poly(vinyl chloride) (PVC) to form PVC-matrix nanocomposites, and the thermal stability, mechanical properties and flame retardancy of the as-prepared PVC-matrix nanocomposites were investigated. Thermogravimetric analysis results showed that the presence of ZHS, DOPO-VTS or DOPO-VTS–ZHS in the PVC matrix led to a slight thermal destabilization effect under air, which promoted the formation of char residue and increased the char yield at 600 °C. Limited oxygen index (LOI) tests revealed that they also imparted PVC nanocomposites with higher values of LOI over pure PVC. Besides, the cone calorimeter result revealed that the three kinds of flame retardants could obviously reduce the peak heat release rate (pHRR), the total heat release (THR) and smoke products of PVC. Among them, DOPO-VTS–ZHS had the most significant effect. Particularly, with the incorporation of 5 wt% of DOPO-VTS–ZHS, the pHRR of DOPO-VTS–ZHS/PVC was significantly reduced by 39%, and the THR was decreased by 50%, compared with those of neat PVC. The dramatically reduced fire hazards are generally attributed to the synergistic effect of DOPO-VTS and ZHS, containing good dispersion of DOPO-VTS–ZHS, catalytic char function of DOPO and ZHS. In addition, DOPO-VTS–ZHS nanoparticles are also able to significantly increase the tensile strength of the PVC matrix.