This paper described a novel and bottom-up in situ polymerization method for preparing poly(vinyl alcohol) (PVA)/graphene oxide (GO) nanocomposites. The approach is conducted by mixing vinyl acetate, methanol, initiator and GO with the aid of high power ultrasound, followed by free radical polymerization and alcoholysis reaction, resulting in the in situ formation of PVA matrix and molecular level dispersion of GO. X-ray diffraction, transmission electron microscopy and scanning electron microscopy revealed that GO nanosheets were completely intercalated, uniformly dispersed and highly oriented along the surface of the nanocomposite films. Centrifugation experiments, Fourier-transform infrared spectroscopy and differential scanning calorimetry evidenced the existence of strong hydrogen bond interactions between the GO nanosheets and PVA matrix. Furthermore, the mechanical properties and water vapor barrier properties of PVA/GO nanocomposites had been strikingly enhanced by adding a small amount of GO (with GO loading as low as 0.01–0.08 wt%). Compared with the neat polymer, the tensile strength increased from 42.3 to 50.8 MPa and Young's modulus increases markedly from 1477 to 2123 MPa at the GO incorporation of only 0.04 wt%. Moreover, it gave rise to about 78% decline in the coefficient of moisture permeability.