Hyperbranched polysilane (HBPSi) grafted reduced graphene oxide (rGO), marked as HBPSi-rGO, was synthesized by the reaction of hydrosilylation. Fourier-transform infrared spectroscopy (FTIR) and X-ray photoelectron spectroscopy (XPS) indicated that the surface of graphene had been successfully functionalized. Subsequently, HBPSi-rGO was incorporated into the bismaleimides (BMI) resin to prepare composites so as to improve its performance. The tribological, mechanical and thermal properties of the composites were studied systematically. Results show that the addition of the appropriate content of HBPSi-rGO can enhance the mechanical properties including the impact and flexural strengths of the BMI resin. In addition, the thermal stability of HBPSi-rGO/BMI nanocomposites is also superior to that of the pure BMI resin. It is worth noting that when the content of HBPSi-rGO is 0.6 wt%, the frictional coefficient and the wear rate is decreased by 44.6% and 77.4%, respectively compared to those of the neat BMI. Scanning electron microscopy (SEM) reveals that wear mechanism of neat BMI is mainly fatigue wear, but it turns mainly adhesive wear after the incorporation of HBPSi-rGO. The main reason can be that the interface adhesion was enhanced due to the reaction of nucleophilic addition between HBPSi-rGO and BMI matrix.