Stretch-flange-formability is an important property for ultra high strength steel (UHSS) sheets for press-forming. In this study, microscopic deformation behaviors during punching and following stretch-flange-forming were investigated using three types of 980 MPa grade UHSS sheets with either two ferrite/martensite dual-phase structures or a martensite single-phase structure in order to clarify how the microstructure affects the stretch-flange-formability of UHSS sheets. The results of this investigation revealed following conclusions. Microscopic plastic-flow or micro-void density generated by punching is not the dominant factor of the stretch-flange-formability of UHSS sheets. During hole-expanding, cracks were mainly initiated at the fractured surface part and the cracks became longer and deeper from the punched surface with the increase of hole-expanding ratio. Deep cracking resistance in this process is important to improve the stretch-flange-formability. The existence of strain gradient induced by hole punching is considered to be one of the reason for the highest hole-expanding ratio of the martensite single phase steel. During hole-expanding, the micro-cracks propagate mostly along the phase interfaces in the dual-phase steel sheets in the case of poor stretch-flange-formability, while the micro-cracks are tend to propagate through ferrite and martensite phases in the case of high stretch-flange-formability. The analysis of the hardness of ferrite and martensite suggests that the difference in hardness is the dominant factor of the stretch-flange-formability of the dual-phase steel. In addition, the volume fractions of phases also influence the formability.