Microstructure evolution during the reverse transformation of low alloy steel consisting of lath martensite and chromium carbide was examined using in-situ electron backscatter diffraction at high temperatures. Austenite grains nucleated during the reverse transformation were categorized into two types: austenite grains nucleated along the lath boundaries with almost the same crystal orientation as the prior austenite (type A), and austenite grains nucleated at the prior austenite grain boundaries or inside the prior austenite grains with a different crystal orientation (type B). After the reverse transformation was finished, the prior austenite grains were reconstructed by the rapid growth and coalescence of type-A grains, which was called the “austenite memory phenomenon.” Here, misorientation remained in the reconstructed austenite grains. Hence, upon heating to a higher temperature, type-A grains were invaded and were eventually replaced by type-B grains, resulting in a new fine-grained microstructure; this was similar to recrystallization. Therefore, these results showed that the austenite memory phenomenon occurred when the nucleation and growth of type-A grains was more dominant than those of type-B grains and that the degree of grain refinement depended on the nucleation and growth rate of the type-B grains.