The optomechanical instability in the quantum regime

We consider a generic optomechanical system, consisting of a driven optical cavity and a movable mirror attached to a cantilever. Systems of this kind (and analogues) have been realized in many recent experiments. It is well known that these systems can exhibit an instability towards a regime where the cantilever settles into self-sustained oscillations. In this paper, we briefly review the classical theory of the optomechanical instability, and then discuss the features arising in the quantum regime. We solve numerically a full quantum master equation for the coupled system, and use it to analyze the photon number, the cantilever's mechanical energy, the phonon probability distribution and the mechanical Wigner density, as a function of experimentally accessible control parameters. When a suitable dimensionless 'quantum parameter' is sent to zero, the results of the quantum mechanical model converge towards the classical predictions. We discuss this quantum-to-classical transition in some detail.