High efficiency video coding (HEVC) introduces a well-designed quad-tree structure to improve coding efficiency. During quad-tree building, the encoder tries all possible coding unit (CU) partition schemes to determine the optimal one with the minimum rate distortion cost, which introduces high computation complexity. To deal with this problem, a fast algorithm based on edge information is proposed to accelerate CU splitting in this paper. Firstly, the depth and prediction mode of adjacent blocks are obtained and used to facilitate rough partition. Secondly, the edge pixel values of this CU are counted and analyzed to fine-tune the partition. Experimental results show that the proposed algorithm can reduce coding time by 35.7% on average, with an acceptable loss of Bjøntegaard delta rate of 0.6%.