I will introduce an arithmetic version of the classical Rallis' inner product formula for unitary groups, which generalizes the previous works of Kudla, Kudla-Rapoport-Yang and Bruinier-Yang. As Rallis' formula concerns the central L-values of automorphic representations with certain epsilon factor 1. The arithmetic one concerns central L-derivatives of those with the epsilon factor -1, whose central L-values vanish automatically. This formula, which is still a conjecture for higher rank, relates the canonical height of special cycles on certain Shimura varieties and the central L-derivatives.