The bipolar filtration of Cochran, Harvey and Horn initiated the study of deeper structures of the smooth concordance group of the topologically slice knots. We show that the graded quotient of the bipolar filtration has infinite rank at each stage greater than one. To detect nontrivial elements in the quotient, the proof uses higher order amenable Cheeger-Gromov $L^2$ $\rho$-invariants and infinitely many Heegaard Floer correction term $d$-invariants simultaneously. 
This is joint work with Jae Choon Cha.