Look at the graph again. CO2 (thick black line) is rising from the start of the graph.

320 ppm is used because that was the concentration in 1965 when scientists began reporting on this phenomenon more formally.