Yes, the kidney will retain bicarbonate, but unfortunately, under normal conditions the kidney already reabsorbs close to 100% of bicarbonate. So unfortunately this mechanism cannot be increased enough to fully compensate for respiratory acidosis.

The kidney has other methods to indirectly raise the pH, like excreting ammonium (which would otherwise be reacted with bicarbonate in the liver). Other acids can also be excreted. However, renal compensation for respiratory acidosis takes days. The lungs are much faster at removing acid from the body than the kidney is.