This is a hard one. You can simply say, "M schedule is really busy, but I am usually (or always, if this is the truth) open for squash and pool"

It is up to you to set boundaries. Eventually, he will accept them.