Totally depends on the content. If I'm reading something by Pinker on neuroscience, I'll generally have to peel back about two or three paragraphs. When I'm reading something like Warhammer fantasy, I can normally just pick up and read and it'll fall back into context quickly.

When I set a book down for a while, I'll typically just skim the previous few pages upon picking it back up to see if I remember what was going on.