I retain characters more often than plot, but what seems to happen is that I latch on to specific moments, turns of phrase, and dialogue as touchstones for me to recall what happened in the book. Kind of like freeze-frame.