If I had about a thousand files selected and then clicked on one of them it would take a few seconds to deselect all the others because the listview and the status bar would update after each deselection. The same would happen if the up or down arrow was pressed.
So I made a patch for myself. I might have broken something with it but I haven't noticed any bad side-effects yet.
I expect that there's a better way to do this but I didn't want to spend too much time just trying to understand how the whole thing is designed.