It really depends on your build. If you have high evasion but low dodge, I'd go low. Being able to save 4 damage on every melee attack can stack up (though I guess knives and fists are even less than that).
Otherwise, I tend to always go Medium. It might save you from a sniper rifle Aimed Shot once in a while.