It's more that your computer should be considered compromised regardless if you allow someone to gain physical access to it. You should know where your device is at all times.
I'm not willing to throw away my device's resale value to block a physical compromise attack.
It's about noise. The physical lock on your door doesn't prevent anyone from breaking in. But you can clearly tell someone broke in and it creates noise.
It's not always practical to take your notebook to the toilet with you, so you want to make it as hard as possible for someone to compromise it in a limited time frame.
Security is never absolute, it just about buying time to notice an attack. This applies to the physical lock on your door just as much as it does on your notebook
