Headphones carry sound straight into your ear and draw a clear line between you and the outside world. For tasks that need tight focus, that isolation makes a real difference, since your brain no longer has to filter anything from the room.
Speakers spread sound through the room instead, which feels more natural and less enclosed. If you've been on headphones all day, switching to speakers in the evening gives your ears an actual break.
There's no strict rule, but the general pattern holds, headphones for narrow single-task focus, speakers when you want a wider, more relaxed atmosphere.