I worked at a cluttered desk for a long time, telling myself it didn't matter. For one week I tried leaving only what I actually needed in the moment on my desk, and the difference was bigger than I expected.
Every object in your field of view creates a tiny “what is this, do I need it” question somewhere in your brain. Each one is small on its own, but together they keep quietly pulling at your attention.
You don't need to clear the desk completely, just move whatever isn't relevant to the current task out of view. Simple, but it genuinely works.