The most tiring part of open office noise isn't its volume, it's how unpredictable it is. Someone laughing, a phone ringing, a chair creaking, all random, and attention snaps toward it every time.
Listening to a sound you chose yourself, steady and predictable, through headphones masks that randomness to a real degree. The office noise is still there, it just no longer creates a sharp enough contrast to pull your attention away.
Instead of trying to silence the noise entirely, making it predictable is the more realistic fix for most people working in an open office.