In person, silence doesn't bother you much because body language, eye contact, small movements all fill the gap. On a call, all of that disappears and you're left with just the pause.
Your brain reads that gap as a possible problem, did the call drop, is the other person upset, and starts running through scenarios fast. That's why a silence that's really only 3 or 4 seconds ends up feeling much longer.
Just knowing this helps. Silence on a call usually doesn't mean anything broke, it's just a normal gap in a conversation missing its usual cues. A bit of ambient sound in the background tends to make that gap feel a lot less tense.