No it's not just rude, it forcefully breaks the prayer because for a brief moment, they weren't praying to God, they were praying to the person passing in front of them. It's the same reason we can't pray facing a photograph or poster with someone's face on it.
We're not allowed to bow down before ANYONE, praying or not. It doesn't matter that we know, it matters that we do not under any circumstances bow down to (or pray facing) any person or face.
tldr
It's the symbolism that matters.