Why I have problems

I think it's because we mentally group the bottom and right buttons in a separate group from the top and left ones. I've never had an issue with confusing the location of the X button on the PS and SNES controllers either, but fuck me it took a long to nail down the opposite positions of the X and Y buttons between Nintendo controllers and Xbox 360 controllers.

It all depends on what you learned first.