And the Swiss are scary

Yeah, no. That was not the reason Hitler didn't attack Switzerland. He needed "neutral" countries like Switzerland and Sweden as economic partners much more than conquering them would have benefitted Germany during the war.