We have this game as well (Wer hat Angst vorm schwarzen Mann?) and according to Wikipedia, the black man either stands for a shadowy figure, a man in black clothes, a chimney sweeper, a man with dark skin or the black plague (it's more than 200 years old)
And regarding the black Peter, my German penpal in middle school gave me a cardgame called Schwarze Petra! She told me it's a common game in Germany (she was from Dresden, maybe it's a regional thing?), it's basically like a memory game where you have to reunite little couples of Petra and her Peter (two bakers, two butchers...) except for one, Schwarze Petra, who has no Peter because she's too dirty (she's a chimney sweeper!!). So when you get her your turn is over and you have to reshuffle the cards.
These two games share some eerie similarities!
These two games share some eerie similarities!