Luckily I never lost them

Now that person knows which house they can break into.

Then you hang out in the wrong places

You can lose your keys walking down the street, it doesn't have anything to do with who you hang out with.

Areas. There are certain areas where the robbers are present, usually, and this would give them an opportunity.