You can, but you can’t

I'm guessing it means you can't control the people themselves, but you can control who you spend time with?