Whenever I go to the gym

Thats only in the beginning. True your body would feel sore for a few day when you start but it will slowly fade.

If you aren't struggling, you're doing it wrong.