I guess you could call it a downside

I don't have both

This is true. Doing something you love for a living can turn it into a grind.