Actually a lot of voice actors work really hard. In order to actually be able to make a living out of it you usually have to audition for a lot of parts, even the shitty ones that nobody wants to do. The more recognised VAs get parts whereas the newer ones don't because they haven't had a chance to prove themselves yet. They have to try and get a lot of expression done through their voices to make the character believable without actually doing what the character is doing. Then those roles that include a lot of screaming or voices like Gollum which are really taxing on the vocal cords. Voice acting is not easy.
1. Andy Serkins (Gollum in LOTR and The Hobbit)