The scene they were filming was where they were all at the table, so the big actors, not the stand-ins had to be in the scene. So because he had to be larger than dwarves and hobbits, he had to do his scene and then the others had to do the scene separately so that they could be added together to get the sizing correctly.
Was everyone else already done with their scenes and just CGI'd into the scene with him...?
Nope, this actually happend.