Umm, no. The judge would give you the opportunity to say yes and if you still didn't would hold you in contempt of court, fine you, and possibly hold in jail until you agreed or a predetermined time passed, usually 30 days. If the case was still ongoing you may be called back and have the whole process start again.
The Fifth only protects you from incriminating yourself. Swearing to tell the truth can in no way incriminate you since no question has been asked. Lying after the oath is committing perjury and is a crime in itself. Pleading the Fifth is a way of answering without answering since refusing to answer a question allowed by the judge can lead to them finding you in contempt. The Fifth is there to protect you and your right to be innocent until proven guilty. Now keep in mind if you plead the Fifth and it comes to light the answer would not have incriminated you, you can be found in contempt but it's entirely up to the judge.
There's an orange dude that is going to go down while showing in real time how this works. It's going to be fucking epic.