I did this all the time.

just use the part filled with ink (the long white part with the pen tip)

True, but it just doesn't feel.. Natural, I guess.
With the real pen all together, it's more bulky.
When you use just the pen-y bit it's awkward

or when you try and look for it around your desk with out drawing attention to yourself