Message flag goes to the recipient

I have a problem with the flags, if I am composing a email and I hit the flag to remind me later, the recipient receives the same flag on there end, which in turn screws up their flagging system, has anyone figured out a way to fix this. I also created a custom toolbar with the flags but I cant add custom toolbars to an outgoing message. Any advise??