Insert varchar value to number field

Can i insert the values like this
INSERT INTO sara SELECT To_Number(agreementid) from emp
Here sara table have agreementid as number(18,4)
emp table have agreementid as varchar(255) but it contains only number in that field.

While inserting i got ORA-01722 INVALID NUMBER error. I check there is not character in that field.
Can anybody help me to insert from one table to another.