You must have, at some point, set your BIOS to 'Boot from USB' (or maybe it's called 'Boot from External Device'). That's why it's booting from a pen drive when it's inserted at power-on.
Hit whichever key your laptop requires, IMMEDIATELY AFTER your Splash Screen to gain access to your BIOS. From there you can change the Boot Sequence, altering it so that your HDD appears in the sequence BEFORE your pen drive.
As to constantly booting (successfully) with your pen drive still inserted - that is NOT advisable.
Just as there is a recognised method for Safely Removing a USB pen drive, there is also a reason for it. Sometimes pen drives can be a bit lackadaisical to complete the writing of files, so eventually you may shut down before the USB has finished writing.
On the next start-up, the files you are expecting to find on the pen drive won't be there OR the pen drive WON'T BE THERE as far as the system is concerned.
Any computer component can only die once. ;)