Sometimes it can take longer for an Oracle client to look up the network connection details for a database than it takes to connect to that database.
So you might want to check that your Oracle connection uses a local TNSNAMES.ORA file on your client PC to look up the database name / server / port that you are connecting to.
The Windows registry entry TNS_ADMIN will control which TNSNAMES.ORA file is used. This reg entry will be somewhere in HKLM\SOFTWARE\ORACLE - but exactly where will depend on your Oracle version / client installation details.
If your TNS_ADMIN entry points to a Network server then this could explain your delay.
[If you have no TNS_ADMIN setting then a local TNSNAMES.ORA file will be used].
Hope this helps.