A lot (not all, but a lot) of the overhead of creating connections occurs on the server-side. Look up connection pooling, as it might eliminate some of the overhead. Also you might look at maintaining a set number of persistent connections and reusing them client-side. The client-server communication/handshaking and memory allocation/deallocation operations in opening and closing connections account for a lot of the connection overhead on the client side. I.e., this pattern is fairly slow (pseudo-code algorithm for demo purposes):
Loop 10 Times
::Open Connection to SQL Server
::::Insert a row in a table
::Close Connection to SQL Server
End Loop
The following is much faster than the above example, because the connection overhead is only encountered once, instead of ten times:
Open Connection to SQL Server
::Loop 10 Times
::::Insert a row in a table
::End Loop
Close Connection to SQL Server
You could extrapolate this pattern to a web service, DCOM object or whatever else that requires connection to a SQL database; i.e.:
User connects to web service
::Open connection to SQL Server
::::Insert a row in a table
::Close connection to SQL Server
User disconnects from web service
The above might run a lot more slowly than opening a connection to the SQL Server one time and using that same connection for all separate web service requests.