In most areas, there is one or maybe 2 providers for broadband internet, the cable company and phone for DSL. Both have been granted limited monopoly status in most cases but have to agree to certain common carrier limitations. Both have been subsidized, if by nothing else but the use of emminent domain to run wires across private land without lease or purchase.
Basically, this counters the argument that if the ISP screws this up (either intentionally or not) that the market will correct. Thus the market can’t correct except by a consumer becoming so frustrated that they switch to dialup.
This is the real critical matter with net neutrality. We don’t have a diversified broadband market and a free market approach will just allow monopoly abuses.
Now, a top down approach will not work either, but that would probably be less damaging than unhindered capitalism. The telecos haven’t got a good track record when it comes to not screwing over the consumer. The whole ‘we need less restriction to be able to afford fiber to the home’ thing comes to mind. Prices skyrocketed and here we are with high speed broadband way behind other countries.
QoS could be a useful service, but we have to be very careful in what power we give the telecos/cable companies. They’ve shown before that they can’t be trusted with their monopoly power.