Errors do not have that great an impact
You're right to point out that reassignments take a huge amount of time, but they just don't happen that frequently to affect overall performance. The same goes for ECC errors. A lot of small ECC errors can even be corrected on the fly by the read logic without any extra action by the drive's microprocessor.
Smart controllers with cache do have an impact. In particular, faster controller microprocessors with faster data paths are real advantages. However, most storage systems have layers of cache (disk drive, controller, system, etc.) and there are diminishing returns to be had by cache. And when manufacturers publish their own benchmarks, they often arrange the tests so as much of the IOs can occur out of cache as possible. Thus while cache does help in the real world, it often contributes to manufacturers overselling the performance of their storage subsystems.
