It's very rare that office users use bandwidth continuously, unless they're seriously file sharing, which I assume you have turned off in your filewall.

Most often, a user is using the network to serve files and printers and web pages, and using ANY bandwidth less than 10% of the time. With that in mind, 20 users to an access point would be thoroughly reasonable and give performance in the same ballpark as having 1 user per access point, on average, as experienced by the user.

If your users are doing something different, like very frequently accessing and re-saving 5 Mb Powerpoints and/or accessing 700 Mb video files, that's different. But for garden-variety offices, it's a reasonable recommendation.