I apologize in advance for preaching, but here goes:
An open office solution needs an open office problem statement. Any standardized evaluation criteria will be ineffective for your organization without a concise statement of goals and realistic expectations that have been vetted with as high a level of organizational management as you can manage to pull in.
Solutions don't exist in a vacuum and they cannot be compared objectively except in the framework of providing their solutionto a specific set of 'problems'.
In short: 1) Identify the goals/ expectations/ short-comings you wish to address with this software. Be as specific as possible, encouraging team members to speak to sample documents, processes, etc. that they expect to be 'solved' through this software (Remember that when it's all over, you have to support this s/w and if people had unrealistic expectations, it will be a *real* headache for you).
2) Match s/w features with needs collected above
3) Call their references. Arrange to visit installations and to talk without the salespeople present. Be wary of relationships between vendors and sites that appear 'too close'. You might not get an unbiased story.
4) Don't forget performance. benchmark with other organizations if possible. Feature-rich software can be slow.
I'm not sure this helped and it's certainly not original, but sometimes these points need repeating.
Best of Luck with your project!
