Tech & Work

Expert scores and ranks online legalese: The results might surprise you

Michael Kassner interviews an attorney who reads (yes, reads) and ranks online user agreements. Guess who came out on top and where TechRepublic ranked?

Last night, Microsoft began notifying users, of its online services, about changes to the Microsoft Services Agreement. A cursory glance might make it seem like all that's happened is some innocent redrafting. Microsoft says they've "modified the agreement to make it easier to read and understand" by introducing a "question and answer format."

But take care. There are two key changes — and they are both bad for users.

That's from Andrew Nicol. It gives you an idea as to what he's interested in. From the sound of it, I'm thinking we may want to pay attention.

Pay attention to what?

I've written a lot about online user policies, and a common thread I've observed is everyone believes something needs to be done, but no one seems to know what that something is. Andrew didn't want to wait, so he decided to do what he thought might help. And, you might interested in what that is.

I caught up with Andrew while he was commuting through New York's Union Square, and asked him why he decided to get involved in this thorny issue.

This seemed like a great opportunity to use my knowledge of the law and my interest in technology to try to bring these issues to the attention of more people, and ultimately pressure the sites into being more reasonable.

I started by reading only Terms of Service, but then realized it made more sense to provide a comprehensive report card for each company, taking into account all of their agreements with users as well as their actual practices.

Read every word?

I thought I heard Andrew say he personally reads in entirety the Terms of Service (ToS) and Privacy Policy of the site he is examining. With a healthy dose of skepticism, I asked again to make sure.

That's right. For each site, I read all of the legal agreements that it has with its users (which normally include the terms of service and privacy policy, sometimes an acceptable use policy).

I have a master spreadsheet with the relevant provisions in each category from each site which lets me compare the site to its peers. I also spend some time investigating the site's actual practices (for example, how it responds to government data requests). Whenever possible, I speak to someone from the company about my concerns.

That is ambitious. Not long ago, I wrote an article reporting how two privacy experts determined each of us could easily spend 200 hours a year reading all the legalese that we are presented while on the Internet.


Andrew decided to share what he found with the rest of us. The result is a website called Clickwrapped. I wondered about the name Clickwrapped. I was about to ask Andrew, but thought it best to figure this one out myself. It turns out the term clickwrap has significance. This is an excerpt of the term's definition on Wikipedia:

A clickwrap agreement is a common type of agreement often used in connection with software licenses. The name "clickwrap" came from the use of "shrink wrap contracts" commonly used in boxed software purchases, which contain a notice that by tearing open the shrinkwrap, the user assents to the software terms enclosed within.

When you visit Andrew's website, the first thing you see is the following diagram:

Click the image to enlarge.

Wikipedia is worse than Google and Facebook? No way. I settled down after reading that the higher the score, the better.

Each site is scored out of 100, with points allocated equally between four categories: Data Use, Data Disclosure, Amendment & Termination, and Miscellaneous. Although scoring necessarily involves the exercise of some discretion, we try to be as clear as we can about the criteria and we explain on each review page the reasons for the score we have awarded.

If you hover your pointer over each color bar of a specific website, a new window will open with the numerical results. If you click on the same colored bar, you will be sent to Andrew's explanation of why the website received that ranking.

Results table

The following chart is for those fond of numbers:

So, Wikipedia is the best of the bunch, but self-serving Facebook and Google are right up there? I asked Andrew if the high ranking of Google and Facebook was a mistake.

Going into the project, I was aware that both of these sites had been widely criticized for failing to respect their users' privacy. A lot of this is just because of their size: they are the two most popular sites on the internet, and so it is to be expected that they get the most attention in the press.

What I found is that neither of these sites deserve all of this negative press — at least not when they are compared to their competitors. Facebook was a surprise. It has definitely made some mistakes in the past, but right now, it ranks well on most issues.

Once again, I was skeptical, deciding to investigate how Andrew came up with the rankings. As Andrew mentioned, he looks at four categories asking the following questions.

Data Use:

  • What data does the site collect?
  • What can the site do with content you post?
  • Does the site get more rights to your content than it needs?

Data Disclosure:

  • When can the site disclose your information to others?
  • Does it tell you if the government wants your data?
  • Is it transparent about government requests?

Amendment & Termination:

  • Does the site give you notice when it changes its terms?
  • Under what circumstances can your account be terminated?
  • Does the site let you take your data to another service?


  • What else should you know about the site's legal agreements?

I asked Andrew to take a look at TechRepublic's legal policies (officially CBS Interactive) and see what he thought. Andrew wanted me to mention that this was only a crude first-pass assessment and must be viewed as such. Can you tell he's an attorney?

Here are the results for TechRepublic:

A tie with Microsoft... .

A few more questions

I took advantage of being in contact with someone who actually reads the ToS and Privacy Policies of sites — and, as a technically inclined attorney, understands them. I was curious as to his threshold, asking him which website policies would be acceptable and which weren't?

Like anyone else, I would find it difficult to run away from some popular sites. For example, even if I had more concerns about Facebook's terms, I would probably still be on there because I use it so much and have invested so much of my time and effort in its platform.

But, since doing this research, there are some services I am avoiding. I tell my friends to use Dropbox rather than Google Drive. And I now avoid PayPal and use other online payment services.

Next, I wanted to find out what it would take to obsolete the Clickwrapped website?

This is a great question. Clickwrapped will no longer be needed when there is a fair balance between company rights and user rights. There are a few simple actions that each company could take to achieve such a balance.

  • They should limit how they use our data.
  • They need to be more reasonable about handing over our personal information to the government.
  • They need to go back and take a look at their agreements and eliminate any terms that are unnecessarily one-sided. For example, they should not be allowed to close our accounts at any time and for no reason.

Final thoughts

As for agreeing with Andrew's opinion about ranking or not, at least there is some digital discussion going on. That has to be better than automatically agreeing. I also find Andrew's honesty about using Facebook regardless of its ranking refreshing.


Information is my field...Writing is my passion...Coupling the two is my mission.

Editor's Picks