IT Employment

The problem with learning Big Data in school? Not having any data

The biggest stumbling block for those teaching these classes and those wanting to learn is little to no access to large, relevant data sets.

Unless you've been living under 12 feet of ice in the Antarctic Plateau, you know that Big Data is the latest IT buzz term. IT careerists all over are advising folks get on the Big Data wagon.

Colleges are scrambling to create programs in business intelligence and Big Data analytics for those who want to work in a world overrun by data. But how's this for cruel irony? The biggest stumbling block for those teaching these classes and those wanting to learn is little to no access to large, relevant data sets.

It makes sense. What entity is going to hand over its data to be experimented on? The findings come from a recent survey of professors and students (professors at 243 universities in 43 countries and students at 96 universities) conducted by Teradata and Barb Wixom, an associate professor at the University of Virginia.

If you're thinking about taking classes in order to do data analysis in an IT/discipline, it might be a good idea to ask your advisor specifically about the availability of data to work with.


Toni Bowers is Managing Editor of TechRepublic and is the award-winning blogger of the Career Management blog. She has edited newsletters, books, and web sites pertaining to software, IT career, and IT management issues.


"Pursuit of professional advancement as one's chief or sole aim:" is the definition of a "careerist", so why would anyone take any notice of someone like this? They are far too self centered to be any kind of authority on where IT should be heading.


Two thoughts, Toni where did you get the data that supports your data not available hypothesis? And for those interested in exploring Big Data - watch for courses on Coursera.

rduncan 2 Like

Public data sets are freely available, Google have 103 sets - first search result returned ..ask you local weather station, get cenus data from around the world, make an FOI request- I think perhaps you meant hosting your own large data sets on fast storage is the problem - use the cloud!

yhk331 1 Like

I just want to let people know that you do have access to big data. For example, Kaggle offers competition which comes with large amount of data.

ahanse 1 Like

Better to check the credentials of those teaching...


When I was working with big data, for test systems I wrote an SQL script that read text files of names and stuff. And made up data. Then I just let it run for a few hours, creating a couple million records.

Archives Steve
Archives Steve 1 Like

I agree with dbmarketing. The Federal government has lots of big data sets available for the asking. Many are published on The National Archives has more.

dbmarketing 2 Like

When I was in my political science program we pulled data off of the census results continually, and compared it to data from other federal and state programs. If millions upon millions of records isn't big enough for ya, I can't help you :)

Editor's Picks