How will the Snowden case affect data analysis?
The amount of data we all produce each day is staggering, and this Big Data can be both used to improve the goods and services we use, and also for more sinister ends.
Edward Snowden undoubtedly acted against the law when he released information about how intelligence services use big data to watch individuals and groups, but sometimes the lines between right and wrong when it comes to how and when to use big data analysis are blurred.
Governments have been slow to catch on the valuable information gained through analysing big data, and their experts in the past have been less skilled than those working for private companies. However this is all changing and governments are giving more and more people the top level clearance needed to examine personal data which flies back and forward over the internet.
There are many groups across the world who try to hack into databases and exploit valuable data for their own criminal or terrorist means, but at the same time there is a need to restrict access to the data to those who really need to see it due to concerns about privacy violations and civil liberties.
Big data is also big business, and several companies have grown up, mostly in the Silicon Valley area, which specialise in helping governments and companies analyse the big data at their disposal and improve their targeting of certain people or groups. Software engineers tread a fine line between what their software is capable of and what is within the law and is morally right.
Government departments have been historically very poor at procurement, and their systems have not been designed with big data analysis in mind.
This is rapidly changing and a growing number of companies are focusing purely on helping government departments across the globe.