Onwards to data-driven world, part 1: Where are we heading to
Onwards to data-driven world, part 1: Where are we heading to?
Data analytics is changing the world. Nowadays it is impossible to avoid being exposed to this idea over and over again. There are variations to it, most of the time it is artificial intelligence or deep learning that is going to change everything, while it was data science and big data just some years ago. However, the main idea remains. The world is going to be changed by same type of activity where knowledge is extracted from data and used for benefit.
So what it actually means that data analytics is going to change something? And is it even true? To me the answer is very much connected to software. Wall Street Journal introduced the famous headline Why Software Is Eating The World back in 2011 and the idea is still relevant. Data driven forces of digitalisation, automation, artificial intelligence, and autonomous agents are all driven by software.
Software is both the starting point and the final destination for modern data driven systems. Everything needs to start from something that is controlled by software, or at least observed by it. From there on it flows through data analysis pipelines, which are typically programs written by data scientists and running on cloud. That is, software running on software. Finally, we arrive at the result of data analysis, which increasingly is some form of automation instead of reports for humans to act upon.
In essence, data analytics cannot happen without software. And for data analytics to make a difference, more software is needed. Are we incorrect in saying that data scientist is the hottest profession, if it is the software developer who’s in control? Will programmers be kings of the data-driven world?
To me these revolutions are two sides of a same coin. Data revolution is not possible without software, but similarly, software cannot keep on eating the world without data analytics. I see that we have reached a point where the logic and rules of software systems cannot by typed in by programmers, like it has been for decades. Hand-crafting hard-coded rules is too slow and too much work to maintain. We need to start learning the rules from data and the next generation of software needs to be driven by data.
To answer my question, software developers will not be the kings, but they will be the enablers of the data driven revolution. The focus of professional software development will drift towards more demanding tasks and the number of entry level programmer positions will probably decrease, being replaced by new positions in all levels of the data science profession.
In the next part I’ll dig deeper into how I see the practical change in software and programming.
The author is development manager of data analytics at CSC