Innovation

How machine learning will affect IT operations and careers

On both a consumer and enterprise level, machine learning is utilized more frequently than people realize. Find out why and how you can implement it at your business.


Machine learning (ML) remains an area of strong investment these days as businesses seek to automate operations via intelligent mechanisms, which can adjust and adapt as needed. This reduces the need for human intervention—provided the right series of controls are in place.

However, there is no one-size-fits-all approach to adopting machine learning; most companies and the departments therein approach the concept from different perspectives with an array of various objectives. Some of these objectives are more coherent than others, which produces inefficiencies and unexpected outcomes, and may eventually cause a machine learning shakeout.

Before companies take on any machine learning project, they need clear goals so as to establish an effective machine learning strategy which drives real business value.

SEE: Research: Companies lack skills to implement and support AI and machine learning (Tech Pro Research)

I spoke to Scott Clark, CEO of SigOpt, a SaaS automation platform organization, who shared ways to eliminate inefficiencies and produce better outcomes through machine learning. Below are excerpts from that interview.

Scott Matteson: What are some examples of daily usage of machine learning?

Scott Clark: On both a consumer and enterprise level, machine learning is utilized a lot more frequently than people realize. That chatbot on eBay's website asking if you need help with anything? It's using machine learning to deliver highly interactive customer service. Running late to a meeting and checking the traffic on your GPS? Your navigation service is using machine learning to average daily experiences and estimate areas of congestion.

On a bigger scale, practitioners use machine learning to improve their organization's services and increase the company's bottom line. Credit card companies implement the technology to detect fraud, hedge funds use it to improve their algorithmic trading models, insurance companies refine their risk models with machine learning, and more.

Most machine learning work is done to improve organizational practices and happens behind the scenes. It is similar to the invisible hand that guides the free market - imperceptible at times, but constantly driving better, more efficient outcomes.

It is similar to the invisible hand that guides the free market - imperceptible at times, but constantly driving better, more efficient outcomes. Scott Clark

Scott Matteson: When is machine learning more useful than humans? Where do humans do things better?

Scott Clark: Humans are good at forming the big picture, asking and answering the questions: "What problem do I want to solve? What do I want my business to achieve?" Once you've determined the type of solution you want to obtain, that's when machine learning becomes most effective.

Humans are good at asking the right questions, while machine learning is good at ingesting massive amounts of information (often more than a human could look at in a lifetime) to arrive at the right answers. Together they can have a massive business impact, but you need to make sure you're asking the right questions first and not letting the cart lead the horse.

Data scientists should develop these model pipelines themselves, leveraging their unique intuition and expertise about the data and application. However, the key to truly effective machine learning is to automate - and optimize - tasks that don't benefit from domain expertise. Certain aspects of data preparation, training operations, and model tuning are three steps in the machine learning process, which have a large impact on the accuracy and effectiveness of a model but are orthogonal to the expertise required of the practitioner to solve a specific application and should be automated.

In this sense, it is almost always going to be true that humans should provide domain expertise that informs modeling, and machines should augment them by automating the tasks in the modeling process that are either computationally intensive or relate to scaling infrastructure in a way that does not lend itself to human expertise.

SEE: Special report: How to implement AI and machine learning (free PDF) (TechRepublic)

Scott Matteson: Which industries will benefit the most by machine learning? The least?

Scott Clark: Machine learning is about reducing the cost and increasing the accuracy of prediction. Nearly every business can benefit from better prediction. Historically, many predictions were inaccessible because they were too costly or required too much data to be manually analyzed. Machine learning is changing this in applications from predicting what you'll buy next, to what the car in front of you is going to do, to how the stock market will respond to a tweet, and beyond.

I fully expect machine learning to transform predictions, and in turn, business models, across nearly every single industry. Certain industries are more prepared than others to take advantage of this change. So I see this evolving in parallel across different fields at different rates, over time.

Today, we work with a lot of industries who are a bit more advanced in machine learning. This includes algorithmic trading, finance, insurance, enterprise technology, government, and "AI for AI" services. But we are already beginning to explore use cases that cut across healthcare, manufacturing, robotics, energy, utilities, media, entertainment, and more. I truly believe we are on the cusp of machine learning fundamentally transforming every market in a positive way.

SEE: Artificial intelligence: Trends, obstacles, and potential wins (Tech Pro Research)

Scott Matteson: How can IT professionals safeguard their careers or be flexible in adapting in the face of machine learning?

Scott Clark: There's a huge demand for data scientists - in August alone, employers were seeking 151,717 more data scientists than exist in the US. The fear that machine learning will take jobs away from IT professionals is misplaced; rather, IT professionals have an opportunity to adapt their skillset in order to fill this need and take advantage of the 'quant crunch.'

In August alone, employers were seeking 151,717 more data scientists than exist in the US. Scott Clark

There are a variety of tools available that automate machine learning processes, making the barrier to entry lower for IT pros not well-versed in AI and ML. For example, SigOpt's automated hyperparameter tuning takes manual optimization off the table, allowing practitioners to focus on the big picture. Even with these tools, there are so many problems to solve and so much yet to be done that I think it will be a long time before there is any shortage of demand for IT professionals.

Similarly, there are a variety of boot camps and other training programs that we encounter where technical professionals are acquiring advanced machine learning skills, allowing IT professionals to pivot into this new field. My team interacts with students from Insight Data Science, Metis, Berkeley, Galvanize, and other programs across the Bay. These are great opportunities for upskilling.

Scott Matteson: What are some of the ways machine learning eliminates efficiencies?

Scott Clark: There are at least three ways in which this happens.

First, machine learning automates tasks that are computationally challenging or impossible for humans. Imagine doing ten-dimensional optimization in your head, or looking at a billion data points coming out of an airplane engine to determine if it needs maintenance. These are just fundamentally challenging problems that are ideal for ML to tackle.

SEE: IT leader's guide to the future of artificial intelligence (Tech Pro Research)

Second, it serves as an extension of humans in a way that drives efficient outcomes. Imagine improving recommendations to such an extent that you never have to watch a bad movie again (unless you want to). Companies like Netflix and Amazon are on this path.

Imagine improving recommendations to such an extent that you never have to watch a bad movie again (unless you want to). Companies like Netflix and Amazon are on this path. Scott Clark

Finally, it empowers humans to tackle entirely new challenges. When you vastly reduce the cost of prediction in time and energy, you quickly get to a point where business models change. Certain businesses that were inefficient become efficient, expanding the potential for new types of services that provide efficiency or value to society.

Scott Matteson: What is the future of machine learning?

Scott Clark: Machine learning has been around for decades, and so many of the mathematical foundations are actually very well established, while others are just now being explored deeply.

These foundations are equivalent to powder kegs of latent explosive potential that are prepared to transform a variety of industriesAt SigOpt, we often think much more about the "boring" side of machine learning, the applications to enterprise and how this will evolve in the future; and we have only scratched the surface. There are so many opportunities for machine learning to evolve businesses, providing efficiencies and generating revenues in lockstep.

Ultimately, I predict that the cost of prediction will go so low as to fundamentally transform the way most businesses operate, and systemically introducing so many efficiencies that the net value to society will have been vastly underestimated.

We look forward to joining with our enterprise partners to evolve these kinds of productive machine learning systems.

Also see

istock-1056607854machinelearning.jpg
Image: monsitj, Getty Images/iStockphoto

About Scott Matteson

Scott Matteson is a senior systems administrator and freelance technical writer who also performs consulting work for small organizations. He resides in the Greater Boston area with his wife and three children.

Editor's Picks

Free Newsletters, In your Inbox