August 10, 2021
Big Data Analytics & Machine Learning – Today’s Competitive Advantage
As the step beyond the Descriptive Analytics, which provides description, classification and clustering of data, Predictive Analytics have been in use in certain industries for some time, as the process of utilizing data mining, statistics and modeling to make predictions.
Predictive Analytics have operated on collected, numerical, structured data traditionally. With the proliferation of data, both structured and unstructured, the ability to realize value out of the big data has become a key competitive advantage for not only the business world, but also the governments, not the least of those dealing with the geopolitical challenges.
The deep neural network (DNN) breakthrough in the 2011 ImageNet competition has driven fast advancement of DNN based machine learning (ML) applications over the last decade. While only representing the low-level classification and object recognition part of the broad artificial intelligence (AI) research, DNN and reinforcement learning (RL), another part of the ML, are pushing the adoption of ML and AI into a wide variety of new applications at an accelerating pace.
Predictive Analytics is a major beneficiary among the various ML/AI applications. The ML/AI algorithms not only afford more flexible and adaptable analyses of structured numerical data , but are increasingly integrating the ability to identify complex patterns from unstructured data, including visual, auditory, cultural and emotional cues, to produce insights that make prospecting and decision making more effective. While it is impractical and error prone for humans to pull complex patterns out of voluminous high dimensional data sets, properly trained and continuously evolving ML/AI models can take on many such tasks in different applications.
Prescriptive analytics is a next frontier beyond predictive analytics, where ML/AI is enabling the recommendation and, in some cases, the implementation of the recommended actions. Of course, as disruptive technologies, ML/AI are applicable well beyond the advanced analytics, and is thus often referred to as “the most important tech of the 21st century.”
Market Projections
Couple market research reports published in late 2020 and early 2021 both projected 24.5% Compound Annual Growth Rate (CAGR) through 2025 and 2026 respectively, for the Predictive Analytics market, forecasting $21.5 billion by 2025 and $22.1 billion by 2026.
Market research reports on the broad encompassing ML/AI market have projected much larger and faster growing market demands, as can be expected. The Million Insight’s report in 2020 projected a 43.8% CAGR from 2019-2025, from $6.9 billion market size in 2018 to $96.7 billion in 2025, with ~8% in hardware, ~36% in software, and ~56% in services.
The most recent Fortune Business Insights ML market research report published in June of 2021 forecasted 38.6% CAGR, from $11.33 billion market size in 2020 to $152.24 billion in 2028. Amidst the unprecedented and staggering global impact of COVID-19, ML solutions/services witnessed a positive demand shock across all regions.
While the large enterprises accounted for the largest ML market share in 2020, the small & medium enterprises (SME) are increasing their technology spending’s to deploy ML/AI solutions.
Industry wise, IT & telecommunications, automotive & transportation, and healthcare are largest verticals investing in ML/AI, while banking, financial services and insurance (BFSI), retail, manufacturing, advertising & media are the verticals following close behind.
Use Cases
Many industry use cases of advance big data analytics are enumerated in articles on the web, such as “How Machine Learning Can Boost Your Predictive Analytics” and “Smart Implementation of Machine Learning and AI in Data Analysis: 50 Examples, Use Cases and Insights on Leveraging AI and ML in Data Analytics“, where the later article also mentions ML/AI use cases beyond analytics. Here is a quick summary of some common advanced analytics use cases to provide a feel of how the integration of ML/AI boost the values of analytics across different industries and scenarios.
The most common use case in sales & marketing (S&M) is to identify and acquire prospects with attributes similar to existing customers. At the broad level, customer segmentation based on customer responses and purchase patterns enables marketing strategies tailored to each segment’s characteristics.
In e-Commerce, customer behavior and preferences analyses enable personalized, customized product recommendations. Prioritizing known prospects, leads, and accounts based on their likelihood to take action could lead to radical improvement of lead conversion rates. Taking it one step further, combining historical data points of customer behavior with market trends for 360-degree view of the prospective customer forms data-based, streamlined S&M activities, instead of spray & pray.
At the other end of the customer engagements, identification of customers on the verge of leaving could lead to insights that enable proactive adjustments of products, content, packaging, and services for better customer retention. And, on the operations end, more accurate predictions of product demands and timing help optimize the supply chain, inventory, logistics and warehousing.
In financial services, advanced analytics identify risk areas and profiles from historical and current transactions for fraud detection, defaults predictions, etc. These enable decisions to prevent or mitigate risks. On the flip side, effective demand forecasting allows improved operations planning, which in turn enables longer range revenue projections and planning.
The healthcare sector is experimenting with ML/AI based advanced big data analytics across a wide range of tasks, while adapting processes and protocols for effective adoption of technology assistance. Streamlining large sets of unstructured data (symptoms, treatments, outcomes, etc.) and deriving insights facilitate more accurate and faster diagnosis, enable improved medication and treatment prescription, and reduce recurrence and/or readmission likelihood. As well, better resources demand predictions and planning facilitate operational efficiency and patient care improve.
With digital transformations and continued expansion of network connectivity, every business and organization need to face the cybersecurity risks and protect their business-critical assets from cyber-attacks. Processing of vast amount of structured and unstructured data, and analyzing the traffic in real-time to identify and tract unusual patterns could help business and organizations ward off attacks or minimize damages. Automated collation and compilation of data over time into reports and actionable insights are critical to enabling design of protection strategies and adjustment/development of IT and business processes, which could help reduce the scope of human errors, often the primary security risk that opens up the attack surface.
Technologies, Platforms, Tools, Applications
Advanced analytics are, by necessity, built on top of a scalable big data infrastructure foundation, as depicted in the “Roadmap: Data Infrastructure“. Many big tech powerhouses have open-sourced tools, models, and platforms that added to and expanded on the pioneering Apache Hadoop batch processing, offering capabilities such as the streaming-data real-time processing (e.g. Apache Spark), data cleansing & integration, dynamic scalable platforms, ML/AI models, automated ML model training (e.g. AutoML), etc.
Major cloud services providers offer common open source, as well as proprietary, tools and platforms, from AWS, Google, MSFT, to IBM, Oracle, SAP. Various application specific business intelligence (BI) and decision support tools, from descriptive, predictive and increasing prescriptive analytics, are developed on these cloud services.
Growth of the Predictive to Prescriptive Analytics
Each and every one of the industry sectors is in the early stages of adopting ML/AI driven advanced analytics, to effectively deal with and address major challenges, from cybersecurity, environmental sustainability, cost and resource optimization to geopolitical risk management.
As the data volume and diversity continue and the pace of expansion accelerates, the need for intelligent and timely insight extraction from the data, prediction of event developments, and actionable recommendations of course of actions will only become more intense. The development of various autonomous systems is already pushing the boundary and driving the demands for prescriptive analytics.
While the ML/AI solutions aren’t mature or sophisticated enough to support complex autonomous decision-making, such as general clinical diagnosis, handling of large number of corner cases in autonomous driving, etc., they can be used to reduce routine, time-consuming, resource-intensive tasks, to allow personnel who typically attend to those tasks to focus on higher-end work instead. An example in the healthcare sector is to utilize NLP and ML-driven predictive analytics to review a patient’s history in the EMR/EHR, to provide recommendations on what may be most critical, based on the patient’s present symptoms.
As with the descriptive analytics extending to predictive analytics now to help us understand what’s happening, the world will see the proliferation of prescriptive analytics that will help us take necessary and timely actions, sooner than later. And, all of these, before we’ll need to worry about hearing “I’m sorry Dave, I’m afraid I can’t do that.”