Fortifying the Future: Safeguarding Machine Learning in the Age of Advancing Threats
Exploring the Crucial Role of Machine Learning Security as the Vanguard of Next-Generation Protection for AI and ML
Welcome to issue #7 of Indiscrete Musings
I write about the world of Cloud Computing and Venture Capital and will most likely fall off the path from time to time. You can expect a bi-weekly to monthly update on specific sectors with Cloud Computing or uncuffed thoughts on the somewhat opaque world that is Venture Capital. I’ll be mostly wrong and sometimes right. Views my own.
Please feel free to subscribe, forward, and share. For more random musings, follow @MrRazzi17
Over the past year, I’ve been watching AI's breakneck pace and innovation with much awe. As the adoption of AI and machine learning models continues to surge, the longstanding “data is the new oil” adage is finally ready for a facelift. In the AI-encompassing world of 2023, “data and models are the new oil” seems more prescient.
Having spent years in security, I’m keenly aware that any momentous technology, however good, can be used for harm (e.g., Social Media, the Internet). This post isn’t about stopping innovation per se or being a benign doomsayer of progress, but rather – and with my security bias in mind – thinking through how security will evolve and the importance of ML security for the next decade onward. To that end, I’ve spent the last several months speaking with the brightest minds ranging from builders, buyers, and general practitioners in the field, and I am excited to share my learnings with you.
Landscape and Newly Addressable Vulnerabilities
Machine learning security, or ML security, protects the integrity, confidentiality, and availability of machine learning models, datasets, and the overall workflow process. While traditional cybersecurity measures have typically focused on securing networks and applications, ML security proactively safeguards the intricate algorithms and models that power machine learning systems.
The need for robust ML security measures becomes more pressing as we delve deeper into the tectonic AI shift. The proliferation of data breaches and cyberattacks, coupled with the adoption of OSS models across the enterprise, has created a perfect storm of potential vulnerabilities. Attackers are increasingly targeting machine learning models to manipulate outputs, compromise their functionality, or extract sensitive information.
Some of the critical vulnerabilities facing enterprises today are:
Adversarial tracks on existing models
Model theft and hijacking
Inability to mass provision guardrails internally (e.g., zero trust controls)
Ransomware embedded within the models which are privy to adopting OSS models
Retaining model integrity throughout the model lifecycle (pre/post-production)
Machine learning models have become more sophisticated, complex, and opaque. Deep learning models can contain millions of parameters and layers, making comprehending their inner workings and vulnerabilities difficult. Some vulnerabilities are becoming common, but we’re still privy to new attack vectors in the coming decade.
Emerging Trends
Machine Learning isn’t new, but to that end, we’re entering what I believe will be a pivotal decade as it relates to the practice of ML and the securing of the supply chain itself. I think that the industry and particularly the importance of securing the sector is further propped up by three key emerging trends that only spotlight the importance of focusing on securing the ML supply chain:
Increase in AI-based attacks: 30% of all AI cyber-attacks are leveraging trending data poisoning, AI model theft, and adversarial samples attached to AI power symptoms1
Data Proliferation: Enterprises today are omitting petabytes of data daily given platform shifts such as mobile and remote work and are responsible for collecting customer data and, more importantly, securing the data
Shared Confusion of Responsibilities: ML touches several teams from BI, Data science teams, ML Engineers, IT, Security, and more – akin to the shared responsibility model pioneered by the hype scalers, more companies are having difficulty owning the responsibility of the models, and thus exposing themselves to nefarious attacks
Propped by these trends and several others, ML security will remain top-of-mind for mature and sophisticated organizations (more below).
One common topic as of late, at least at the board level, has been, what is your “AI” strategy whether you’re at the application layer or infrastructure layer as a company. AI is top of mind for many enterprises. With it, I'm seeing a more significant portion of sophisticated companies (with technical prowess internally) look to adopting open-source models to incorporate some componentized of “AI-ifcation” in their products.
Here also lies the double-edged sword, the OSS adoption is more farseeing because it’s likely that data that enterprises collect from their customers can’t (and shouldn’t) leave their environment, whether for regulatory concerns, compliance, or mere security oversight. So, coupled with OSS adoption, the need to ensure the data/models don’t leave your environment, and a narrow minority of sophisticated enterprises which fall into this bucket, machine learning security companies have considerable tailwinds behind them.
Deployment Types
As I’ve been thinking about the deployment methods of ML security, it’s essential to start with the workflow for ML, generally for most enterprises. I am not an ML expert, so please excuse the highly simplified workflow. There are many tools and ways to secure the ML supply chain. I’ve segmented the security posture against ML into four distinct categories:
Data Preparation – Synthetic Data Providers: Synthetic data providers are companies or services that specialize in generating synthetic or artificially created data that mimics accurate data while maintaining privacy and anonymity. Use cases and examples below:
Data Generation: Synthetic data providers develop algorithms and methodologies to generate synthetic data. They aim to create data that closely resembles authentic data's statistical properties and patterns, which involves generating data points, building relationships between variables, and replicating the characteristics of the target data
Privacy Protection: One of the critical advantages of synthetic data is that it protects the privacy and confidentiality of individuals in the original data set. Synthetic data providers employ data anonymization, de-identification, and differential privacy to ensure that personal information cannot be linked back to individuals
Data Customization: Synthetic data providers allow users to specify their desired characteristics and features in the generated data. This customization could include defining the distribution of variables, identifying correlations between attributes, and controlling the overall structure of the data set. Users can tailor the synthetic data to suit their needs and research requirements
Realistic Data Representation: Synthetic data providers strive to create data that accurately represents the real-world scenarios and challenges of the target domain. This involves capturing the complexities, patterns, and anomalies present in the original data, enabling users to conduct meaningful analysis and testing
Scalability and Diversity: Synthetic data providers often offer scalable solutions that can generate large volumes of data to match the requirements of different use cases. They can create diverse data sets with various scenarios, outliers, and edge cases, comprehensively representing the target domain
Validation and Evaluation: Synthetic data providers may offer tools and techniques to assess the quality and validity of the generated synthetic data. This could involve statistical analysis, visualizations, and comparison with the original data set to ensure that the synthetic data accurately captures the characteristics of the actual data
Integration and Deployment: Once the synthetic data is generated, providers may assist users in integrating it into their existing workflows and systems. They may offer APIs, data formats, or integration support to facilitate the seamless incorporation of synthetic data into various applications and platforms
Training and Validation of Models – AI Firewall:
Behavioral Analysis: AI firewalls learn from historical data/models and establish user, device, and application baseline behavior. They can then detect deviations from the expected model patterns, such as unusual network traffic, access attempts, or data transfer, which could indicate a security breach
Zero-Day Threat Detection: Zero-day vulnerabilities are previously unknown security flaws that cyber attackers exploit before security patches or fixes are available. AI firewalls can analyze network behavior and identify anomalous activities that could indicate zero-day attacks, providing early warnings and proactive defense mechanisms. Given the rise of OSS modes, Zero Day detection is becoming imperative
Validating and Deploying Models – Model Injection Protection:
Adversarial Attacks: Adversarial attacks involve manipulating input data to mislead or deceive machine learning models. These attacks can aim to bypass security measures, cause misclassification, or exploit vulnerabilities in the model. Model injection protection helps detect and mitigate such attacks
Input Validation: Model injection protection involves validating and sanitizing input data before it is used for inference or decision-making by the machine learning model. This process ensures that the input adheres to the expected format, range, or constraints, reducing the risk of injecting malicious or unauthorized content
Data Integrity Checks: Model injection protection may include mechanisms to verify the integrity and authenticity of the model itself. This can involve using cryptographic techniques, digital signatures, or checksums to ensure that the model has not been modified or tampered with since its creation or deployment
Model Verification: Model injection protection can involve periodic or continuous monitoring of the machine learning model to ensure its integrity. This can include verifying the model's architecture, parameters, and weights to detect any unauthorized changes or manipulations
Runtime Anomaly Detection: Model injection protection may employ anomaly detection techniques during runtime to identify abnormal behavior or unexpected outputs from the machine learning model. This can help detect potential model injection attacks and trigger appropriate response mechanisms
Access Control and Authentication: Model injection protection can involve implementing access control mechanisms and authentication protocols to prevent unauthorized access or modification of the model. This includes securing the infrastructure hosting the model, controlling permissions for model updates, and implementing robust authentication mechanisms for authorized users
Monitoring of the Model– AI/ML Observability:
Monitoring and Logging: AI/ML observability companies provide monitoring and logging capabilities to track the behavior and performance of AI/ML models, infrastructure, and data pipelines. They collect and analyze various metrics, logs, and events to provide visibility into system health, resource utilization, data quality, and other relevant parameters
Performance and Efficiency Optimization: These companies help organizations optimize the performance and efficiency of their AI/ML systems. They offer tools and techniques to identify bottlenecks, inefficiencies, and resource constraints, enabling businesses to fine-tune their models, improve training and inference speeds, and optimize resource allocation
Data Quality and Drift Monitoring: AI/ML observability companies focus on monitoring data quality and detecting data drift in AI/ML pipelines. They help organizations identify missing or partial data, anomalies, concept drift, or changes in data distributions that may impact model performance. Businesses can take corrective actions and maintain model accuracy over time by monitoring data quality and drift
Anomaly Detection and Root Cause Analysis: These companies offer anomaly detection capabilities to identify unusual behavior or deviations in AI/ML systems. Analyzing metrics, logs, and other data sources can help businesses identify and diagnose issues, enabling timely intervention and root cause analysis to address problems and ensure system reliability
Model Explainability and Interpretability: AI/ML observability companies provide tools and techniques to enhance model explainability and interpretability. They offer insights into how models make predictions, feature importance, and model decision-making processes. This helps organizations understand and validate model behavior, ensure fairness and transparency, and comply with regulatory requirements
Alerting and Notifications: AI/ML observability companies offer alerting and notification mechanisms to proactively notify stakeholders about system issues, anomalies, or deviations from expected behavior. This enables timely response and action to mitigate potential problems or performance degradation
Visualization and Reporting: These companies provide visualization and reporting capabilities to present AI/ML system metrics, trends, and insights in a user-friendly and actionable format. Visualization tools help stakeholders understand the performance, behavior, and impact of AI/ML models and facilitate decision-making processes
Integration and Collaboration: AI/ML observability companies ensure seamless integration with existing AI/ML infrastructure, data pipelines, and monitoring ecosystems. They may offer APIs, integrations, and collaboration features to streamline the observability process and facilitate cross-functional collaboration between data scientists, developers, and operations teams
Security Product Suite and Sophistication
Considering the methods for ML security above, it’s worth acknowledging the overlap of types of security products and use cases that might occur. Categorically, security products within the industry often overlap, whether an endpoint, IAM, or Vulnerability management – ML security isn’t immune to this. ML Security overlaps with several distinct categories of securing ranging from authentication to endpoint protection, and in line with security in general, the tools in the space should complement the entire product suite that a CISO might be using. Suppose you’re a new company in the area. In that case, it's with considering the type of security sophistication you’re selling to, where often they’ll use 2-3 vendors, and the overlap of the product suite. Machine learning security spreads across several product suites and can be seen as an additive buy from buyers.
Based on expert interviews, ML security is needed in the “Best Practice” and “Most Advanced” security organizations. These organizations often will optimize for having one to two best-in-breed tools to ensure redundancy from a security standpoint. They will often have an annual budget for security nearing $50M and greater. If you’re an ML security company, I highly recommend targeting mature enterprises squarely within these characteristics. It’s also likely that the level of internal and customer data teams (e.g., ML, Data, DevOps) will hit a local maximum where ML security is top-of-mind, and the need to secure it is pivotal. After speaking with several CISOs and buyers, I understand that architecting a product where you can deploy it in a customer environment (e.g., VPC) will be crucial.
Industries of Relevance
Arguably, while nascent, ML Security is highly relevant to a subset of industries ranging from technology to industry. I’ve segmented the industries into three buckets: security maturity (e.g., cybersecurity maturity segments), Data Science Sophistication, and the likelihood of adopting OSS models internally. While it’s a primitive approach to understanding the industries needing ML security the most, it helps lay the foundation for who is more susceptible and, therefore, likely to purchase a new emerging competitor. Unsurprisingly, the “technology” industry seems to be a relevant target given their sophistication and likelihood of adopting open-source models. The other two industries of relevance are financial software – considering the customer data and level of sophistication needed to run real-time models for consumer-related products. The last industry of relevance is insurance. Given the nature of insurance, where the bottom line is tied to the model itself, adversaries fully acknowledge the models' importance and how the inputs can momentously affect the outputs.
Current Lay of the Land
The market is still relatively nascent; thinking through the current providers today, I bucket them into three emerging categories:
Incumbent Providers (e.g., Amazon Sagemaker, Google AI):
Providers are known to provide complete end-to-end capability from warehousing the data, security, training, and model deployment)
Designed for “Most Advanced” security segments with ancillary but necessary enterprise adjacencies such as logging
ML/AI Observability (e.g., Arize, Arthur, Fiddler)
Providers are known to be able to monitor, troubleshoot, and fine-tune models in production and pre-production environments
The initial focus is inwards and toward performance to check against responsible AI (bias), data exploration, model performance, and drifting
Less focused on security which tends to be outward in (thx Chiraag for the clear distinction!)
Emerging Competitors (e.g., Robust Intelligence, HiddenLayer, Protect.ai)
Able to protect models against adversarial attacks
Implement AI firewall and access control across the continuum of model deployment
Able to check for model validity from popular open-source mediums such as Hugging Face, et
Implement model verification procedures to continuous monitoring of the machine learning model to ensure its integrity
Go Forward, Nuanced thesis
After speaking with several experts and CISOs – ML security will be top of mind for a few enterprises, mainly within a small subset of industries. It’s unclear what the winning combination will look like, but we’re undoubtedly on the precipice of a significant shift in security. So in ending, I leave you with a list of particulars about the market and what one needs to encompass, or more or less, my thesis:
•As data continues to increase across the warehouse and collaboration layers organizations, a vendor that can isolate each part of the value chain in Zero Trust fashion (data, collaborator, model) will be favorable in the market.
•The solution must be tailored to sophisticated organizations and adopt a top-down GTM approach.
•The ideal vendor must be open to defining an MSSP approach, given the nuance of the industry and the outsourced trust earned with large organizations from MSSPs regarding their security needs.
•The ideal solution is to be deployed into customer environments architecturally (VPC) and scale across clouds.
•Targeting sophisticated enterprises, the ideal provider will need to be narrowly focused on the training data, validating, and deploying data layer – larger enterprises shy away from the one-shop-fits-all approach.
•The ideal provider will want to command average and above SaaS ACVs, as represented by some market leaders and enterprises’ propensity to pay/add new tools.
•OSS model defensibility will be critical; given the fast-moving nature of OSS, the ideal provider will want to establish a security research arm early to get ahead of new attack vectors.
•The initial wedge should be focused on ”best practice” and “most advanced” cybersecurity maturity segments.
I hope to revisit my points of view on ML security occasionally. If you think I’m dead wrong and/or missing crucial components to my thinking, please don’t hesitate to reach out so I can further stretch and refine my reflection in the space! Additionally, if you’re building something within ML/AI security, I’d love to chat with you! Feel free to drop me a note at zain [at] ridge.vc or a line on Twitter @MrRazzi17.
Thanks to Mark Dorsi, Kyle Hirari, Rob Fry, Shomik Ghosh, Chiraag Deora, Ashmeet Sidana, Alex Mackenzie, and others who have challenged my assumptions and stretched my priors for this piece!
Gartner