Deep Dive Into The Security for AI Ecosystem
Deep Dive Into The Security for AI Ecosystem
Actionable Summary
AI/LLMs have been the buzziest topic across technology. Within cybersecurity, there has been a debate about the role that AI will play in security. Broadly, there are key categorizations for the impact of AI in security. AI for security involves infusing security into existing security categories to enhance areas like automation. The second is security for AI and models in production.
This report focuses on securing Generative AI models. We discuss enterprise adoption of AI, how companies are securing these models, and the vendor landscape in this market. People must understand the difference between predictive and generative AI workloads, as they have distinct budget allocations and security needs. Within this landscape, most companies fall into one of these areas: governance, observability, or security.
2025 will be a big year for the industry, as AI experiments currently taking place in 2024 either manifest or don’t. With it, we anticipate that the market will be larger, creating room for multiple winners. If the latter, then we’ll see heavy consolidation. This market holds significant long-term potential. However, we believe the space is quickly evolving, and it's too early to determine clear market leaders.
Currently, vendors are choosing to buy instead of build. We also observed that CISOs/CIOs are not the only sole evaluators; we see an increase in wallet share from AI research and sometimes CDOs. For many companies, ACVs are hovering around $50K, all the way up to north of $200K for solutions.
The biggest issues today revolve around enterprises having control over how data goes in and out of their enterprise and how their employees access data—whether PII or data necessary for their jobs. Shadow AI is real. CISOs and teams want and need visibility into what is being used across employees. Over the long term, we see security risks revolving around models in runtime.
Core insight
AI Security is a new market with some historical patterns of existing cybersecurity markets; the market is slowly maturing based on usage. When looking back at previous major paradigm shifts for enterprises, the evolution of cloud security saw multiple generations of companies—many early entrants were acquired, and the second generation fared moderately well. Still, it wasn’t until the third generation that we saw clear winners. Similarly, in the AI space, we expect to see a sequence of security vendors emerge, each building on the lessons of their predecessors. However, unlike the cloud, the rapid pace and lower entry barriers of AI mean that we might see a more accelerated—and possibly chaotic—evolution of security solutions.
To begin to define security for AI, it is essential to understand the “how.” The way AI models are developed, sourced, and deployed has undergone a significant transformation, introducing new layers of risk. Traditionally, organizations had a straightforward choice: build models in-house, use open-source models, or rely on third-party, hosted solutions. However, the rise of foundation models—massive, pre-trained models that can be fine-tuned for specific tasks—has shifted the landscape, leading to a widespread reliance on third-party models. This reliance introduces substantial security concerns. While open-source models offer flexibility and cost savings, they may also have hidden vulnerabilities. Even with safer formats, the potential for embedding malicious code within the model architecture itself remains a significant threat.
New Paradigm Shift for The Enterprise
It’s fair to say that ever since the release of ChatGPT in 2022, the world of AI has felt like a rollercoaster that isn’t stopping anytime soon. The initial surge of enthusiasm, fueled by the promise of AI-driven productivity, has simmered down, but the momentum is far from over. While most white-collar jobs remain intact, the hype around AI has evolved, with LLMs and AI capabilities beginning to weave themselves into everyday applications. Generative AI is no longer just a buzzword—it’s becoming a fundamental part of how we create, reason, and interact with technology.
Today, AI enables us to generate images, videos, texts, and audio, supplementing human creativity in ways previously unimaginable. More importantly, AI’s ability to engage in multi-step, agentic reasoning allows it to simulate human-like decision-making processes. This evolution marks a significant shift, one that mirrors a similar transformation in the tech landscape over the past two decades—the rise of the cloud.
A Parallel to the Cloud: Learning from Past Shifts
The last major tectonic shift in technology was the advent of the cloud, which revolutionized applications, business models, and the way people interact with technology. Just as the cloud replaced traditional software with a new model of service delivery, AI is now poised to replace services with software—a change that could have exponential implications. As with any platform shift, the lessons learned from previous generations are critical in navigating the current landscape. The cloud’s evolution taught us that while early movers may set the stage, it often takes multiple generations for true winners to emerge.
This same principle applies to AI. Deploying AI/LLMs within enterprises is a paradigm shift akin to the impact of the cloud. We are likely in the early stages, where the eventual leaders in AI security are still taking shape. As enterprises wrestle with decisions about deploying open vs. closed models and as new types of cyber threats emerge, we anticipate that the landscape of AI security will undergo several iterations before the most successful players are established.
The Double-Edged Sword of Innovation
With every new technological shift comes the double-edged sword of innovation. While AI holds immense potential, it also presents new opportunities for malicious actors. The cost of launching sophisticated, nation-state-like cyber attacks is plummeting, and AI’s capabilities are lowering the barriers to entry for attackers more than the cloud ever did. This reality underscores the urgent need for robust security measures and presents a fertile ground for both startups and established companies in the cybersecurity space.
Looking back at the evolution of cloud security, we saw multiple generations of companies—many early entrants were acquired, and the second generation fared moderately well. Still, it wasn’t until the third generation that we saw clear winners. Similarly, in the AI space, we expect to see a sequence of security vendors emerge, each building on the lessons of their predecessors. However, unlike the cloud, the rapid pace and lower entry barriers of AI mean that we might see a more accelerated—and possibly chaotic—evolution of security solutions.
The Next Generation of Security for AI
The analogy with cloud security serves as a roadmap for understanding the potential trajectory of AI security. Just as economies of scale in cloud infrastructure led to the dominance of tech giants like Amazon, Microsoft, and Google, we might see similar dynamics in AI, where the winners will be those who can scale and innovate rapidly. In the case of mobile technology, the early innovator (iOS) built a closed ecosystem, while Google quickly followed with an open ecosystem (Android) that consolidated the rest of the market. AI security could see a similar dichotomy between closed and open models, with the market consolidating around the most effective approach.
As we stand at the precipice of this new era, the landscape is murky, and the path forward is still being charted. However, one thing is clear: the evolution of AI security will be a critical determinant of how enterprises adopt and deploy AI technologies. Just as cloud security eventually matured and stabilized, we expect AI security to follow a similar trajectory—one that we will explore in depth throughout this report.
Broad Categorization of AI
AI will impact security in many ways. The two primary categorizations are AI in security and Security for AI. The categorization of companies across AI varies depending on the source. In general, a core similarity exists with all the companies in these areas:
Companies that scan model repositories and dependencies for sensitive data
Companies that act as proxies that sit in the network path and inspect for data along an enterprise’s network traffic
Companies that sit at the application layer - either as extensions or embedded as code help to enforce data security and access right policies.
AI for Security
These are solutions used to automate security solutions across security operations and developers. Although it is not the focus of our article, we wanted to highlight it. It includes how we can leverage AI across a number of areas:
SOC automation: Using AI to improve task efficiency for SOC analysts.
AppSec: We can use AI to do better code reviews, manage open-source or third-party dependencies, and apply for security in applications.
Threat Intelligence: Improvement in AI can better help security teams with threat intelligence and automate manual areas of penetration testing and anomaly detection.
Security for AI
This is the focus of our article. It focuses on security AI/LLMs in production environments. A key shoutout to the team at Menlo Ventures for helping to share our knowledge here. Broadly, many of the companies in the space fall into these categories:
Governance: They help organizations build governance frameworks, risk management protocols, and policies for implementing AI.
Observability: They monitor, capture, and log data from these AI models when they have been deployed in production. They help capture the inputs and outputs to detect misuse and give teams full auditability.
Security: They implement security of the AI models - specifically identifying, preventing, and responding to cyber attacks for AI models.
AI Access Controls:
Securely access to GenAI applications and secure enterprise AI applications
Pre-model building/training of data
Data Leak Protection and data loss prevention
Identifying PII Identification and redaction
Red teaming
Model building/lifecycle management
Model vulnerability scanning and monitoring
AI Firewall and network traffic inspection
Threat detection & response
Enterprise Adoption
Before diving into the role of security for AI, it's critical we understand how enterprises are deploying them today. The promise of Gen AI is its role to drive productivity and cut costs as companies incorporate technologies like LLMs and RAG.
Despite its promises and leaders' willingness to use AI to drive better results, many enterprises are still figuring out how to use it. Some organizations are further than the rest, but many are still in the experimental mode—evaluating its pros and cons before deploying it at a large scale.
💡 Predictive vs. Generative AI. It’s important to be clear that predictive and generative AI are different; both serve different purposes. Predictive AI focuses on analyzing existing data (trained labeled datasets) to make informed future predictions or decisions, such as recommending products. In contrast, generative AI goes beyond analysis to create entirely new content, like text, images, or music, that mirrors the patterns found in its training data. It is easy for people to mix it up.
Enterprise Adoption of AI
Morgan Stanley recently published some data in the report, “US Tech: 2Q24 CIO Survey - Stable Budgets, Nervous CIOs,” showing CIOs are prioritizing initiatives around Net AI/ML projects. As of 2Q24, Artificial Intelligence (AI)/Machine Learning (ML) was #1 on the CIO priority list, increasing in net prioritization (~16% of CIOs citing it as a top priority) from 1Q24. AI/ML realized the largest QoQ increase in prioritization in their 2Q24 survey. It is important to note that they are called generative AI. Although not surprising, the MS Q2 data underscores increasing CIO fervor surrounding Generative AI technologies. Additionally, 75% of CIOs now indicate direct impacts from Generative AI/LLMs to 2024 IT investment priorities, an uptick from 73% of CIOs in 1Q24 and 68% of CIOs in 4Q23).
Interestingly, despite CIOs prioritizing AI/ML, there is a moderation for the actual implementation of these AI/ML projects within their projects. They made it clear that they were waiting for Gen-AI spending to drive software budgets higher materially. A relevant quote is important to add: “At the same time, we await an innovation-driven spending cycle (driven by Generative AI) to drive Software budgets higher, but the timing of that cycle if anything, may be getting pushed out given a lack of conviction in the reference architectures for how to build Generative AI solutions and the relative nascency of Generative AI solutions being built into packaged application offerings.”
In general, most of the data from Morgan Stanley proves our hypothesis that while most enterprises want to drive AI initiatives, it will still be a long haul until complete execution and deployment. Based on our discussions, many companies believe that it will take them until 2025 before they have large implementations and establish spend for their products.
At this point, the benefits of Gen AI are clear. However, the risks are much bigger and unclear, making large-scale deployment much slower. Boston Consulting Group's (BCG) report of 2,000 global executives last year found that more than 50% actively discouraged GenAI adoption because of a wide variety of risk factors. Some of these risks center around security, governance, regulations, and outputs of these models.
Any enterprise moving fast to deploy AI today is likely faced with how they can deploy governance protocols around managing how sensitive data flows in and out - either for the customers they serve or internally. Companies have to manage how they enforce both internal policies and external regulatory frameworks around PII. Also, the cost of training models is still expensive, and some of the model outputs still produce hallucinations.
Enterprise Approaches to GenAI Tools
As most of this report has discussed, many enterprises have either opted to block access to GenAI tools such as ChatGPTentirely by blacklisting their URL within the organization's firewall. While this approach reduces risk, it may lead to productivity losses and does not address issues related to remote workers or unmanaged devices.
Risks Associated With GenAI Tools
Unintentional Data Exposure: Employees might inadvertently expose sensitive data by pasting it into ChatGPT, leading to unintended disclosures.
Malicious Insider Activity: A malicious insider could manipulate GenAI tools by feeding it sensitive data through carefully crafted questions, enabling data exfiltration when the same questions are asked from an unmanaged device.
Risky AI-powered Browser Extensions: These extensions can exfiltrate any data from any web application your users are using.
Enterprises are using browser extension security companies like LayerX to protect their devices, identities, data, and SaaS applications from web-based threats that traditional endpoint and network solutions may not address. These threats include data leakage, credential theft through phishing, account takeovers, and the discovery and disablement of malicious browser extensions, among others.
Here, we see the role of Browser Extensions. By deploying a browser security extension, organizations can apply monitoring and governance actions within GenAI tools like ChatGPT. This includes blocking access, issuing alerts, and preventing specific actions such as pasting or filling in sensitive data fields. These controls help employees enhance productivity while safeguarding sensitive information.
Since GenAI tools are accessed via a web session, the extension can oversee and govern how users interact with it, including determining which actions are permitted based on the context.
Enterprise Adoption of AI/LLM: Open vs. Closed Source Models
As we explore how enterprises will adopt AI/LLMs in the future, it's equally important to assess whether companies will adopt open or closed models, how that impacts the future, and the role of security. There is still much debate on whether open vs. closed-source models will be the leading models on the market. The major CSPs and AI companies are investing a lot of capital to win.
Closed-sourced LLMs are leading the way.
Today, the leading models are closed-source companies like ChatGPT-OpenAI, Claude by Anthropic, and gradually BERT/Gemini by Google. As of today, closed-sourced models like OpenAI are the leading model used by enterprises globally.
The recent Emergence Capital (“Beyond Benchmarks report”) estimates OpenAI with ~70-80% market share of LLM deployments today, with Google Gemini a distant second at ~10%. Enterprises often choose closed-source models like Open-AI because of their proven stability and performance at scale. These models come with comprehensive support, regular updates, and integration services, making them ideal for mission-critical applications. There is also Claude by Anthropic, which is focused on ethical AI and is designed to be safer and more aligned with enterprises interested in ethical AI.
Closed-source models like OpenAI offer plug-and-play solutions that can be easily integrated into existing enterprise systems. This can appeal to enterprises looking for quick deployment without extensive in-house AI expertise.
Morgan Stanley’s research shows Microsoft is a significant beneficiary. They observed in their 2Q survey that 94% of CIOs indicate plans to utilize at least one of Microsoft's Generative AI products (Microsoft 365 Copilot & Azure OpenAI Services screening as the most popular) over the next 12 months, a significant uptick from 63% of CIOs in 4Q23 and 47% of CIOs in 2Q23. Looking ahead to the next three years, a similar 94% of CIOs indicated plans to use Microsoft's Generative AI products, which suggests this level of adoption should be durable.
Also, it is vital to note that Microsoft’s Security for LLMs for AI/LLM is gaining traction. According to Microsoft’s last earning call, over 1,000 paid customers have used Copilot for Security.
Open-source models
The popular ones today are LLaMA 2 by Meta and Mistral. There has been a debate about open-source models because they allow enterprises to customize and fine-tune the models to suit their specific needs. This flexibility is crucial for organizations with unique requirements or those that wish to differentiate their AI applications from competitors. The most significant benefit is that with open-source models, enterprises can deploy the models entirely within their infrastructure, ensuring complete control over data privacy and security. This is particularly important for industries with stringent data protection regulations.
Cost considerations—While the initial setup and expertise required to deploy open-source models may be higher, the long-term cost is generally lower as there are no ongoing licensing fees compared to closed-source model companies. Enterprises that have in-house AI expertise often prefer this route for its cost benefits.
The role of security - In closed vs open models
It is vital to consider the role of cybersecurity vendors in a world where either open or closed models take the most market share.
In an open-source world, cybersecurity vendors must account for the vast diversity in model deployments. Each enterprise may modify the model differently, requiring the vendor to offer adaptable and customizable security solutions. This adds complexity but also creates opportunities for niche solutions tailored to specific industries or enterprise needs. Open-source models provide full transparency into the model's architecture and code, allowing cybersecurity vendors to analyze and secure the model thoroughly. Since enterprises deploying open-source models often retain complete control over their data, vendors must focus on securing on-premises infrastructure and ensuring data privacy.
In a closed-source model world, closed-source models tend to have more standardized deployments across enterprises. This uniformity can simplify the development of security solutions. With closed-source models, vendors may have limited access to the model's internals, making it challenging to develop deeply integrated security solutions. They may have to rely on APIs or other interfaces provided by the model's owner, potentially limiting the depth of security measures that can be applied. There is also a question on whether cybersecurity vendors need to partner with Open-AI to indeed provide in-depth security or whether Open-AI would buy more in-depth security solutions.
Enterprise Risks & Attacks
The emergence of generative AI and large language models (LLMs) has played a pivotal role in bringing AI security and reliability to the forefront. As enterprises deploy AI/LLMs, they introduce a new set of risks and vulnerabilities that cyber attackers can exploit. Although these issues have existed for some time, they were often marginalized, with critics dismissing them as theoretical problems rather than practical concerns. The ambiguity surrounding the responsibility for AI security—whether it was a safety issue for security teams or an operational matter for machine learning (ML) and engineering teams—has now been resolved.
Blurring lines between the control and data plane
One of the more subtle yet profound challenges posed by AI, particularly generative models, is the convergence of the control plane and data plane within applications. In traditional software architectures, there is a clear separation between the code that controls the application and the data it processes. This separation is crucial for minimizing the risk of unauthorized access or manipulation of data.
Generative AI models blur this distinction by integrating control and data within the model itself. This integration often exposes inference processes directly to users, creating a direct link between user inputs and model outputs. Such exposure introduces new security vulnerabilities akin to the risks associated with SQL injections in conventional applications. An everyday use case is prompt injection attacks, where malicious users craft inputs to exploit model behaviors. These attacks are an emerging threat in the context of LLMs.
Moreover, the use of third-party models raises concerns about data security. Proprietary data, including sensitive information such as personally identifiable information (PII), is often fed into these models during inference, potentially exposing it to third-party providers with unknown or insufficient security practices. The risk of data leakage, whether through model providers misusing the data or through unintended outputs, underscores the critical need for rigorous data security measures in AI systems.
Classification of risks
The industry is working on standardizing the top risks and attacks, but so far, we haven’t experienced a significant breach that has redefined the industry. The most popular standards used across the industry are the OWASP Top 10 Risks for AI/LLMs or MITRE ATLAS. We’ll explore the other risks in more depth later.
Prominent risks include membership inference attacks, model extraction attacks, evasion attacks, backdoor attacks, jailbreaking attacks, and reinforcement risks. At the most basic level, the biggest problem for many enterprises is managing how their enterprise data flows in and out as employees use chatbots and access GenAI applications, especially given the 44% growth in employee usage of GenAI apps.
However, moving forward, organizations will face large-scale risks —primarily around managing autonomous agents and LLMs in production. One of the best ways to understand the risk factors with AI is by looking at the entire supply and where to develop a model. Google created the AI security framework. Similar to how we think of risks across the software supply chain, enterprises face different types of attacks across the ML supply chain.
Similarly, but at a broad level. We can evaluate the risks of ML supply chain attacks. Models can be manipulated to give biased, inaccurate, or harmful information used to create harmful content, such as malware, phishing, and propaganda used to develop deepfake images, audio, and video leveraged by any malicious activity to provide access to dangerous or illegal information.
Adversarial & Model Inversion Attacks
These are sophisticated forms of cyberattack in which the attacker subtly manipulates input data to deceive an AI model into making incorrect predictions or decisions. These attacks take advantage of the model’s sensitivity to specific features within the input data. By introducing minor, often imperceptible changes—such as altering a few pixels in an image—the attacker can cause the AI model to misclassify the data.
For example, in an image recognition system, a carefully crafted adversarial attack might result in a model incorrectly identifying a stop sign as a speed limit sign, potentially leading to dangerous real-world consequences. These types of attacks are particularly concerning in critical applications like autonomous vehicles, healthcare diagnostics, and security systems, where the reliability and accuracy of AI models are paramount. The ability of adversarial attacks to subtly undermine AI models without obvious indicators of compromise highlights the need for advanced defensive strategies, such as adversarial training and robust model evaluation techniques.
Model Inversion is an attack that enables adversaries to reconstruct sensitive input data, such as images or personal information, by analyzing an AI model's outputs. This poses serious privacy risks, particularly in applications like healthcare and finance. Protecting against this requires careful management of model exposure and output handling.
Data Poisoning
It is another significant threat to AI systems, involving the deliberate injection of malicious or misleading data into the training dataset of an AI model. The goal of this attack is to corrupt the model’s learning process, leading to biased or incorrect outcomes. For instance, if an AI model is designed to detect fraudulent transactions, an attacker could introduce poisoned data that falsely labels certain fraudulent activities as legitimate. As a result, the model may fail to detect actual fraud when deployed in a real-world environment. The danger of data poisoning lies in its ability to compromise an AI model from its foundational level. Once a model has been trained on poisoned data, the inaccuracies and biases are deeply embedded, making it challenging to identify and rectify the issue. This type of attack is particularly insidious because it can remain hidden until the model is deployed, by which time the damage may already be significant.
Model outputs
Generative models, by their very nature, can produce an almost limitless range of outputs. While this flexibility is a key strength, it also introduces a vast array of potential challenges. The evaluation for such a model is relatively straightforward, focusing on categorical correctness within a narrow scope.
In contrast, the evaluation of generative AI models is far more complex and multifaceted. It’s not sufficient to merely determine whether the model’s output is technically correct. Stakeholders must also assess the output's safety, relevance to the context, alignment with brand values, potential for toxicity, and risk of generating misleading or "hallucinated" information. This complexity makes the task of ensuring the reliability of AI systems both challenging and nuanced, requiring a broader and more sophisticated evaluation approach.
The performance evaluation of discriminative models is typically straightforward. It relies on well-established statistical metrics to quantify how accurately the model’s predictions align with known outcomes. These models, often trained on labeled datasets, allow for a clear, objective assessment of performance based on historical data.
However, generative models, such as those used in chatbots and other natural language processing (NLP) applications, present a different set of challenges. The quality and appropriateness of their outputs are not only multi-dimensional but also highly subjective. What is considered "good," "safe," or "useful" can vary significantly depending on the context, audience, and individual preferences.
Enterprise Adoption of AI Security Tools
Throughout our research, we spoke with several CISOs, CIOs, and buyers who were involved in assessing AI security solutions. We sensed that the current adoption – while still nascent – is starting to mature. The majority of the CISOs are purchasing new Generative AI solutions (increasingly) based on the following:
First, this is still a buy-compared-to-build solution, as most of the in-house AI talent is allocated to developing models or LLMs. At the same time, simultaneously, security teams need to be equipped to create a solution. This includes not only security but also teams where there are cross-functional overlaps, such as data, BI, or even new Generative AI teams.
Security aspects
Second, organizations and enterprises are trying to ensure that the employees and data are currently secure. Here, we see the traditional methodology borrowing the ZTNA framework, including DLP, CASB, and some SWG-like solutions acting as the proxy between what employees are using directly (interfacing with ChatGPT) in addition to using a SaaS-based tool that currently leverages AI (e.g., Canva). Simply, AI in/egress. When we asked these leaders how they’re deciphering the best solution among the many start-ups competing in the space, it usually is a mixture of the following:
It needs essential visibility into who is using what at all times
To ensure that no sensitive data is shared with an external AI/LLM platform
There is the basic RBAC for who has access to what (branded as Shadow AI).
Thirdly, enterprises are starting to buy AI security solutions ahead of actual need. While this sounds counter-intuitive, we heard from several public company CISOs that the board/shareholders are demanding that we have AI Guardrails and security measures in place. The analogy we like to use: It’s almost like hiring a set of bodyguards to protect a blank canvas, but the general belief is that once the canvas is painted, it’s nearly going to be a Picasso or a Mona Lisa. Relatedly, we’ve heard that more significant, more sophisticated enterprises will require companies to have an “AI-SBOM,” which is a detailed inventory or list of all components, dependencies, and elements used in creating and deploying AI or LLMs. In the context of AI, the SBOM includes the following:
Data Sources: The datasets used to train the AI model, including metadata about where the data was sourced, how it was processed, and any biases or limitations.
Algorithms and Models: The specific machine learning algorithms, models, or frameworks (e.g., TensorFlow, PyTorch) used in the development process, including versions and configurations.
Software Libraries and Dependencies: All software libraries, packages, and dependencies that the AI system relies on, including their versions.
Training Processes: Details about the training processes, including the environment, resources, and any specific hyperparameters used.
Hardware Dependencies: If the AI system requires specific hardware (e.g., GPUs), this would also be included in the SBOM.
Security and Compliance Information: Information related to security vulnerabilities, compliance with regulations, and any certifications or audits conducted on the AI system.
A subset of sophisticated customers was the first movers with respect to adopting AI. We spoke with one customer who created one of the first customer-facing chatbots using AWS and LangChain. They currently use a Generative AI security solution that sits in front of the chatbot (via API) to ensure that no sensitive information is being shared externally. In the same vein, buyers are getting smarter about evaluating the efficacy of ensuring that no sensitive information is leaked. Here, companies (which we’ll cover below) are also applying their LLMs in addition to constant fine-tuning to ensure that the efficacy is high (>90% passes POCs). At the same time, latency is low (no good metric today). Related to fine-tuning, we’re gathering from CISOs/CIOs and buyers – knowing outright that this fits in the buy category, they’re starting to ask providers how much adversarial testing has gone through their models and what results, if any, they can share. Some sectors are leveraging these providers: Government, Financial Services, Automotive, Oil/Energy, Travel, and Tech/e-commerce. Across all of the sectors, there tend to be user cases centered around model security, perimeter defense, and user security. We’ve seen the travel industry become one of the faster movers to adopt chat agents. Hence, adopting an AI security provider is imperative.
While it’s easy to be disparaging about how accurate the Chatbot use case is for the long-term of the industry, it’s a clear indication that there is an appetite for using a holistic solution that protects employees, sensitive data, and customer/internal-facing AI applications. The next clear sign of evolution for AI security is protecting agentic applications. Helping enterprises to safely develop and deploy their first non-determining computing applications that go beyond their first chatbots feels like a huge opportunity and, by far, is the most premature today. We are starting to see some companies take this leap, such as Apex Security, with their new AI agent's risk and mitigation services.
Vendor Landscape Discussion
We found Gartner’s latest report on AI Impacts on cybersecurity and the CISO to be the most comprehensive look at the relevant attack surfaces that exist in enterprises today. Using the framework from Gartner, we slotted the relevant companies securing the surface area in accordance with their segments.
Governance
Governance is a crucial first step in implementing AI security for enterprises. It establishes robust frameworks that define roles, responsibilities, and policies, ensuring compliance with regulations and ethical guidelines while promoting transparency and explainability of AI models.
IT risks in AI governance encompass both traditional vulnerabilities and new considerations introduced by generative AI, such as IP infringement and toxic outputs. The proliferation of AI regulations globally, with over 58 countries presenting their own approaches, further complicates this landscape.
Effective AI governance requires a two-pronged strategy: 1) Gain a comprehensive understanding of GenAI usage across the organization, and 2) Develop and implement well-defined policies to map and mitigate associated risks. This involves tracking AI licenses, identifying internal AI projects, and monitoring third-party AI integrations to create a secure AI ecosystem.
Implementing access rights:
Vendors focus on helping companies implement governance around access rights and controls across their SaaS apps and non-human identities. Solutions from Knostic AI, Antimatter.io, Unbound Security, and Protopia AI are crucial for ensuring proper access configurations for large language model data and maintaining robust data security.
There are several vendors within compliance and governance, including Arthur.ai, Askvera, Fairnow, Cranium, Credo AI, Holistic AI, and Knostic.
Data Privacy, Data Leaks Prevention, and Model Poisoning
Vendors: Antimatter, Gretel, Skyflow, BigID, Bedrock Security and Nightfall AI.
Enkrypt AI: They ensure that personally identifiable information (PII) is redacted before any prompt is sent to an LLM, while DAXA enables business-aware policy guardrails on the LLM responses.
Antimatter: They manage LLM Data by ensuring proper access controls, encryption, classification, and inventory management.
Protecto: They secure retrieval-augmented generation (RAG) pipelines, implement role-based access control (RBAC), and help redact PII.
Synthetic data protection: Companies like Gretel provide API support, allowing you to train GenAI models with “fake data” and validate the use cases with quality and privacy scores. Also, companies like Tonic help de-identify datasets, transforming production data to make it fit for training while maintaining a pipeline to keep the synthetic data current.
End-to-end Solutions (Build To Runtime)
Some vendors have emerged as end-to-end visibility or last-mile support companies (covering testing, evaluation, and performance monitoring), especially across multi-model environments, that will gain traction. Examples include Prompt Security + Portal26 + Lasso Security + Troj.AI + Hidden Layer + Robust Intelligence
Red teaming has emerged as a core component of when implementing these models. Companies are capitalizing on the regulatory requirements around Biden’s AI executive order and European Union Act to build red-teaming solutions. Some of these companies include Mindgard, Prompt Security, Repello AI, and SydeLabs (ProtectAI).
Model Lifecycle
Initial steps into the development of the models, in addition to deployment
HiddenLayer: The company provides detection and response capabilities for both Generative AI and predictive AI models, focusing on detecting prompt injections, adversarial attacks, and digital supply chain vulnerabilities. HiddenLayer stands out with its agent-based security for AI platforms, including its AI model scanner and AI detection and response solutions.
Radiant AI: Radiant AI sits in between the model layer and the application layer. While Radiant isn’t directly competing for security budget as they help teams identify outliers within the models for performance in addition to security, they see some pull into security due to their Model Gateway, which sits in between sandbox environments and AI models. As a result, they provide things such as alerting (with a framework) and access controls on the models themselves, which tend to be attractive for CISOs who need basic RBAC
Application Layer
Prompt Security: Prompt Security is focused on building for the three use cases: securing employees (via browser), securing developers (via Agent), and LLM applications via SDK/proxy. We see the company as one of the early leaders in the life-cycle and AI application security modules.
Protect AI: Protect, one of the first movers in the space is building an end-to-end platform, starting with the application layer and continuing with full end-to-end model development. They focus on providing Zero trust in the models themselves and host red teaming capabilities.
Mindguard: Mindgard’s core strength lies within its AI red teaming platform that identifies risks, provides standardized assessments, and offers effective mitigations for both Generative and Predictive AI systems. They help run secure AI applications with compliance and responsibility at the forefront. Their solution can enable security teams to dynamically red team any AI model hosted anywhere – on public model hubs like HuggingFace, your internal platforms, or your favorite cloud provider.
Liminal: Like many of its peers, Liminal focuses on securing employees' interactions with AI/LLMs via the browser in addition to desktop applications, explicitly helping companies deploy AI applications securely.
Swift Security: Swift Security is focused on protecting three types of use cases: LLM applications, developers using AI, and LLM applications themselves. You can deploy Swift as an inline proxy; they’re also focused on inspecting data in motion for greater security.
AI Sentry: AI Sentry is focused on securing Generative AI applications by enforcing access control for RAG, as well as for the AI applications and agents themselves.
Apex Security: Apex Security has architected the product to cover use cases ranging from data leakage (employees) to AI exploits and prompt injections, which typically happen at the application layer. The product today is currently based on an agentless architecture, and it can plug into many cloud environments quickly, which helps it bolster its detection engine.
Calypso: Calypso is focused on securing the LLM and AI models themselves rather than the employees. The company acts as a proxy via API between multiple LLMs and companies’ applications. It can deploy on-prem and in the cloud without interfering with latency and provides complete visibility.
Troj-AI: Troj.ai is a security for AI company focused on enterprises and delivered via software installed in a customer environment (on-prem or cloud). They are focused on enterprises building their own applications using any model (custom or off-the-shelf). Their solution particularly focuses on understanding the inputs and outputs coming from the model and securing them at runtime. They assess risk with models at build time (with pen-testing) and protect them in runtime by analyzing the prompts to and from the model. A core premise behind their solution is that most enterprises want their models closer to the user, and they don’t want their data to leave the enterprise.
Lasso Security: Lasso Security’s approach centers on its ability to operate between LLMs and a variety of data consumers, including internal teams, employees, software models, and external applications.
Unbound Security: Unbound is focused on accelerating enterprise adoption of AI usage through comprehensive visibility and control of AI apps. It has a browser plugin to monitor employee usage of AI apps and an AI gateway for application-based interactions.
Other players are AI Shield, Meta, Cadea, AppSOC, Credal, Encrypt AI, etc.
Observability
Businesses require comprehensive monitoring and logging of AI system activities to detect anomalies and potential security incidents in real time. Continuous monitoring of AI models is essential for identifying drifts, biases, and other issues that could compromise security and reliability. A few of the startups here include
Prompt Security, Lasso, and Helicone offer monitoring solutions. SuperAlign provides governance for all models and features in use, tracks compliance, and conducts risk assessments.
Lynxius focuses on evaluating and monitoring model performance, particularly for critical parameters like model drift and hallucinations.
Harmonic Security helps organizations track GenAI adoption and identify Shadow AI (unauthorized GenAI app usage). It prevents sensitive data leakage using a pre-trained "Data-Protection LLM." Their innovative approach engages directly with employees for remediation rather than simply alerting the security team.
Unbound Security: Solves for shadow AI by not just cataloging AI apps in use but by proactively steering employees to IT-sanctioned AI apps in each category. Unbound also performs Just-in-Time employee-led remediation when employees attempt to share sensitive information.
Future Predictions
We remain excited about the possibility of what AI security will be in the future. Compared to other “bubbles,” AI security, while still nascent in terms of actual need, prioritization for the CISO, and long-term viability, is very much a clear “buy” vs build. So, we believe that there will be concentrated winners and a handful of losers. The latter will lead to heavy M&A, and we anticipate we’ll see some of this towards the second half of FY2025. Why the second half of FY2025? By then, we’ll have clarity into real-production (customer-facing) LLM applications and whether or not they’ll be widely used. Some predict that AI will still be speculative when measuring the impact from productivity to costs.
On the consolidation front, you’ll expect to see familiar names such as PANW, Zscaler, Cisco, and other large incumbents. We believe that with the pace of innovation happening in LLMs and AIs—and with AI security—larger incumbents will be propelled to buy several start-ups in the space that have the agility to keep up with the market need and, in addition to the talent. Here, we anticipate that teams will get acquired based on their reasonable AI/ML and security talent.
Next year will be telling; we’ll get a sense of how much AI actually goes into production. Between the board-level pressure of using AI to deploying it securely, it’s a tailwind that many companies are riding; call it “first base,” and to get to “second base,” AI security companies will need to develop the most frictionless products that don’t get in the way of production, ensure no latency is introduced, and importantly, give complete visibility into what is happening both across employees and applications.
We’re early, but things are becoming more real. As for security, what an exciting time—to be living through yet another sizeable tectonic shift, which will unlock productivity and business while also introducing vulnerabilities, some of which are starting to become known, while others will be new entirely. We’re excited to see this space unfold and hopefully revisit it in the years to come to see how much we got wrong and, if we’re lucky, how much we got right.
 | A guest post by
|