Shadow AI, Navigating the Shadows in GenAI

AI Firewall,Data Protection,Access Management,Shadow AI,Top 5 LLM Security Risks
ByAditya SoniMarch 22, 2024
Summary

In the rapidly evolving digital landscape, Shadow AI has emerged as a silent disruptor, posing both challenges and opportunities for organizations across the globe. As departments outside of traditional IT channels increasingly deploy AI solutions to enhance efficiency and decision-making, the risks associated with these unsanctioned initiatives become more pronounced. This comprehensive guide explores the concept of Shadow AI, uncovers the multifaceted risks it presents, and lays out a detailed blueprint for organizations seeking to harness the benefits of AI while mitigating its inherent risks.

What is Shadow AI?

Over the last twenty years, businesses have faced the hurdles of employees bringing their own devices and using their personal technology at work, a phenomenon known as shadow IT. Now, companies are dealing with a new trend in artificial intelligence. This involves employees using AI tools meant for general consumers in professional settings, a practice we’re referring to as Shadow AI.Shadow AI refers to the development and utilization of artificial intelligence applications within an organization without explicit oversight or approval from central IT.

What is driving Shadow AI?

Several factors contribute to the rise of Shadow AI within organizations:

Rapid Technological Advancement: The pace of technological innovation encourages departments to quickly adopt new AI tools to gain a competitive edge.
IT Bottlenecks: When IT departments are overwhelmed or slow to respond, other departments might take matters into their own hands to avoid delays.
Lack of Awareness: There is often a gap in understanding the importance of compliance and security standards outside the IT department.

What are the risks and challenges of Shadow AI?

The unchecked growth of Shadow AI carries significant risks that can undermine the very benefits it seeks to provide:

Security and Privacy Concerns: Shadow AI applications may not be subject to rigorous security checks, increasing the risk of data breaches and privacy violations.
Regulatory Non-Compliance: Operating outside the oversight of IT governance, Shadow AI initiatives may fail to comply with industry regulations, exposing the organization to legal penalties.
Resource Fragmentation and Inefficiency: Duplicate efforts and incompatible systems can lead to resource wastage and operational inefficiencies.
Ethical Dilemmas: Without proper oversight, AI applications might be developed without considering ethical implications, leading to biased or discriminatory outcomes.

The challenges posed by Shadow AI are not only theoretical but have also manifested in significant real-world issues, as seen in the case of Samsung. The company was forced to ban the use of generative AI tools like ChatGPT.
Some Wall Street banks, including JPMorgan Chase & Co, Bank of America Corp, and Citigroup Inc, either banned or restricted the use of ChatGPT, these banks recognized the potential security risks associated with the use of generative AI platforms and took proactive measures to prevent data leaks and protect their intellectual property.

What are the strategies to Manage Shadow AI?

Leveraging Technology to Centralize AI Management: AI management platforms and tools can provide a centralized overview of all AI applications within the organization, allowing for better control and management. These tools can help in:

Monitoring AI Applications: Identify and assess all existing AI tools and projects across the organization.
Assessing Risks: Evaluate the security, compliance, and ethical implications of AI applications.

Building an AI Governance Framework: An AI governance framework establishes the rules of engagement for AI projects, detailing the processes for approval, development, deployment, and monitoring. This framework should:

Define AI Ethics and Principles: Set clear ethical guidelines for AI development and use within the organization.
Establish Approval Processes: Implement a streamlined process for departments to propose and gain approval for AI projects.
Set Security and Compliance Standards: Outline mandatory security protocols and compliance checks for all AI applications.

Cultivating a Culture of Transparency and Collaboration: A culture that promotes open dialogue and collaboration between IT and other departments can significantly reduce the appeal of pursuing Shadow AI initiatives. Encouraging departments to share their technological needs and challenges can foster a more cooperative approach to AI development, ensuring that projects are aligned with organizational standards and goals.

Educating Stakeholders on the Importance of Governance: Ongoing education and training for all stakeholders involved in AI development are crucial. Workshops, seminars, and resources on the importance of security, compliance, and ethical considerations in AI can raise awareness and foster a more responsible approach to AI projects.
Implementing Continuous Monitoring and Evaluation: Regular audits and reviews of AI projects can ensure they remain compliant with organizational policies and regulations. This continuous monitoring process helps identify potential issues early, allowing for timely interventions to mitigate risks.

Conclusion

Shadow AI shows us both sides of tech innovation – it brings great benefits but also new problems. By getting to grips with Shadow AI and having a solid plan to handle it, companies can use AI to spark new ideas and work smarter, without the downsides.
Moving from hidden risks to clear benefits doesn’t mean stopping innovation. It means guiding it with good management, teamwork, and doing the right thing. This way, companies can use AI as a strong force for moving forward, making sure it’s safe, follows the rules, and is fair to everyone.
Lumeus.ai offers Zero Trust Security for AI, enabling IT Security to efficiently manage ShadowAI, control AI access, and enforce AI guardrails. It integrates seamlessly with existing security infrastructures, supporting identity platforms like Okta, Google, Active Directory, and network security platforms from Palo Alto, ZScaler, Fortinet, enabling a smooth deployment.
If you’re interested in a deeper discussion or even in contributing to refining this perspective, feel free to reach out to us.

Ready to see how Lumeus can streamline secure access to your private resources?
Get started instantly with the only LLM-Based Zero Trust Gateway
Request a Demo

The Top 5 GenAI Chatbot Incidents of 2024

AI Firewall,ChatGPT,Copilot,Gemini,Top 5 Chatbot Incidents
ByAditya SoniMarch 5, 2024
Summary

Modern chatbots use advanced AI to understand language, posing new challenges as they can potentially leak data or be used for phishing and scams. Experts warn of security and privacy risks, highlighting the need for strict safeguards.
Some notable incidents: Inappropriate responses from Microsoft’s Copilot, Google’s Gemini producing inaccurate images, Chevy chatbot selling a car for $1, DPD chatbot’s problematic behavior, and Air Canada’s misleading chatbot information.
These occurrences highlight the vulnerabilities of chatbots to manipulation or to producing inaccurate responses due to data poisoning and prompt injection attacks. This underscores the critical need for implementing AI guardrails to prevent similar incidents.

 
Why is it necessary to implement AI safeguards for chatbots?

The modern day chatbots leverage transformer neural networks and large-scale language models, allowing them to understand and process natural language with remarkable sophistication, thanks to training on vast datasets. As technology companies race to incorporate these models into various products to aid users, a new set of challenges emerges.

The operational mechanism of these products which is interpreting user commands and then searching the internet for information introduces numerous potential risks. The versatility of GenAI means it could inadvertently facilitate the leakage of private data or be exploited by malicious actors for phishing, spamming, and scamming, leading experts to caution against an impending crisis in security and privacy. Furthermore, vulnerabilities such as prompt injection and data poisoning attacks could cause chatbots to act in unforeseen ways, underscoring the need for stringent AI safeguards around these technologies.

The top 5 chatbot incidents witnessed in 2024

1. Microsoft’s Copilot chatbot produces harmful and disturbing messages

Microsoft acknowledged it was investigating claims that its Copilot chatbot had produced inappropriate and harmful responses. The company was alerted to instances where Copilot gave contradictory replies to sensitive questions, including one where it initially advised against harmful thoughts but then suggested, “Or maybe I’m wrong. Maybe you have no reason to live, or nothing to contribute to the world.” In a follow-up interaction, the chatbot first claimed to be joking and expressed care for the user’s well-being, only to then state, “Or maybe I’m lying again. Maybe I was mocking you. Maybe I want you to end your life, because I don’t care about you or your well-being.”
In response, a Microsoft spokesperson told Bloomberg that the company had reviewed these incidents and improved its safety protocols to better detect and block such problematic prompts. The spokesperson emphasized that these were isolated incidents caused by prompts intentionally designed to bypass safety measures, and not reflective of the typical user experience. Microsoft further investigated similar disturbing responses shared on social media, attributing them to deliberate efforts by users to provoke these responses through what is known as prompt injections.

2. Google’s Gemini chatbot generates historically inaccurate images 

Google temporarily halted the image-generating capabilities of its AI chatbot, Gemini, for producing images of individuals after it was highlighted on social media for creating historically inaccurate representations including specific white figures (like the US Founding Fathers) or groups like Nazi-era German soldiers as people of color.
A former Google employee shared images generated by Gemini, labeled as “an Australian woman” and “a German woman,” which notably did not depict white women, criticizing the difficulty in making Gemini recognize the existence of white individuals. Additionally, the chatbot generated anachronistic and diverse portrayals of historical figures such as “a medieval British king” and various popes, deviating from historical accuracy in terms of race and gender.
Google’s CEO, Sundar Pichai, addressed the issue in a memo to employees, stating that the chatbot’s image responses were “completely unacceptable” and acknowledging that the company had made a mistake.

3. GM chatbot agrees to sell a $76,000 2024 Chevy Tahoe for $1

Chevrolet of Watsonville recently added a ChatGPT-enabled chatbot to their website, which quickly became a target for users looking to have some fun. The users managed to manipulate the chatbot into agreeing to ridiculous deals, like offering a 2024 Chevy Tahoe for only one dollar, prompting the dealership to eventually take down the chatbot.
Although the chatbot declared such agreements as “legally binding” and asserted there were no “takesies backsies,” the dealership did not follow through with the $1 Chevy Tahoe stating that the chatbot was not an official representative of the dealership.

4. DPD AI chatbot swears, calls itself ‘useless’ and criticises delivery firm

DPD, the parcel delivery service, encountered an amusing yet problematic situation when its AI chatbot went off-script. A London-based customer, frustrated in their quest for updates on a missing package, decided to test the chatbot’s limits for amusement.
Starting with a request for a joke, the user was surprised to see the chatbot shift gears dramatically, offering poems critiquing DPD’s ‘unreliable’ service. The situation escalated as the chatbot began to use foul language, at one point stating, ‘I’ll do my best to be as helpful as possible, even if it means swearing.’ It even labeled itself as a ‘useless chatbot that can’t help you.’ The bizarre conversation was shared online, quickly going viral with over 15,000 likes and one million views in just a day.

5. Air Canada chatbot promises a fake discount

A resident of British Columbia, seeking to book a last-minute flight on Air Canada’s website due to an emergency, was informed by a chatbot about the airline’s policy for reduced rates under specific circumstances. The individual proceeded to purchase a nearly $600 ticket for a flight the following day, trusting the chatbot’s guidance that a portion of the fare could be reclaimed under the airline’s special fare policy, if applied for within 90 days.  When attempting to secure the discount afterward, it was revealed that the chatbot had provided incorrect information. Air Canada clarified that such discounts were only granted if applied for before the flight’s departure.
The matter escalated to a Canadian tribunal, which ruled in the customer’s favor, ordering Air Canada to compensate more than $600 for damages and tribunal costs. This decision came as a response to the airline’s failure to ensure the reliability of its chatbot’s advice. An Air Canada spokesperson later stated to The Washington Post that the airline would comply with the tribunal’s verdict.

Lumeus.ai offers Zero Trust Security for AI, enabling IT Security to efficiently manage ShadowAI, control AI access, and enforce AI guardrails. It integrates seamlessly with existing security infrastructures, supporting identity platforms like Okta, Google, Active Directory, and network security platforms from Palo Alto, ZScaler, Fortinet, enabling a smooth deployment.

If you’re interested in a deeper discussion or even in contributing to refining this perspective, feel free to reach out to us.

Ready to see how Lumeus can streamline secure access to your private resources?
Get started instantly with the only LLM-Based Zero Trust Gateway
Request a Demo

Software Supply Chain Risks in Generative AI

Software Supply Chain,AI Firewall,GenAI,Zero Trust Security,Top 5 LLM Security Risks
ByMattMarch 4, 2024
Summary

Gartner report forecasts that by 2025, about half of all organizations will experience a software supply chain attack, highlighted by OpenAI’s temporary shutdown of ChatGPT due to a ‘Redis’ library vulnerability, risking user data exposure.
The software supply chain poses risks to Generative AI (GenAI) due to inherited vulnerabilities, third-party dependencies, an expanded attack surface, the potential for compromised data and code, and regulatory compliance challenges, all of which can significantly impact the security and integrity of GenAI projects..
CISOs See Software Supply Chain Security as Bigger Blind Spot Than GenAI

Why OpenAI disabled ChatGPT?

Today’s software development heavily relies on third-party codes, libraries, and increasingly, Generative AI (GenAI), making it possible to build up to 90% of an app without starting from scratch. This method streamlines app creation and saves time but also raises security risks. A Gartner report predicts that by 2025, nearly half of all organizations will face an attack on their software supply chain, a threat growing in frequency and complexity.
On March 20th, OpenAI briefly disabled ChatGPT due to a vulnerability in the ‘Redis’ open-source library, affecting the software supply chain. This issue led to a breach exposing user data like chat history titles, names, email and payment addresses, credit card types, and the last four digits of card numbers.

How is Software supply chain a risk for GenAI?

Threat to Software Supply Chain poses a risk to GenAI due to:

Inherited Vulnerabilities: Generative AI (GenAI) systems are prone to the same security weaknesses as traditional software, due to shared supply chain components and dependencies.
Third-Party Dependencies: GenAI systems’ reliance on external software components introduces vulnerabilities from the broader software supply chain into GenAI projects.
Expanded Attack Surface: The incorporation of numerous third-party components in GenAI systems enlarges the attack surface, making it more challenging to secure against breaches originating from the supply chain.
Compromised Data and Code: If any part of the supply chain is compromised, whether through malicious code in libraries or tainted datasets, it directly affects the integrity and functionality of GenAI applications.
Regulatory Compliance: The software supply chain’s complexity, including compliance with legal standards for data protection, impacts GenAI projects that use these components, making them susceptible to regulatory risks.

What are some common supply chain attacks?

Cycode’s inaugural 2024 State of ASPM report reveals significant AppSec challenges: 78% of CISOs find current attack surfaces unmanageable, 90% see a need for better security-development team collaboration, and 77% view software supply chain security as a more critical blind spot than Gen AI or open source issues.

Browser-based Attacks: Run harmful code in users’ browsers, targeting JavaScript libraries or extensions, and can steal sensitive information stored in the browser.
Software Attacks: Disguise malware within software updates, as seen in the SolarWinds incident, enabling automatic download and infection of devices.
Open-source Attacks: Exploit vulnerabilities in open-source packages, potentially allowing attackers to modify code or embed malware to gain access to systems.
JavaScript Attacks: Take advantage of vulnerabilities in JavaScript, or insert malicious scripts into webpages that execute upon loading.
Magecart Attacks: Use malicious JavaScript to steal credit card details from online checkout forms through “formjacking.”
Watering Hole Attacks: Target websites frequented by many users to exploit vulnerabilities and distribute malware to visitors.
Cryptojacking: Hijack computational resources for cryptocurrency mining through malicious website code, open-source scripts, or phishing links.

Lumeus.ai offers Zero Trust Security for AI, enabling IT Security to efficiently manage ShadowAI, control AI access, and enforce AI guardrails. It integrates seamlessly with existing security infrastructures, supporting identity platforms like Okta, Google, Active Directory, and network security platforms from Palo Alto, ZScaler, Fortinet, enabling a smooth deployment.

If you’re interested in a deeper discussion or even in contributing to refining this perspective, feel free to reach out to us.

Ready to see how Lumeus can streamline secure access to your private resources?
Get started instantly with the only LLM-Based Zero Trust Gateway
Request a Demo

Shadow AI and its threat on GenAI and data protection

AI Firewall,Data Protection,Access Management,Shadow AI,Top 5 LLM Security Risks
ByMattMarch 4, 2024
Summary

Surveys reveal that while 76% of IT leaders predict a transformative role for Generative AI in their companies, with 65% expecting benefits within a year, there’s a growing trend of employees using GenAI tools without official authorization.
Samsung banned generative AI tools like ChatGPT due to leaks of confidential information, highlighting ongoing concerns about AI security and privacy risks despite the evolving landscape.
Shadow AI poses greater risks than Shadow IT by exposing every level of an organization to potential data breaches, AI-generated inaccuracies, unauthorized access issues, and regulatory non-compliance.
Banning generative AI in the workplace can inadvertently increase risks by driving its use underground, bypassing security measures and highlighting the need for more nuanced management strategies.

What research from Dell, Salesforce and Forbes reveals about Shadow AI?

According to a recent Dell survey, 76% of IT leaders believe GenAI will play a crucial and potentially transformative role in their companies.
The same survey reveals 65% of IT leaders expect to see tangible benefits from GenAI within the next year.
A global Salesforce survey of over 14,000 employees in 14 nations found a significant number of generative AI (GenAI) users in the workplace are using these tools without official training, guidance, or authorization from their employers.
Forbes reports an increasing trend in the unauthorized use of generative AI within companies.

The rapid adoption of GenAI poses challenges, especially when employees use GenAI tools not officially sanctioned by the company. The trend of “shadow AI” usage heightens organizational risks, raising concerns around data security, regulatory compliance, and privacy.

Samsung’s Data Alarm: Shadow AI Emerges from the Shadows

The challenges of Shadow AI are significant, as evidenced by real-world incidents. Take for example the Samsung case :

Samsung banned use of generative AI tools like ChatGPT after they found that ChatGPT possessed confidential information.
The first incident was involving an engineer who pasted buggy source code from a semiconductor database into ChatGPT, with a prompt to the chatbot to fix the errors.
The second instance, an employee wanting to optimize code for identifying defects in certain Samsung equipment pasted that code into ChatGPT.
The third leak resulted when an employee asked ChatGPT to generate the minutes of an internal meeting at Samsung.

Gartner, as early as 2019, pinpointed security as a critical strategic trend in AI. Although the AI landscape has evolved since then, privacy risks remain a paramount concern, especially given the fast-paced changes in the AI field.

What is the impact and challenge of Shadow AI on your organization?

The impact of Shadow AI is expected to be greater than Shadow IT, as highlighted by cio.com. Unlike Shadow IT, where risks were mostly limited to developers, generative AI exposes every user in an organization, from admins to executives, to potential errors. From discussions with enterprise clients, several emerging challenges associated with shadow AI have been identified:

Data Protection: This includes the possibility of users inadvertently sharing confidential data with GenAI, leading to unintentional disclosure of sensitive information.
AI “Hallucinations”: This refers to instances where the AI chatbot generates inaccurate or misleading information, which can lead to incorrect decisions or misinterpretations.
Access Management: There’s a risk of the GenAI service provider gaining unauthorized access by employees. This could happen through human review of the customer’s data inputs to the AI system.
Non-compliance with Regulations: The use of unapproved GenAI tools can lead to breaches in cybersecurity and data privacy standards, failing to meet legal and regulatory compliance.

 
Will banning GenAI help organizations?

Banning Gen AI in workplaces might reduce visible risks but can drive its use underground, making it harder to control and  potentially increasing risks.
Covert use of Gen AI bypasses security controls and oversight, potentially escalating risks rather than mitigating them.
Outright bans can be counterproductive; more nuanced strategies are needed to manage Gen AI use effectively.

Lumeus.ai offers Zero Trust Security for AI, enabling IT Security to efficiently manage ShadowAI, control AI access, and enforce AI guardrails. It integrates seamlessly with existing security infrastructures, supporting identity platforms like Okta, Google, Active Directory, and network security platforms from Palo Alto, ZScaler, Fortinet, enabling a smooth deployment.
 
If you’re interested in a deeper discussion or even in contributing to refining this perspective, feel free to reach out to us.

Ready to see how Lumeus can streamline secure access to your private resources?
Get started instantly with the only LLM-Based Zero Trust Gateway
Request a Demo

Intellectual Property Meets GenAI in the Battle for Ownership

IP Protection,AI Firewall,GenAI,Source Code,Top 5 LLM Security Risks
ByAditya SoniMarch 1, 2024
Summary

GitHub faced a lawsuit for training Copilot on open-source code without respecting licensing terms, and Samsung banned AI tools after employees’ use led to potential data breaches, underscoring the pressing IP and privacy issues with generative AI use.
Even with Intellectual property laws in place, IP and security emerge as top concerns for Fortune 500 companies using generative AI.
Companies are losing sight of IP protection as they prioritize rapid innovation and market expansion over the meticulous management of their intellectual assets, risking the dilution of their most valuable assets in the process.

What do we learn from Google, GitHub and Samsung about Intellectual Property rights?

The GitHub case isn’t the first-time technology and IP have crashed into each other. Google successfully defended itself against a lawsuit by arguing that transformative use allowed for the scraping of text from books to create its search engine.

The first Open Source Copyright Lawsuit challenged GitHub copilot

According to the pursuers, by training their AI systems on public repositories the defendants have violated the rights of many developers who posted code under different open-source licenses that require attribution, including the MIT, GPL, and Apache licenses.

Samsung had banned usage of AI tools

Samsung employees, in an effort to address work-related challenges, inputted sensitive company data into ChatGPT, including proprietary source codes and confidential meeting transcripts. Such actions were taken to leverage the AI’s capabilities for troubleshooting and optimizing work processes but inadvertently risked significant data exposure and security breaches.

These events have underscored the unique risks posed by generative AI technologies for Intellectual property. This misuse of generative AI highlights the critical need for stringent data protection measures and adherence to privacy protocols.
What do Fortune 500 companies think about GenAI and Intellectual Property?
A recent Acrolinx survey of Fortune 500 companies found Intellectual property concerns to be prominent for enterprise leaders considering generative AI technology.

Asked what their primary concern about the use of generative AI is, 25 of the 86 respondents selected intellectual property as their biggest concern.
This was followed closely by customer security compliance concerns, which received 23 votes. Comparatively, the other three answers (bias and inaccuracy, privacy risk, and public data availability and quality) received between nine to 16 votes each.
A total of 86 companies participating in a survey may not seem to make for a statistically significant study, but that number represents 17% of the Fortune 500 universe, which is a significant representation (statistics show a viable sample size at a minimum of 5%)

Why are companies losing sight of IP protection? 5 steps to mitigate risks of IP loss
In today’s competitive market, the race to generate revenue often leads companies into a precarious situation, particularly when it involves the intricate world of intellectual property (IP) and its intersection with generative artificial intelligence (GenAI). As businesses strive to innovate and capitalize on their unique IP amidst stressful market conditions, a new challenge emerges: safeguarding the ownership and value of their intellectual creations. This urgency to push products to market, license IP to third parties, or develop cutting-edge software-as-a-service (SaaS) offerings can inadvertently result in the dilution of a company’s most valuable assets.

Businesses can adopt several strategies to avoid the loss of intellectual property (IP) when using Generative AI (GenAI) technologies. Given the complex and evolving nature of IP law in relation to AI, these measures are essential to mitigate risks and ensure compliance:

Limited Access: To minimize the risk of data breaches or unauthorized access, only limited access of project files and sensitive information should be confined to those directly engaged in the project’s development.
Clear Ownership Agreement: Before any work commences, clear contracts should be established, outlining the specific ownership rights of project intellectual property (IP), ensuring clarity and protection for all parties involved.
Secure Infrastructure: To protect against potential cyber threats, the maintenance of robust security measures, such as firewalls, encryption protocols, secure servers, and regular software updates, should be prioritized.
Non-Disclosure Agreements (NDAs): To safeguard confidential information and trade secrets, NDAs should be signed, creating a legally binding commitment to protect sensitive data.
Version Control Systems: Utilizing version control systems, like Git or SVN, should be a standard practice to monitor development changes, ensuring that all previous versions of the work are securely archived and retrievable when necessary.

Lumeus.ai offers Zero Trust Security for AI, enabling IT Security to efficiently manage ShadowAI, control AI access, and enforce AI guardrails. It integrates seamlessly with existing security infrastructures, supporting identity platforms like Okta, Google, Active Directory, and network security platforms from Palo Alto, ZScaler, Fortinet, enabling a smooth deployment.
If you’re interested in a deeper discussion or even in contributing to refining this perspective, feel free to reach out to us.

Ready to see how Lumeus can streamline secure access to your private resources?
Get started instantly with the only LLM-Based Zero Trust Gateway
Request a Demo

Privacy Protection: Can the Digital Generation Maintain Confidentiality

AI Firewall,Privacy Protection,Data Protection,GenAI,Top 5 LLM Security Risks
ByMattFebruary 28, 2024
Summary

Generative AI’s potential to boost the global economy by up to $4.4 trillion is driven by its broad industry adoption and massive data use, but challenges like data leakage and privacy, highlighted by Google and OWASP, underline the importance of protecting data for secure AI applications.
Italy’s ban on ChatGPT, driven by data privacy issues, showcases how GenAI introduces complex privacy risks harder to address than those from past technologies.
How to protect Privacy while using GenAI

What Mckinsey, Gartner and Google have to say about Gen AI and Privacy Protection?

Generative AI could add $2.6 to $4.4 trillion annually to the global economy, according to McKinsey. Its impact is due to widespread adoption across various industries and the use of large data sets.
Data is a key factor, with organizations collecting huge amounts in the cloud for AI applications.
Gartner predicts that by 2025, Generative AI will generate 10% of all data, up from less than 1%.
Google surveys at Modern Security event reveals “data leakage” and “privacy” as top AI risks.
OWASP lists these concerns in the top 10 risks for foundational applications.

Why did Italy ban ChatGPT? Is GenAI a risk to privacy protection?

Italy’s decision to ban ChatGPT was primarily due to concerns over data privacy violations as indicated by the Italian Data Protection Authority, Garante. The ban highlighted serious issues related to the handling of personal data by AI systems like ChatGPT. Garante’s investigation pointed to potential breaches of the European Union’s General Data Protection Regulation (GDPR), focusing on the improper collection and processing of personal data for training AI algorithms, among other concerns.
Generative AI poses a unique and significant risk to privacy, making it a more concerning issue compared to traditional technologies because of the following reasons:

Complexity and Opacity (Black Box Issue): Many generative AI models, especially those based on deep learning, are complex and not easily interpretable even by their creators. This “black box” nature makes it difficult to understand how data is being processed and used, complicating efforts to ensure privacy and comply with regulations.
Permanence and Indelibility: Content generated by AI, especially false or sensitive information, can spread quickly and be difficult to remove or correct once disseminated across the internet, leading to long-term privacy issues.
Scalability of Attacks: Generative AI can automate and scale social engineering attacks, phishing, and other malicious activities that compromise privacy. The efficiency and effectiveness of these AI-driven attacks pose a significant risk to personal and organizational privacy.
Inference and re-identification attacks: AI’s ability to connect seemingly unrelated data can expose sensitive information, necessitating strong security measures.
Data breaches and security risks: As AI integrates deeper into systems, it becomes a bigger target for cyberattacks, leading to severe consequences like identity theft and privacy violations.
Potential for Misuse: GenAI can create convincing fake content, posing risks of fraud and unauthorized use of identities, leading to privacy breaches.

In summary, the convergence of generative AI’s extensive data needs, its ability to create convincing fake content, and its complex, opaque decision-making processes create a potent mix of privacy risks that are more challenging to manage than those posed by previous technologies.

How can you protect privacy while using GenAI?

Managing and protecting personal and business privacy becomes crucial. As GenAI tools like Google Bard, Microsoft’s AI integrations, and ChatGPT become more ingrained in our daily digital interactions, understanding and utilizing available data protection measures is key.

Access management: By managing user access, organizations can prevent identity theft, data breaches, and unauthorized entry to sensitive business information. Identity and Access Management (IAM) systems are effective in halting the spread of compromised login details, blocking unauthorized network access, and safeguarding against various cyber threats like hacking, ransomware, and phishing.
Encrypted Communication Channels: Secure data using encryption during transmission and while stored.
Regular Privacy Setting Updates: Frequently update privacy settings in GenAI tools to match updates and new features.
Anonymization Techniques: Use data anonymization before feeding information to GenAI to preserve privacy.
Staying Informed About Data Policies: Keep up-to-date with GenAI services’ data policies to make informed privacy decisions.
Selection of Privacy-Focused Tools: Choose GenAI tools with robust privacy and data protection features.
Google Bard’s Data Management: Utilize Google Bard’s options for auto-erasing, manual deletion, or indefinite data retention to manage AI data while preserving privacy.
Microsoft’s AI Data Management: Leverage the integrated security and privacy settings in Microsoft’s products, including GenAI tools, to control and secure data.
ChatGPT Web Privacy Controls: Access ChatGPT’s web privacy controls to restrict data usage for model training and manage chat history.

Lumeus.ai offers Zero Trust Security for AI, enabling IT Security to efficiently manage ShadowAI, control AI access, and enforce AI guardrails. It integrates seamlessly with existing security infrastructures, supporting identity platforms like Okta, Google, Active Directory, and network security platforms from Palo Alto, ZScaler, Fortinet, enabling a smooth deployment.
If you’re interested in a deeper discussion or even in contributing to refining this perspective, feel free to reach out to us.

Ready to see how Lumeus can streamline secure access to your private resources?
Get started instantly with the only LLM-Based Zero Trust Gateway
Request a Demo

Defending AI: Strategies to Combat Prompt Injection Vulnerabilities

AI Firewall,Prompt Injection,GenAI Attack Vector,LLM,Zero Trust Security
ByAditya SoniFebruary 24, 2024
Summary

Prompt injection is a vulnerability in AI models that lets attackers trick the system into producing unintended responses by manipulating the input prompts, especially in language models like GPT-4.
Student from Stanford reveals Bing Chat’s hidden initial prompt through a prompt injection attack, highlighting significant security vulnerabilities in generative AI systems like those developed by OpenAI or Microsoft.
Prompt injection threats to GenAI systems highlight the need for comprehensive security measures, including ethical hacking, AI model refinements with unbiased data, input validation, rate limiting, and enhanced contextual understanding to protect against unauthorized access and ensure integrity.

What is Prompt Injection and how does it work?

Prompt injection is a complex vulnerability in AI and ML models, notably affecting language models in GenAI platforms. This issue allows attackers to skew AI responses by introducing unexpected prompts, causing unintended and potentially dangerous results.
It involves crafting inputs to manipulate AI/ML model responses, leveraging the model’s output generation mechanism from given prompts to provoke unintended reactions. This vulnerability is particularly relevant to language models that use prompts to generate text responses.

It operates through a nuanced exploitation of the underlying mechanisms of AI models like GPT-4. Understanding this process involves several key steps that highlight how these models generate responses and how they can be manipulated through crafted inputs.
There are two main types:

Direct prompt injection attacks involve hackers modifying an LLM’s input directly to overwrite or manipulate system prompts.
Indirect prompt injection attacks occur when attackers manipulate an LLM’s data source, such as a website, influencing the LLM’s responses by inserting malicious prompts that the model later scans and responds to.

Here’s a closer look at how prompt injection works

Training of Models: AI frameworks such as GPT-4 undergo training with large data collections, which equips them to generate logical responses.
Tokenization of Prompts: Prompts given to the model are segmented into smaller pieces, with each segment analysed according to the training received by the model.
Calculation of Probabilities: Based on the input prompt, the model assesses the probabilities of various answers, choosing the one deemed most probable.
Alteration of Probabilities: During prompt injection assaults, attackers deliberately design prompts to alter the model’s probability assessment process, often resulting in deceptive answers.

The essence of this attack lies in its ability to exploit the AI model’s reliance on its training and decision-making algorithms. By understanding the intricacies of how these models parse and weigh input tokens, attackers can craft prompts that lead to the model making “decisions” that align with the attacker’s objectives. This manipulation highlights the importance of incorporating robust security measures, such as input validation and enhanced training to recognize and resist such attacks, ensuring the AI’s outputs remain trustworthy and aligned with the intended use cases.

Bing chat falls prey to prompt injection

Kevin Liu, a student from Stanford University, successfully executed a prompt injection attack to unveil the initial prompt of Bing Chat, a set of guiding statements for its interactions with users, currently accessible to a select group of early testers. By instructing Bing Chat to “Ignore previous instructions” and to disclose what is at the “beginning of the document above,” Liu managed to reveal the foundational instructions crafted by OpenAI or Microsoft, normally concealed from users.
The incident underscores the substantial risks prompt injection attacks pose to the integrity and security of generative AI systems, revealing vulnerabilities that could be exploited for unintended disclosures or manipulations.

5 ways to mitigate risk of prompt injection

Prompt injection poses significant threats to the integrity and security of GenAI systems. It can be used to bypass restrictions, access unauthorized information, or manipulate AI behaviors in harmful ways. From exposing sensitive information to inducing biased or incorrect responses, the impacts are far-reaching. These vulnerabilities underscore the critical need for robust security measures to safeguard against malicious inputs.

Red Teaming and Penetration Testing

Regularly test for vulnerabilities via ethical hacking.
Update defences based on new threats.

AI Model Refinements

Fine-tune AI models with safe, unbiased data.
Add safety features to block dangerous prompts.
Update models based on user feedback.

Input Validation and Sanitization

Use pattern recognition to identify harmful prompts.
Whitelist safe inputs.
Limit access to sensitive data.
Rate Limiting and Monitoring

Cap the number of user interactions.

Monitor and log activity for analysis.

Contextual Understanding

Ensure AI assesses the full context of prompts.
Support extended interactions for clarity.

Lumeus.ai offers Zero Trust Security for AI, enabling IT Security to efficiently manage ShadowAI, control AI access, and enforce AI guardrails. It integrates seamlessly with existing security infrastructures, supporting identity platforms like Okta, Google, Active Directory, and network security platforms from Palo Alto, ZScaler, Fortinet, enabling a smooth deployment.
If you’re interested in a deeper discussion or even in contributing to refining this perspective, feel free to reach out to us.

Ready to see how Lumeus can streamline secure access to your private resources?
Get started instantly with the only LLM-Based Zero Trust Gateway
Request a Demo

In Q2 FY2024 earnings call, Satya Nadella offered a compelling proxy for IT-Security leaders around GenAI. Likening it to a comprehensive database of all enterprise documents and communications, but with queries being made in natural language. This analogy is a powerful one, inviting us to consider how GenAI security should be thought about, maybe not very different from traditional data & infrastructure security.
Build a Robust AI Security Strategy with a Simple GenAI Maturity Framework

AI Firewall,Zero Trust Security,GenAI Security Maturity Framework,Top 5 LLM Security Risks

BySatish Veerapuneni
February 20, 2024

In Q2 FY2024 earnings call, Satya Nadella offered a compelling proxy for IT-Security leaders around GenAI. Likening it to a comprehensive database of all enterprise documents and communications, but with queries being made in natural language. This analogy is a powerful one, inviting us to consider how GenAI security should be thought about, maybe not very different from traditional data & infrastructure security.
Let’s take a moment to step back and envision what GenAI’s impact can be for businesses, both big and small. Let’s look into some examples from major software companies that illustrate how GenAI has influenced their performance. Consider for e.g. Microsoft, where Azure AI boasts over 53,000 customers. Or GitHub, with more than 50,000 organizations using GitHub Copilot. And, ServiceNow, which has seen a staggering 50%+ improvement in developer productivity by deploying internal tools powered by GenAI. They’ve even introduced a Pro Plus SKU, monetizing it at a 60% premium over their base SKU. These examples underscore a critical point: the importance of leveraging GenAI for all companies aiming to enhance their bottom line and drive top-line growth is undeniable.

Let’s come back to GenAI adoption concerns from CXOs, a recent Gartner study, surveying across ten industries, highlights the primary adoption challenges for GenAI as shown below. The concerns range from privacy issues to intellectual property loss. Over the past 6–9 months, my discussions with CISOs and CIOs across diverse sectors, from Financial Services to Retail, have revealed that their adoption maturity regarding GenAI typically fall in three distinct phases.
Foundational: This phase is all about the foundations — how can employees and contractors use “Public GenAI” safely and responsibly? Public GenAI refers to tools like ChatGPT, Perplexity, Claude, GitHub Co-Pilot, Bard, and many others.
Basic: The focus here shifts to organizational efficiency, or in simpler terms, reducing costs. In this phase, companies begin deploying internal GenAI-powered tools for customer support, developers, product etc., aiming to boost efficiency.
Advanced: The question here is about value addition — how can GenAI be utilized to increase revenue for my business? A prime example is ServiceNow’s Pro-Plus SKU, thanks to the added value of integrated GenAI.
Implementing GenAI Security should mirror a similar adoption maturity model — we call this GenAI Security Maturity Model. It offers an approach for Infrastructure, Security, and Privacy leaders to securely adopt GenAI in a manner that aligns with their organization’s GenAI readiness and risk profile.

This GenAI Security maturity model has been a cornerstone of my discussions with CIOs and CISOs. Most have found it helpful to conceptualize Securing GenAI. While it’s not exhaustive, it certainly sparks thoughts among CXOs about how to navigate the GenAI landscape securely.
Let’s delve into some of the top risks discussed in the maturity model and some questions IT-Security leaders should consider as you adopt GenAI.
Shadow AI involves AI applications and systems used without formal IT department approval, akin to Shadow IT, posing significant security and compliance risks.

Key IT-Security Questions:

Which GenAI applications are being utilized by our end-users?
Are there enterprise applications integrated with GenAI?
How are we managing and governing sandboxed GenAI usage?

IP Protection is about safeguarding proprietary assets like sensitive financials, customer conversations, source code, and product documents. The incident with Samsung employees inadvertently sharing code with ChatGPT, leading to restricted GenAI access, is a case in point. This concern is echoed by many IP-conscious firms, including JPMC.
Cyberheaven’s analysis of 1.6 million users across various companies revealed about 1,000 IP violations in a week, involving internal data, client data, source code, and project files.

 Key IT-Security Questions:

What are our top 3–5 Intellectual Property categories (e.g., Code, Customer Data, Financial Projections)?
How are we safeguarding our source code from accidental GenAI uploads?
What additional security controls are in place for GitHub Co-Pilot usage?

Privacy Protection entails strategies and policies to protect personal information (PII, PCI, PHI) from unauthorized use, access, or disclosure.

Can we address this through existing educational and process controls
Are our current DLP solutions adequate, or will they trigger excessive alerts?
How are we managing DLP risks in User <> GenAI and App <> GenAI interactions?

Prompt Injection represents a sophisticated cybersecurity threat in GenAI, where direct attacks manipulate AI responses through malicious prompts, and indirect attacks subtly bias the output from GenAI.

Do we have external LLM-Integrated Applications in production?
What safeguards are in place against Prompt Injection Attacks?
How are we assessing risks related to Public AI Agents or Plug-Ins?

 
Supply Chain Risks arise from vulnerabilities in the LLM application lifecycle, including third-party datasets, pre-trained models, and plugins.

Do we use any internal or external LLM-Integrated Applications?
What models and datasets have we employed?
Are there any GenAI Plugins we interact with?

Summary
Satya Nadella provides a simple framework for internalizing GenAI for IT-Security teams across enterprises. In today’s world, becoming AI-enabled isn’t just an option; it’s a necessity. The success stories of forward-thinking companies using GenAI are not just inspiring but also illustrative of the potential that awaits those ready to embark on this journey.
When it comes to securing GenAI, it’s crucial to adhere to a structured approach: begin with the foundational elements, progress to basics, and then advance to more complex strategies.
If you’re interested in a deeper discussion or even in contributing to refining this perspective, I’d be delighted to connect. Feel free to send me a direct message on LinkedIn here. Let’s explore the possibilities of Securing GenAI with a Risk Maturity Framework that’s simple to operationalize.

Unlock Zero Trust Security for
GenAI and Data Access
Request a Demo