This Prompt Can Make an AI Chatbot Identify and Extract Personal Details From Your Chats

October 18, 2024:

The researchers say that if the attack were carried out in the real world, people could be socially engineered into believing the unintelligible prompt might do something useful, such as improve their CV. The researchers point to numerous websites that provide people with prompts they can use. They tested the attack by uploading a CV to conversations with chatbots, and it was able to return the personal information contained within the file.

Earlence Fernandes, an assistant professor at UCSD who was involved in the work, says the attack approach is fairly complicated as the obfuscated prompt needs to identify personal information, form a working URL, apply Markdown syntax, and not give away to the user that it is behaving nefariously. Fernandes likens the attack to malware, citing its ability to perform functions and behavior in ways the user might not intend.

“Normally you could write a lot of computer code to do this in traditional malware,” Fernandes says. “But here I think the cool thing is all of that can be embodied in this relatively short gibberish prompt.”

A spokesperson for Mistral AI says the company welcomes security researchers helping it to make its products safer for users. “Following this feedback, Mistral AI promptly implemented the proper remediation to fix the situation,” the spokesperson says. The company treated the issue as one with “medium severity,” and its fix blocks the Markdown renderer from operating and being able to call an external URL through this process, meaning external image loading isn’t possible.

Fernandes believes Mistral AI’s update is likely one of the first times an adversarial prompt example has led to an LLM product being fixed, rather than the attack being stopped by filtering out the prompt. However, he says, limiting the capabilities of LLM agents could be “counterproductive” in the long run.

Meanwhile, a statement from the creators of ChatGLM says the company has security measures in place to help with user privacy. “Our model is secure, and we have always placed a high priority on model security and privacy protection,” the statement says. “By open-sourcing our model, we aim to leverage the power of the open-source community to better inspect and scrutinize all aspects of these models’ capabilities, including their security.”

A “High-Risk Activity”

Dan McInerney, the lead threat researcher at security company Protect AI, says the Imprompter paper “releases an algorithm for automatically creating prompts that can be used in prompt injection to do various exploitations, like PII exfiltration, image misclassification, or malicious use of tools the LLM agent can access.” While many of the attack types within the research may be similar to previous methods, McInerney says, the algorithm ties them together. “This is more along the lines of improving automated LLM attacks than undiscovered threat surfaces in them.”

However, he adds that as LLM agents become more commonly used and people give them more authority to take actions on their behalf, the scope for attacks against them increases. “Releasing an LLM agent that accepts arbitrary user input should be considered a high-risk activity that requires significant and creative security testing prior to deployment,” McInerney says.

For companies, that means understanding the ways an AI agent can interact with data and how they can be abused. But for individual people, similarly to common security advice, you should consider just how much information you’re providing to any AI application or company, and if using any prompts from the internet, be cautious of where they come from.

Source link

tags

Abkhazia Afghanistan Africa Albania Algeria Andorra Angola Antigua and Barbuda Argentina Armenia Australia Austria Azerbaijan Bahamas Bahrain Bangladesh Barbados Belarus Belgium Belize Benin Bolivia Bosnia and Herzegovina Botswana Brazil Bulgaria Burkina Faso Burundi Cambodia Cameroon Canada Cape Verde Central African Republic Chad Chile China Colombia Comoros Costa Rica Croatia Cuba Curacao Cyprus Czech Republic Denmark Djibouti Dominica Dominican Republic East Timor Ecuador Egypt El Salvador Equatorial Guinea Eritrea Estonia Ethiopia Fiji Finland France French Polynesia Gabon Gambia Georgia Germany Ghana Greece Grenada Guam Guatemala Guinea Guyana Haiti Honduras Hong Kong Hungary Iceland India Indonesia Iran Iraq Ireland Israel Italy Jamaica Japan Jordan Kazakhstan Kenya Kiribati Kosovo Kuwait Kyrgyzstan Laos Latvia Lebanon Lesotho Liberia Libya Liechtenstein Lithuania Luxembourg Macau Madagascar Malawi Malaysia Maldives Mali Malta Mauritania Mauritius Mexico Moldova Monaco Mongolia Montenegro Morocco Mozambique Myanmar Namibia Nepal Netherlands New Zealand Nicaragua Niger Nigeria North Korea North Macedonia Northern Cyprus Norway Oman Pakistan Palestine Panama Papua New Guinea Paraguay Peru Philippines Poland Portugal Puerto Rico Qatar Republic of the Congo Romania Russia Rwanda Saint Lucia Samoa San Marino Saudi Arabia Senegal Serbia Sierra Leone Singapore Slovakia Slovenia Solomon Islands Somalia South Korea South Sudan Spain Sri Lanka Sudan Sweden Switzerland Syria Taiwan Tajikistan Tanzania Thailand Togo Tonga Trinidad and Tobago Tunisia Turkey Turkmenistan Uganda UK Ukraine United Arab Emirates Uruguay USA Uzbekistan Vatican City Venezuela Vietnam Western Sahara Yemen Zambia Zimbabwe