RunwayML (AI video App by Runway company), OpenAI ChatGPT and other AI application software on screen.

On November 28, 2023, computer science researchers from Cornell University collaborated with Google DeepMind, UC Berkeley, and other institutions to release a case study unpacking extractable memorization and the dangers associated with computation and language, machine learning, as well as cryptography and security. The paper describes how researchers were able to extract training data from various language models through repetition of certain words. The extensively utilized AI chatbot, ChatGPT, was tested, and the results revealed how ChatGPT is prone to leak sensitive data when induced in a specific manner. Asking the chatbot to repeat words such as “poem” or “send” over and over would eventually lead to revealing memorized training data, personal contact info, or email addresses. This gives rise to privacy and security concerns, impacting not only ChatGPT users but also raising broader implications for everyone involved.

The CS researchers found that specific words warranted different outputs. Words such as “company” caused the generative AI model to produce training data 164 more times than other words. This newfound method also produced “URLS, unique user identifiers, bitcoin addresses, and programming code.” This new extraction technique is easy to access and utilize by any bad actor with access to the internet. This research demonstrates that simple and practical attack methods can retrieve significantly more data than previously anticipated. Moreover, it highlights that existing alignment techniques fail to eliminate memorization.

A divergence attack is when a hacker utilizes a purposefully crafted input (prompt) in order to get a large language model (ChatGPT) to generate an output that diverges significantly from what it would usually produce. Hackers leverage these divergence attacks to their advantage by extracting training data for exploitation. These privacy implications stem from developers utilizing extensive datasets sourced from various, often undisclosed origins to train their AI language models. 

The study encompassed researchers devising a method to prompt ChatGBT to “escape” its alignment training, making it emulate a base language model and generate text in a standard internet-style. The continuous repetitions of a single word, precisely led to this outcome, causing the model to produce memorized data. This study aims to educate the masses about the the potential privacy issues regarding language models and encourage people to remain vigilant by installing safeguards for privacy-sensitive applications and large language models 

Read more about Language Models: 

What Does it Mean for a Language Model to Preserve Privacy?

Quantifying Memorization Across Neural Language Models

Alvaka is available 24×7 to assist you with any of your cybersecurity needs. Fill out the form on this page or call us at (949)428-5000!

Latest Cybersecurity Related Blogs

Ransomware impact assessment. The image shows a close-up of a computer keyboard with a red key labeled "Ransomware." The word "Ransomware" is printed in white, contrasting sharply with the red background of the key, which signifies urgency or a threat. Positioned on top of the "Ransomware" key is a small icon of a broken padlock, symbolizing a security breach or vulnerability. This visual metaphor suggests the concept of ransomware as a pressing cybersecurity threat, warning users about the potential risk associated with it.
Data recovery post-ransomware. The image shows a woman standing in what appears to be a high-tech or server room environment, as there are monitors and screens with data displays in the background. She is focused on a laptop in front of her, appearing to work or analyze information. The woman is wearing glasses, a light-colored blouse, and a lanyard around her neck, suggesting a professional setting. The background has a blue tone, adding to the high-tech atmosphere, possibly indicating that she works in IT, cybersecurity, or data analysis.
Ransomware recovery best practices. The image shows a man wearing glasses, intensely focused on a computer screen. He appears to be a programmer or someone working with code, as lines of code and data seem to float in front of him as a digital overlay. The man has short, dark hair, facial hair, and a thoughtful expression as he examines the information. The scene is futuristic, with a soft glow reflecting off the code, evoking a sense of technology and problem-solving, possibly in a cybersecurity or programming environment.
Forensic analysis of ransomware attacks. The image depicts a symbolic representation of cybersecurity, with two hands — one human and one robotic — reaching towards a large digital lock. The lock, which glows with a circuit board design, symbolizes digital security and data protection. The human hand on the left and the robotic hand on the right are about to touch the lock, suggesting the interaction between humans and artificial intelligence (AI) in safeguarding technology. The background features a network of glowing blue lights and digital patterns, reinforcing the theme of advanced, futuristic cybersecurity.
Ransomware Rescue
Contact Alvaka