🚨 New blog post alert! @xpnsec.com drops knowledge on LLM security w/ his latest post showing how attackers can by pass LLM WAFs by confusing the tokenization process to smuggle tokens to back-end LLMs. Read more: ghst.ly/4koUJiz

Tokenization Confusion - SpecterOps

Meta's Prompt Guard 2 aims to prevent prompt injection. This post looks at how much knowledge of ML we need to be effective at testing these LLM WAFs.

ghst.ly

BIG NEWS: SpecterOps raises $75M Series B to strengthen identity security! Led by Insight Partners with Ansa Capital, M12, Ballistic Ventures, Decibel, and Cisco Investments. ghst.ly/seriesb #IdentitySecurity #CyberSecurity (1/6)

Come join us, there isn’t a better place to work and show your technical excellence surrounded by the industry’s best if you ask me!

SpecterOps@specterops.io · 2y ago

Our Consulting Services team is growing! 🙌 We are now hiring Consultants and Senior Consultants to join the team as operators, trainers, and program developers. Learn more & apply today! ghst.ly/3PBmGFZ

Interesting choice of words in the title. The model itself wasn’t hijacked but the code in the repository was through pull requests with code injection via branch names 🤯 www.bleepingcomputer.com/news/securit...

Ultralytics AI model hijacked to infect thousands with cryptominer

The popular Ultralytics YOLO11 AI model was compromised in a supply chain attack to deploy cryptominers on devices running versions 8.3.41 and 8.3.42 from the Python Package Index (PyPI)

bleepingcomputer.com

The OWASP LLM and Generative AI Security Project has launched the Agentic Security Initiative to address security challenges posed by autonomous AI agents. This effort focuses on developing best practices to secure agentic LLM and Generative AI applications. genai.owasp.org/2024/12/15/a...

Announcing the OWASP LLM and Gen AI Security Project Initiative for Securing Agentic Applications - OWASP Top 10 for LLM & Generative AI Security

The OWASP LLM and Generative AI Security Project is thrilled to announce the launch of the Agentic Security Initiative designed to tackle the unique security challenges posed by Autonomous AI agents. ...

genai.owasp.org

First dataset for the new @huggingface.bsky.social @bsky.app community organisation: one-million-bluesky-posts 🦋 📊 1M public posts from Bluesky's firehose API 🔍 Includes text, metadata, and language predictions 🔬 Perfect to experiment with using ML for Bluesky 🤗 huggingface.co/datasets/blu...

bluesky-community/one-million-bluesky-posts · Datasets at Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

huggingface.co

If you find yourself with too much free time over the (long) weekend / holidays, I have ~3h Building an LLM from the Ground Up workshop on YouTube that may come in handy: m.youtube.com/watch?v=quh7...

Building LLMs from the Ground Up: A 3-hour Coding Workshop

YouTube video by Sebastian Raschka

m.youtube.com

I really enjoyed reading this paper from OpenAI. If you perform AI assessments, you should read it. I thought they laid out a pragmatic approach to evaluating AI models that should be a component of any organization's assessment methodology. cdn.openai.com/papers/opena...

I couldn't find any PowerShell examples of encrypting/decrypting data w/ Azure Key Vault keys, so I made some: Protect-StringWithAzureKeyVaultKey Unprotect-StringWithAzureKeyVaultKey github.com/BloodHoundAD... Explanatory blog post coming soon.

Add key vault cryptographic op funcs · BloodHoundAD/BARK@e1c82a1

github.com

This State of Generative AI report from Menlo Ventures provided some good insights on where cybersecurity professionals might look for risk in terms of assessments and research. menlovc.com/2024-the-sta...

2024: The State of Generative AI in the Enterprise - Menlo Ventures

The enterprise AI landscape is being rewritten in real time. We surveyed 600 U.S. enterprise IT decision-makers to reveal the emerging winners and losers.

menlovc.com

This looks like a fun challenge to evade prompt injection defenses microsoft.github.io/llmail-inject/

Love this, hoping to do something similar with our assessments.

Chris Truncer@christruncer.bsky.social · 2y ago

It’s always awesome when we (@CISAGov) gets to release a red team report that we worked on, and today is another one of those days! Go check out our latest report and hopefully you can apply some of the same lessons to your environment! www.cisa.gov/news-events/...

“Cybersecurity professionals and ethical hackers need to understand the darker side of hacking to better prepare for potential threats. Unfiltered AI models can provide insights into hacking methodologies and scenarios typically censored, aiding in the development of robust cybersecurity measures.”

Great read on how "China Hawks are Manufacturing an AI Arms Race", a concerning trend for anyone advocating for regulation and safety of AI. An arms-race narrative would ensure an unfettered and unregulated development of AI in almost all contexts. garrisonlovely.substack.com/p/china-hawk...

China Hawks are Manufacturing an AI Arms Race

An influential congressional commission is calling for a militarized race to build superintelligent AI based on threadbare evidence

garrisonlovely.substack.com

Conflicted about this post on prompt injection for multi modal models. Turns out they read instructions and follow them 🤯. All data from input should be untrusted from the system and user prompts and not processed as one. www.lakera.ai/blog/visual-...

The Beginner's Guide to Visual Prompt Injections: Invisibility Cloaks, Cannibalistic Adverts, and Robot Women | Lakera – Protecting AI teams that disrupt the world.

Learn about visual prompt injections, their appearance, and top defense strategies against these attacks.

lakera.ai

Has anyone fired up this PentestGPT during an actual assessment? I did like their pentesting task tree (PTT) to track the status of tests. www.usenix.org/conference/u...

Can the sandbox reach the Internet 👀? Asking for a friend. Are we going for security through obscurity by keeping system prompts private?

Justin Elze@handle.invalid · 2y ago

I thought we stopped doing this? www.darkreading.com/cloud-securi...

Great paper, especially like the parts on data & model provenance and the Supply-chain Levels for Software Artifacts. These could really make offensive security operations challenging.

Mihai Maruseac@mihai.page · 2y ago

We are happy to publish a whitepaper on how we're thinking on securing the AI supply chain both internally and for OSS. This is a culmination of nearly a year of thinking about this space, from people working on AI or security, across multiple Google PAs. research.google/pubs/securin...

Microsoft’s Orca Agent-Instruct dataset has been released! Permissively licensed 1M synthetic instruction pairs covering different capabilities, such as text editing, creative writing, coding, reading comprehension Paper: arxiv.org/abs/2407.03502 Dataset: huggingface.co/datasets/mic...

AgentInstruct: Toward Generative Teaching with Agentic Flows

Synthetic data is becoming increasingly important for accelerating the development of language models, both large and small. Despite several successful use cases, researchers also raised concerns arou...

arxiv.org

Nice write up from Mandiant on some practical use cases for leveraging AI to help red team operations. What are some other use cases ya’ll are thinking of? cloud.google.com/blog/topics/...

AI Enhancing Your Adversarial Emulation | Google Cloud Blog

Learn how Mandiant Red Team is using Gemini and LLMs for adversarial emulation and defense.

cloud.google.com