It's about what's hidden, and what new deficiencies the tech carries with it. @victorojewale.bsky.social opines on the evolution of deployed AI and its limits. victorojewale.substack.com/p/from-exper...

Every Generation of AI Solves One Problem and Hides Another

What brittleness looked like then, what it looks like now, and why “adding a layer” never fully solves it

victorojewale.substack.com

It's been a journey of nearly 3 years, but I'm very excited to announce the CNTR AISLE Portal! 🚀 cntr-aisle.org It’s a new way to review and evaluate the 1,000+ AI bills introduced in the U.S. over the last three years. Check out the Bill Library and our Profiles#AIPolicy #OpenData

AI benchmark evals commonly report "accuracy" metrics — but what’s really being measured? And how should we compute the error bars? New NIST report from my team at CAISI outlines a better statistical framework for eval analysis: www.nist.gov/news-events/...

New Report: Expanding the AI Evaluation Toolbox with Statistical Models

We're Hiring!Please visit the CAISI Careers Page to learn more abou

nist.gov

This has been a massive community project, and we need you all to participate! See more: evalevalai.com/projects/eve...

EvalEval Coalition@eval-eval.bsky.social · 6mo ago

🚀 Launching Every Eval Ever: Toward a Common Language for AI Eval Reporting 🚀 A shared schema + crowdsourced repository so we can finally compare evals across frameworks and stop rerunning everything from scratch 🔧 A tale of broken AI evals 🧵👇 evalevalai.com/projects/eve...

Had the chance to give feedback on this project on CAISI’s behalf. I’m very excited to see this develop!

EvalEval Coalition@eval-eval.bsky.social · 6mo ago

🚀 Launching Every Eval Ever: Toward a Common Language for AI Eval Reporting 🚀 A shared schema + crowdsourced repository so we can finally compare evals across frameworks and stop rerunning everything from scratch 🔧 A tale of broken AI evals 🧵👇 evalevalai.com/projects/eve...

If this kind of work speaks to you, come work with us! My team at CAISI is hiring an Applied Systems AI Research Scientist, among many other roles. www.nist.gov/caisi/career...

Careers at CAISI

About CAISICAISI, within NIST at the Department of Commerce, acts as a startup within g

nist.gov

Ryan Steed@rbsteed.com · 6mo ago

CAISI invites input on any aspect of this draft, including from orgs that conduct AI evals and from users of eval reports (for decision-making, procurement, integration, etc.) Public comment closes March 31 — details here: www.nist.gov/news-events/...

Excited to co-author a new public draft from NIST CAISI on best practices for automated benchmark evals. We want your feedback! Public comment open til March 31. Highlights below 🧵 www.nist.gov/news-events/...

Towards Best Practices for Automated Benchmark Evaluations

Comments Sought on Initial Public Draft of NIST AI 800-2 through March 31

nist.gov

Late notice, but: NIST has a multi-year, funded graduate fellowship + summer intern program (potentially with our team at CAISI!). Due January 15. stemfellowships.org/applicants/

Applicants

Fellowship Information *** Application is currently OPEN *** Number of Awards: Varies annually  Schedule: Online applications open in late August and close on January 15, 2026. Type: Fellowshi…

stemfellowships.org

Are you passionate about exploring what a conceptually cogent, methodologically sound, and well-founded AI evaluation and safety research might look like? Come do a PhD with us. Closing Date: 10 February 2026 Apply here aial.ie/hiring/phd-a...

US CAISI is hiring -- the internal govt name for the role is "IT Specialist" but it is effectively a research scientist role! Salary is $120,579 to - $195,200 per year, and you get to work on AI evaluation within government agencies! Job posting (**closes EOD 12/28/2025**): lnkd.in/exJgkqr5

Also, our team is hiring an AI Research Scientist! www.usajobs.gov/job/851528400

USAJOBS connects job seekers with federal jobs across the United States and around the world as the official employment site for the federal government

<p>NIST works with industry and science to advance innovation and improve quality of life. We're looking for an IT Specialist (AI) to join our team!<br> <br> <a href="https://www.nist.gov/caisi">CAISI...

usajobs.gov

Ryan Steed@rbsteed.com · 8mo ago

Our team at NIST's Center for AI Standards and Innovation (CAISI) just released a blog post with open questions for AI measurement science: www.nist.gov/blogs/caisi-...

Our team at NIST's Center for AI Standards and Innovation (CAISI) just released a blog post with open questions for AI measurement science: www.nist.gov/blogs/caisi-...

Accelerating AI Innovation Through Measurement Science

Building gold-standard AI systems requires gold-standard AI measurement science – the scientific study of methods used to assess AI systems’ properties and impacts. NIST works to improve measurements ...

nist.gov

US CAISI (the equivalent of the US "AI Safety Institute") just put out their approach to AI measurement & there's such a significant portion on construct validity (nist.gov/blogs/caisi-...). Great to see this after ongoing advocating about this issue (arxiv.org/abs/2511.04703)!

After having such a great time at #CHI2025 and #FAccT2025, I wanted to share some of my favorite recent papers here! I'll aim to post new ones throughout the summer and will tag all the authors I can find on Bsky. Please feel welcome to chime in with thoughts / paper recs / etc.!! 🧵⬇️:

Q: What do school buses, desktop computers, and AI have in common? A: The same decades-old laws and processes apply when governments go to purchase them. Our new #FAccT2025 paper examines how these legacy public procurement norms apply to AI. 🧵

Excited to present "Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling" at #CHI2025 tomorrow(today)! 🗓 Tue, 29 Apr | 9:48–10:00 AM JST (Mon, 28 Apr | 8:48–9:00 PM ET) 📍 G401 (Pacifico North 4F) 📄 dl.acm.org/doi/10.1145/...

Our lecture topic in our ML eval class today. ✨ Evals have always played an important socio technical role in policy decision-making and surprisingly little changes when you add "AI" to the mix

Ben Recht@beenwrekt.bsky.social · last yr.

Rossi’s Metallic Rules and why it’s hard to measure benefit when evaluating social programs.

Great webinar featuring Alessandro Acquisti on "The Economics of Privacy at a Crossroads." Alessandro has such a wide view of the privacy literature: both its history and inside & outside of economics. www.youtube.com/watch?v=ecc9...

The Economics of Privacy at a Crossroads | Luohan Webinar

YouTube video by Luohan Academy

youtube.com

TIL Minecraft has responsible AI lessons for kids education.minecraft.net//content/dam...

I was in Paris last week for the #AIActionSummit & was honoured to participate in the “AI in public interest” panel. 👇🏾 my thoughts & reflections on what AI in public interest is/isn’t & some concrete steps/initiatives for `bending the arc of AI towards the public interest' aial.ie/pages/aiparis/

When Toledo police rolled out their Fusus system, allowing officers to tap into the live feeds of privately owned cameras, they promised to only use the power in emergency situations. We obtained data that tells a very different story about when, and who, TPD watches. gizmodo.com/clearly-disc...

‘Clearly Discrimination’: How a City Uses Fusus to Spy on Its Poorest Residents

Fusus’s technology allows police to tap into live feeds from public and privately owned surveillance cameras. In Toledo, Ohio, cops use the power to watch one particular type of location.

gizmodo.com

As someone who has reported on AI for 7 years and covered China tech as well, I think the biggest lesson to be drawn from DeepSeek is the huge cracks it illustrates with the current dominant paradigm of AI development. A long thread. 1/

How often does the FTC base its enforcement proceedings on third party research? Often! ftcreverse.engineering

oh, this is out and they make me sound like an angel but a really good read “We are not evaluating systems for some hypothetical, potential risks in the future,” Birhane says www.fastcompany.com/91238006/how...

How Abeba Birhane is cleaning up AI’s dirty data

“We are not evaluating systems for some hypothetical, potential risks in the future. These audits are uncovering actual real issues, real problems, whether it's racism, sexism, or encoding of stereoty...

fastcompany.com

Come work with @jessicahullman.bsky.social and I as a postdoc! We'll apply approaches from HCI, AI/ML, statistics, and decision theory to design new tools & methods for evaluating human-AI decision-making (e.g., tools to elicit, represent, & validate specifications of real-world decision problems).

Jessica Hullman@jessicahullman.bsky.social · 2y ago

I'm seeking a postdoc to work with me and @kenholstein.bsky.social on evaluating AI/ML decision support for human experts: statmodeling.stat.columbia.edu/2024/12/10/p... P.S. I'll be at NeurIPS Thurs-Mon. Happy to talk about this position or related mutual interests! Please repost 🙏

tough morning for data brokers, great morning for people in addition to the CFPB FCRA rule (bsky.app/profile/benw...), the FTC announced a case against one who sold location data of visits to healthcare and places of worship, securing a ban on sale of this type of data www.ftc.gov/news-events/...

a man with a beard is sitting in a golf cart with his hand on his head .

Alt: dj khaled "another one" meme

media.tenor.com

Ben Winters@benwinters.bsky.social · 2y ago

This rule is massive - data brokers have run wild, selling sensitive data about Americans to the highest bidder with zero transparency or accountability -- this is an excellent update to enforce existing law in the modern age. Big W for privacy and security. www.consumerfinance.gov/about-us/new...

The proposed consent order with IntelliVision Technologies would prevent the company “from making misleading claims about its software. That would include any misleading statements about its performance identifying people of different genders, ethnicities, and skin tones.”

The FTC is cracking down on a firm that claims its AI-powered face recognition tech has 'zero' bias

A software provider won’t be allowed to lie about the accuracy of its artificial intelligence-powered facial recognition technology under a new proposed order from the Biden administration.

yahoo.com

Something (someone) to be thankful for today — very excited to see this!!

Dr Abeba Birhane@abeba.blacksky.app · 2y ago

Friends, this week marks a monumental step forward in my path towards pushing for a greater ecology of accountability in the age of AI. Thrilled to officially announce the AI Accountability Lab (AIAL) @theaial.bsky.social at Trinity College Dublin is launching today, Thursday Nov 28, 2024 aial.ie

Friends, this week marks a monumental step forward in my path towards pushing for a greater ecology of accountability in the age of AI. Thrilled to officially announce the AI Accountability Lab (AIAL) @theaial.bsky.social at Trinity College Dublin is launching today, Thursday Nov 28, 2024 aial.ie

Professors and researchers: are you booking speakers for the Spring semester? I have talks and workshops on data-driven investigations using crowdsourcing with WhatsApp, auditing black box AI systems and advanced web scraping. inspectelement.org/apis.html gist.github.com/yinleon/4e39...

Finding Undocumented APIs

Introduction, case studies, and exercises for finding and using undocumented APIs hidden in plain sight.

inspectelement.org