Kato Coaching
@kato-coaching.com
Turning testers into trusted advisors. | Keynote speaker | Coach | Trainer Free resources and my book: https://t.mtrbio.com/kato-coaching
Asked AI for 30 rows of signup test data. Got 30 tidy, valid, useless rows. Treating it as a pair instead of an oracle, constrain, generate, verify, surfaces the bug the happy-path data missed. Two-minute demo on a real form. #softwaretesting
Stop Treating AI Like an Oracle: Better Test Data in Two Minutes
If AI keeps handing you clean, useless test data, that's down to ho...
youtube.com
The argument for not testing a prototype is that you don't know if it's worth keepin. That's no longer true. I'm partnering with SmartBear on a sponsored series of content about tools for the AI era. Opinions are entirely my own.
When Testing Costs an Hour, Not Days - Kato Coaching
I’m partnering with SmartBear on a sponsored series of content about tools for the AI era. As always, all opinions are entirely my own. A while ago I watched a mobile app go from a sales prop to a production system in the space of a single meeting. It had been built as a demo, […]
kato-coaching.com
Buying a testing agent: easy. Knowing if it works, recalibrating it, watching its reasoning, drawing its authority line: not in the box. The five questions I'd start with: https://kato-coaching.com/fighting-ai-with-ai-easy-part/ Paid partnership w/ SmartBear.
Fighting AI with AI is the easy part - Kato Coaching
I’m partnering with SmartBear on a sponsored series of content about tools for the AI era. As always, all opinions are entirely my own. SmartBear ran a survey in January 2026 of 273 software testing and quality decision-makers, mostly directors and above, and the writeup, their AI Software Quality Gap Report, lets the numbers speak […]
kato-coaching.com
Holy guacamole. Long-ish read, pretty jaw dropping. bobdahacker.com/blog/fifa-hack
I Could've Rickrolled the Entire FIFA World Cup. All I Needed Was My ID.
How I found that anyone could register on FIFA's public Agent Platform, gain access to the Football Data Platform's Streaming Management panel, and get RTMP ingest URLs and stream keys for every live ...
bobdahacker.com
You can't manually test AI-generated code at AI-generation speed. That math isn't mathing. 40% of teams generate 41%+ of their code with AI. 92% still test by hand. Source:
AI Software Quality Gap Report | SmartBear
Explore how AI is reshaping software quality. Get insights on testing gaps, risk, and how teams can maintain control as development accelerates.
smartbear.com
Everyone in QA can describe their AI's bad habits without thinking. Nobody finishes "what I really want from this tool is…" If you've been generating tests with AI and correcting the same handful of things every week, what would actually help?
The research on code review has been consistent for over a decade: it's strong at knowledge sharing, weak at catching defects. When teams insist on "humans review every line" of AI output, they're doubling down on the thing we know doesn't work. kato-coaching.com/human-in-the-loop-code-review
The AI keeps making the same test mistake because the decision to stop is buried in a Slack thread it's never read. Write the rule. Put it in the file your AI reads at generation time. Walkthrough: https://youtu.be/ccFaZdGUSkw #AIinTesting #QA
Where AI test rules live (and how to write one that works)
Your AI keeps making the same test mistake. You keep leaving the sa...
youtu.be
> 65% of QA teams describe AI output as "noisy, incorrect, or requiring rework." > > Mapping where the verification time actually goes. 5-min survey, no opt-in: >
AI Playbook Course: Pre-Launch Survey
Five questions, three minutes. Your answers shape the next AI Playbook cohort.
docs.google.com
Last night Alan and I talked about the new cool kid on the block, and how it's changing testing. Or maybe not? I'll link Apple podcasts, but it's available everywhere you get your podcasts.
Episode 230: Kat Obring is Back
Podcast Episode · AB Testing · May 5 · 54m
podcasts.apple.com
Our free web development bootcamp launch party is today! Join: @vivaldi.com's @brucel.bsky.social will be showing us the lovely side of HTML Me for how good notes drive learning @freecodecamp.bsky.social's @quincy.bsky.social for a fireside chat about learning We would love to see you there!
Launch Party
Virtual kickoff party to start the bootcamp. You don’t need to prepare anything, but you may want to come prepared to take notes. We’ll have several guests, including: freeCodeCamp Founder Quincy Laws...
badwebsite.club
It’s no secret that I love working with claude code. If you’ve not tried it yet but would like to: turns out I’ve got three passes for a free week of it. Grab it here claude.ai/referral/WY0...
Join Claude!
You've been invited to try Claude
claude.ai
I have a Claude Code skill that creates google slides for me, and then adds speaker notes. Since the update to Sonnet 4.7 it constantly "forgets" how to do the latter, and tells me it's incapable of it. Anybody else have this problem? Suggestions?
The "testing mindset" isn't a personality trait. It's trained. Developers often don't think about quality the way QA engineers do because they haven't had the same deliberate practice. That's fixable, but only if the culture reinforces it too.
Before you conclude AI isn't worth it, measure how long correction actually takes. One experiment, a real task, less than a day. Free guide: resources.kato-coaching.com/5-ai-experiments #SoftwareTesting #QA #AIinTesting
The Easter Bunny has 10,000 eggs to hide across 47 gardens before sunrise. His team suggests using AI to plan the hiding spots. Before handing a task to AI, run it through a simple framework. Here is how the Easter Bunny uses this framework to avoid a disaster. 🐇🥚
Can AI save the Easter Bunny?
Before you hand a task to AI, you need to know if it's actually a good candidate. The Easter Bunny has 10,000 eggs to hide across 47 gardens before sunrise. ...
youtu.be
Getting the task type wrong makes the model choice irrelevant. The scorecard tells you what kind of solution actually fits before you commit to one. https://resources.kato-coaching.com/scorecard #SoftwareTesting #AItools #QA
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
resources.kato-coaching.com
TIL: it's much easier to get Claude Code to play nicely with the Google workspace MCP than it is to get Gemini to use it. What the hell? Yes, I am experimenting with a multiple agent setup, but man, it's painful.
AI gave you 12 scenarios. Now what? The list is raw material, not a test plan, and your judgment is what turns it into one. New video on how to triage what AI gives you: https://youtu.be/0awmamL1l2c #softwaretesting #AIinTesting
How to Evaluate AI-Generated Test Scenarios
AI can give you a decent list of test scenarios - if you prompt it well. But the list is starting material, not a test plan. The work of deciding which scena...
youtu.be
Agile gave us JIRA. DevOps gave us pipelines. Neither answered the harder question. AI evals is doing the same thing: the tooling is excellent. The question underneath it remains. https://kato-coaching.com/the-hard-part-of-ai-evals-isnt-the-tooling/ #AITesting #QualityEngineering #SoftwareTesting
The Hard Part of AI Evals Isn't the Tooling - Kato Coaching
Additional thoughts on session three of the AI Evals and Analytics Playbook1. The first part is here2. Every major shift in how we build and ship software has been followed by a wave of tooling that automates the tractable part and leaves the actual problem to the practitioner. Agile gave us story pointing ceremonies and […]
kato-coaching.com
Most AI pilots produce impressions, not evidence. An acceptable error rate isn't a feeling. It's a number you write down before the experiment starts. https://resources.kato-coaching.com/scorecard #SoftwareTesting #AItools #QA
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
resources.kato-coaching.com
"The profession keeps arriving at “quality is systemic” as though it’s a fresh revelation, every time a new technology makes the verification frame obviously inadequate. It happened with continuous delivery. [...] It is happening now with AI agents."
The Verification Trap - Kato Coaching
Someone posted a thinking piece in a Slack channel I’m in last week, long and earnest and well-structured, arguing that quality engineering needs to evolve for the age of agentic AI, that we need to stop thinking about quality as testing and start treating it as a systemic property. I read it and felt a […]
kato-coaching.com
Every AI demo works because the vendor picked the use case. When you're back at your own codebase, the conditions are different. Define your problem before you evaluate the tool. https://resources.kato-coaching.com/scorecard #SoftwareTesting #AItools #QA
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
resources.kato-coaching.com
The vague prompt gets you a generic list. Two minutes of context - scope, users, risk angle - gets you scenarios worth thinking about. Short video on how to close that gap. https://youtu.be/55SLTvKq7rE #softwaretesting #AIinTesting
How to write AI prompts that give you useful test scenarios
Most AI prompts for testing are vague — and you get vague output back. "Write test cases for the SplitPay feature" returns a generic list you already knew. N...
youtu.be
The AI evals field calls it a rubric. If you've done serious testing work, you probably know it by a different name. https://kato-coaching.com/the-anatomy-of-a-metric/ #SoftwareTesting #AITesting
The anatomy of a metric - Kato Coaching
Session two closed with a question I couldn’t answer.1 When a product scores 78% on a given metric, what tells you whether that’s good enough to ship? I flagged it as something session three would probably address, and it did, though not in the way I expected. The question can’t be meaningfully answered until you’ve […]
kato-coaching.com
When AI adoption is handed to IT, you get a deployment. Deciding what good output looks like and how to verify it belongs to the people doing the work. https://resources.kato-coaching.com/scorecard?utm_source=Bluesky&utm_medium=post&utm_campaign=scorecard #QualityEngineering #TechLeadership
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
resources.kato-coaching.com
When teams say the AI tool isn't working, I ask about the process before the AI arrived. Usually the requirements were always vague. The tool is just making that visible. https://resources.kato-coaching.com/scorecard #SoftwareTesting #AIinTesting #QA
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
resources.kato-coaching.com
Six months into an AI pilot with no success criteria is just six months of accumulated impressions. You can't pass or fail a hope.
The AI Use Case Scorecard (Free)
Most AI output problems start before you open the tool. This free scorecard helps you check before you spend hours correcting output.
resources.kato-coaching.com
I am working on a food tracker for my very specific needs, and Claude just turned into every developer I ever worked with.
"The hard part is scalability, not automation." That line from session one of "AI evals and analytics" confused me. Session two explained it. Full write-up in my blog: https://kato-coaching.com/the-ai-evals-field-chose-a-flawed-tool-and-stuck-with-it/ #AIEvals #SoftwareTesting #QA
The AI evals field chose a flawed tool and stuck with it - Kato Coaching
Session one left me with two things I hadn’t resolved.1 The first was a line the instructor said almost in passing: “the hard part is scalability, not automation.” I wrote it down because it piqued something, but I couldn’t quite work out what problem it was pointing at. The second was a question I kept […]
kato-coaching.com