Regarding testability behaves like a black box and can't "tell" us what it's
thinking. It takes data as input and returns a decision. The problem is finding
the "right" inputs that return the "wrong" decisions.

Software is used to automate things at massive scale (this is where AI really
shines), therefore mistakes there have a much larger impact that then same
mistakes made by individual humans.

Some AI systems have to handle reality, not like most like "classical" software
where we very often define the scope of the environment for the software. That
makes it incredibly important for testers to work on uncovering the unknown
unknowns.

It's also dangerous to only focus on the outputs: building a good AI will
replace the human experts who are needed to verify the quality of decisions
during development. Quality means also ensuring that the best body of knowledge
is considered.