We're not going to be governed by AI because we're ready, it's going to be because the people who own it have secured enough power to make it happen. Whether it takes the form of government or capitalism.
I do not want vibe-coding the law, especially criminal law. I do not want vibe-coding the tax rules. I do not want vibe-coding traffic safety.
And, in fact, we won't be governed by AI, even if we are. If we're governed by AI, we're really governed by whoever trained the AI, and/or whoever curated the training data. Do we want to be governed by them? Again, no, with expletives.
Of course it will be every bit as bad as the people who implement it. But that just kinda highlights the core problem.