how you never have anyone who can actually “manage” the AI workers.

You just use other AI to manage those worker AI. Experiments do show that having different instances of AI/LLM, each with an assigned role like manager, designer, coding or quality checks, perform pretty good working together. But that was with small stuff. I haven’t seen anyone wiling to test with complex products.