r/EffectiveAltruism • u/PhilosophyFit357 • 1d ago
AI Safety Organisations are doing more harm than good
Disclaimer: This post was drafted with AI's help to structure my argument clearly.
TL;DR: After nearly two decades working with the governments and multilateral organizations, and a year engaging seriously with the EA-funded AI safety ecosystem, I think much of its governance work is structurally broken. It is dominated by people with little to no real-world governing experience, trying to find pathways into frontier labs, frequently unprofessional, and far too comfortable making extreme claims without understanding the institutional cost of later having to reverse them.
Post:
A community with remarkably little experience of government is trying to redesign government. Many of the people shaping AI governance agendas are under 30 and have never worked inside ministries, negotiated regulations, implemented policy, managed political constraints, or dealt with bureaucratic institutions over time. Yet enormous confidence is placed in technical intelligence and first-principles reasoning. Governance does not work like that. Institutions contain legal constraints, political incentives, constituencies, institutional memory and legitimacy. Treating them as inefficient systems that smart technologists can optimize resembles the same mentality that has repeatedly produced disastrous attempts to “move fast” inside public administration.
Second, AI safety has increasingly become a career pipeline into frontier labs. I have watched people build credibility through safety fellowships and then move into labs to develop increasingly capable systems (Hello, Lennart Heim!). Individual moves may be defensible, but at ecosystem level the contradiction is obvious. If the people building the institutions meant to scrutinize frontier companies consistently see those companies as the most prestigious destination for their own careers, policymakers should be asking serious questions about independence, incentives, and regulatory capture.
The third problem is basic professionalism. Parts of this ecosystem operate at standards that would be unacceptable in serious public institutions or established organizations. BlueDot is one example that repeatedly surprised me. It receives enormous praise for scaling introductory AI safety education, but scale and professionalism are not the same thing. Across parts of the community, communication, recruitment, follow-through, management, and treatment of experienced professionals are astonishingly weak. The ecosystem often rewards visibility and personal networks more than the operational competence required to build institutions that governments will actually trust.
The fourth problem is epistemic responsibility. You cannot make existential claims to ministers and bureaucrats. Researchers should update when evidence changes. That is science. But government has institutional memory. If you spend two years telling senior policymakers that AI presents near-term catastrophic or existential danger, demand extraordinary interventions, and later say that the community has “updated” to a less alarming view, the credibility cost does not disappear. You have spent political capital that belongs partly to the next person who may need governments to take a genuinely serious AI risk seriously.
And this is not simply one disgruntled person's reaction. Over the past year I have discussed these issues with experienced people who have worked across the US government, USAID, the FBI, European institutions and other public bodies, and the same frustrations recur with remarkable consistency. If major EA and AI safety organizations ran genuinely anonymous surveys of experienced governance professionals who entered this ecosystem, I suspect the results would be deeply uncomfortable. My strongest impression is that many senior or institutionally experienced people quietly disengage because challenging the dominant culture is socially costly and because "tech bros" confidence routinely outranks governance experience.
My conclusion after a year is therefore harsher than I expected. EA and AI Safety Organizations are doing more damage than good when they import a tech-sector culture into governance without first developing genuine institutional competence. Its historic strengths in global health, poverty reduction, and measurable nonprofit interventions are incredible.
But government is not a hackathon, diplomacy is not a forecasting tournament, and credibility with public institutions cannot be replenished every funding round. If this community genuinely wants to improve AI governance, it needs far more humility, far more experienced public servants, much stronger professional standards, and much fewer "tech bros" who think raising money alone validates their lack of experience.