The Agentic AI Security Series: Who Is Accountable When AI Gets It Wrong? (Part 6 of 6)
FranklyTECH
The Agentic AI Security Series (Part 6 of 6)
Who Is Accountable When AI Gets It Wrong?
Throughout this series, we have examined what Agentic AI is, how much authority it should receive, why Human In The Loop remains essential, whether employees are prepared for it, and how regulation and oversight might help guide its development.
All of those discussions ultimately lead to one final and perhaps more difficult question: Who is accountable when Artificial Intelligence gets it wrong?
Imagine an AI Agent exposes confidential information, approves a financial transaction it should have rejected, compromises security, provides inaccurate information that leads to a consequential decision, or takes an action that causes financial or physical harm. Who is responsible? Is it the employee using the system, the manager overseeing the process, the organization deploying the technology, the developer who created it, the technology company that provided it, or even the government agency that approved or used it?
As Artificial Intelligence becomes more autonomous, determining responsibility may become more complicated. However, accountability cannot be allowed to disappear simply because technology participated in the decision.
Limited Regulation, Strong Oversight, and Accountability
My thinking about AI regulation has evolved throughout this series. As I discussed in Parts 5 and 5B, I believe there may be a legitimate role for limited, high level regulation establishing basic guardrails around security, privacy, transparency, accountability, national security, and especially high risk uses of Artificial Intelligence. But limited regulation is only the beginning.
Something this consequential should have broad bipartisan support and significant input from experts in Artificial Intelligence, cybersecurity, national security, technology, business, law, healthcare, education, privacy, and other affected fields. Artificial Intelligence is simply too important to our economic future and national security to become controlled by whichever political party happens to be in power. I do not pretend to know exactly what that structure should look like. There are people with far more expertise in government, law, technology, cybersecurity, and national security who should help determine that. However, I believe strongly in the principles behind it.
Oversight should be as neutral, knowledgeable, balanced, transparent, and politically independent as reasonably possible.
Its objective should not be to regulate every AI application or determine which ideas are allowed to exist. Its purpose should be to oversee compliance with limited guardrails, examine serious incidents, identify emerging risks, and help ensure that accountability follows when those safeguards are deliberately or negligently ignored.
AI oversight should protect the country, its people, and its security, not a political party, government agency, corporation, or special interest. Regulation Should Be the Minimum, Not the Entire Standard
Even with limited regulation, institutions cannot simply satisfy the minimum legal requirements and consider their responsibility complete. Corporations, government agencies, universities, technology providers, and other institutions using Artificial Intelligence should establish their own internal governance standards in conjunction with whatever limited regulations exist. We already follow this concept in other areas of organizational responsibility. Institutions establish internal policies for privacy, cybersecurity, confidential information, financial controls, acceptable technology use, and the handling of personal information. Artificial Intelligence should be approached in much the same way.
Organizations should establish clear policies defining which AI systems may be used, what information those systems may access, what actions AI Agents are permitted to take, when Human In The Loop approval is required, how AI activity will be monitored, how incidents will be reported, and who ultimately has responsibility when something goes wrong.
Regulation should establish the minimum guardrails, while institutions take responsibility for governing themselves within and, when appropriate, beyond those guardrails. That creates multiple layers of protection rather than assuming government regulation alone can anticipate every possible AI risk.
What Is a Corporation? What Is a Government?
Before we can decide who should be accountable, I believe we need to answer an even more fundamental question: What exactly is a corporation? And, for that matter, what exactly is a government?
A corporation isn’t the building. It’s not the walls, computers, logo, or name displayed outside. A corporation is ultimately made up of people.
People decide which AI systems will be purchased, developed, connected, and deployed. People determine what information those systems can access. People decide how much authority an AI Agent receives. People establish safeguards, approve policies, sit on boards, manage employees, and determine how much risk an organization is willing to accept.
I believe the same principle applies to government. Government is not the Capitol building, a courthouse, an agency headquarters, or the technology running inside those buildings. Government is ultimately made up of people too.
Elected officials, appointed officials, agency leaders, employees, contractors, technology specialists, military personnel, and many others make decisions on behalf of government. When Artificial Intelligence is purchased, developed, deployed, or used by a government, people are still making the decisions behind it.
That distinction matters because institutions can sometimes become shields behind which individual responsibility disappears. We hear that "the company made the decision" or "the government decided." But somewhere within that corporation or government, people made decisions that contributed to the outcome.
That does not mean everyone within the institution is responsible. It means accountability should follow the people who had the authority, responsibility, knowledge, intent, and involvement appropriate to the decision.
Accountability Must Apply at Every Level
If accountability is going to be fair and blind, it cannot apply only to executives, boards of directors, government officials, or technology companies. It must extend to employees as well. If an employee deliberately ignores established AI safeguards, knowingly violates clearly communicated rules, intentionally circumvents security controls, or knowingly uses Artificial Intelligence for an unauthorized or unlawful purpose, that individual should be accountable for those decisions.
The same standard should apply throughout an institution. If a manager instructs employees to circumvent safeguards, that matters. If an executive knowingly ignores documented risks, that matters. If a board receives credible warnings and deliberately fails to act, that should be examined. If a government official knowingly authorizes an irresponsible use of Artificial Intelligence, that should also be subject to appropriate accountability. However, there is an important distinction between intentional misconduct and an honest mistake.
Artificial Intelligence is developing rapidly, and people are going to make mistakes. Someone may misunderstand a policy, use an AI system incorrectly, fail to recognize a risk, make an incorrect judgment, or simply make an honest mistake. If every mistake results in severe punishment, employees may become afraid to use AI, experiment with it, or even report problems when they discover them. That could ultimately make AI less safe rather than more safe.
Accountability should therefore be proportional to the action, intent, circumstances, repetition, and consequences. An honest, unintentional mistake might require education, retraining, corrective guidance, or a warning. A repeated violation after appropriate education and previous warnings may justify a stronger response. Recklessly ignoring known safeguards should carry more serious consequences, while deliberately using Artificial Intelligence to steal information, circumvent security controls, commit fraud, compromise systems, or intentionally cause harm belongs in an entirely different category.
The consequence should be proportional to the misconduct.
That principle should apply regardless of title. An entry level employee should not become the scapegoat for an AI failure caused by inadequate training, unclear policies, poor security controls, or decisions made by management. At the same time, an employee who deliberately violates established safeguards should not escape accountability simply because the organization ultimately bears responsibility for its workforce.
Fair accountability must work in both directions.
Education Must Be Ongoing
This is where education becomes essential, and I do not believe AI education can be treated as a one time event.
Artificial Intelligence is rapidly. The systems employees use today may have significantly different capabilities tomorrow. New features will emerge. AI Agents will gain new capabilities. New security threats will develop. Company policies will change. Regulations may change. What we teach someone today may need to be updated six months from now, next month, or potentially even sooner.
Organizations therefore have a responsibility to provide ongoing AI education and awareness rather than assuming a single class, annual course, or policy acknowledgement is enough.
Employees should continually be educated about which AI systems they are permitted to use, what information may and may not be entered into those systems, what actions AI Agents are authorized to take, when Human In The Loop approval is required, what new risks have emerged, and what employees should do when they discover something that appears wrong.
This becomes especially important when we begin talking about accountability. Organizations cannot establish AI policies, fail to keep employees educated as the technology evolves, and then hold those employees fully accountable for unknowingly violating requirements they were never taught or changes they were never informed about.
At the same time, employees have responsibilities as well. When an organization provides clear policies, ongoing education, updated training, reasonable safeguards, and opportunities to ask questions, employees have a responsibility to understand and follow those requirements.
Education and accountability therefore need to work together. An honest mistake may require additional education, coaching, retraining, or a warning. Repeated negligence after appropriate education and previous warnings may require a stronger response. Deliberately violating safeguards or knowingly using Artificial Intelligence for an unauthorized or unlawful purpose should carry consequences proportional to the misconduct. As Artificial Intelligence continues to evolve, our education must evolve with it.
"The AI Did It" Cannot Become an Excuse
As Artificial Intelligence continues to evolve, our education must evolve with it. But education alone does not solve the accountability question, especially as Agentic AI becomes increasingly capable of making decisions and taking actions without a person directing every individual step.
Saying "the AI did it" cannot become an automatic excuse when something goes wrong. At the same time, we have to recognize that sometimes the AI really may have made the decision. An AI Agent may choose between alternatives, initiate an action, or respond to a situation without a human explicitly telling it what to do at that moment.
AI may also hallucinate, generating information that appears credible but is inaccurate or unsupported. With traditional AI, a hallucination might simply result in a bad answer that a person can hopefully recognize and correct. With Agentic AI, the stakes can be much higher. If an AI Agent believes inaccurate information and has the authority to act on it, that hallucination could potentially become an email sent, a record changed, a transaction initiated, access granted, or another action taken before a person realizes something is wrong.
That is precisely what makes Agentic AI different and why accountability becomes more complicated. The important question is no longer simply who made the individual decision, but who gave the AI the authority to make that decision, what safeguards were in place, and who was responsible for monitoring the outcome?
People and institutions still determine whether an AI system is purchased, developed, deployed, and connected to corporate or government systems. People establish its initial permissions, determine what information it can access, define the boundaries within which it operates, establish safeguards, and decide how much autonomy it receives. Once that authority is granted, however, Agentic AI may make decisions and take actions within those boundaries without asking a person for permission at every step.
That creates a new level of responsibility. If an organization gives an AI Agent broad authority, the organization cannot simply distance itself from the consequences when the Agent uses that authority in an unexpected way or acts on inaccurate information. Determining accountability may require examining the entire chain: Who developed the system? Who tested it? Who deployed it? Who established its permissions? Who determined its level of autonomy? Who monitored it? Were known risks or hallucinations ignored? Were employees properly and continually educated? Did someone deliberately misuse it? Who had the ability to intervene?
Accountability may sometimes belong primarily to one person and at other times be shared among several people, organizations, developers, providers, or government entities. What matters is that it does not disappear simply because an AI Agent made the final decision.
The more independent decision making authority we give Artificial Intelligence, the more clearly we must define accountability before something goes wrong.
An AI Agent should never receive more authority than an organization is prepared to accept responsibility for.
Oversight Makes Accountability Possible.
This is where the oversight argument from Part 5B becomes critical. Limited regulation establishes basic guardrails. Institutional self governance creates additional controls. Ongoing education ensures people understand how to use AI and their responsibilities as the technology changes. Oversight monitors what is actually happening. Accountability determines the appropriate response when something goes wrong.
Oversight should not begin only after an incident occurs. It should be continuous. Individuals, companies, and government organizations should understand what their AI systems can access, what actions those systems can perform, what decisions require human authorization, and who possesses the authority to stop and possible correct the system when necessary. Oversight can include internal controls, independent audits, board accountability, government monitoring of especially high risk activities, security controls, ongoing employee education, transparency, continuous review, incident reporting, and lawful intelligence and security monitoring when warranted.
In my opinion, AI itself will eventually become one of the most valuable technologies supporting that oversight. AI systems can potentially help monitor other AI systems, identify unusual behavior, detect patterns, recognize possible policy violations, maintain audit trails, and alert people when intervention may be required. That does not eliminate the Human In The Loop. It makes the Human In The Loop more effective.
AI can monitor, analyze, document, and raise warnings. People must remain responsible for interpreting what those warnings mean, investigating what happened, applying judgment, and determining the appropriate response. Human In The Loop should never simply mean that someone clicks an approval button. It should mean that a person has the knowledge, authority, and responsibility to question AI, reject its recommendation, stop its action, and accept accountability for the ultimate decision.
Accountability Should Be Transparent
Oversight without transparency can easily become another closed system protecting itself. When serious AI incidents occur, I believe the public should receive meaningful information about what happened, why it happened, what impact resulted, and what corrective actions were taken, except where disclosure would compromise national security, individual privacy, an active investigation, legitimate confidential information, or create additional security risks.
Transparency does not mean exposing proprietary technology, sensitive personal information, or information that could help bad actors exploit systems. It means providing enough information for people to understand whether something went wrong, whether appropriate safeguards existed, and whether those responsible responded appropriately.
Trust will not come simply from organizations telling us their Artificial Intelligence is safe. Trust will come from responsible governance, effective oversight, appropriate transparency, honest reporting, independent review, ongoing education, and visible accountability when systems fail.
Bringing the Agentic AI Security Series Full Circle
Throughout this series, we have examined how Artificial Intelligence is moving from answering questions to taking actions. We have discussed authority, security, Human In The Loop, employee education, regulation, global oversight, institutional self governance, and accountability.
Each subject ultimately returns to the same principle: As Artificial Intelligence becomes more capable, human responsibility must become clearer, not weaker.
Organizations can delegate repetitive tasks to AI. People can allow AI to analyze large amounts of information, make recommendations, interact with systems, and take certain actions. Artificial Intelligence may even help monitor other Artificial Intelligence. What we cannot do is delegate away responsibility for the outcome.
Corporations are made up of people. Governments are made up of people. Technology companies, boards, management teams, and workforces are all made up of people. Different people possess different levels of authority and responsibility, and accountability should reflect those differences. Somewhere along that chain, responsibility must remain attached to identifiable people or groups.
We can delegate tasks to Artificial Intelligence. We can delegate authority to Artificial Intelligence. We can even allow Artificial Intelligence to make certain decisions. But we cannot delegate away human accountability.
When Artificial Intelligence goes wrong, whether because of the technology itself or because of how someone chooses to use it, accountability cannot simply disappear. Someone, or some group, must ultimately be responsible.
But I want to close this series with something even more fundamental.
When I talk with people about Artificial Intelligence, the conversation can become passionate very quickly. Some people are excited about it, while most others are deeply concerned or completely against using it. They worry that AI is hurting society, eliminating jobs, creating security and privacy risks, or advancing too quickly. I understand those concerns. Many of those concerns are exactly why I wrote this series.
However, I believe there is one fundamental fact we all need to accept:
Artificial Intelligence is here to stay.
AI is not going away because we are uncomfortable with it. It is not a temporary trend that we can ignore until something else comes along. There is too much money being invested, too much global competition, too much potential reward for the next breakthrough, and too many extraordinary opportunities ahead. AI is already integrated into the applications, systems, devices, and technologies we love using every day, and that integration will continue to grow.
Putting blinders on will not slow Artificial Intelligence down. It will only cause the people and organizations wearing those blinders to fall further behind. With technology advancing at today’s pace, catching up may become increasingly difficult.
That is why I believe we need to teach AI to all, and we need to teach it now. Employees need education. Executives need education. Government officials need education. Students need education. Educators themselves need education. We all need to understand what AI can do, what it cannot do, where the risks are, how Agentic AI changes the equation, and how these technologies can be used securely, responsibly, and effectively to advance society in a positive healthy way.
That does not mean ignoring the dangers. Education is what allows us to recognize those dangers, understand them, and make better decisions about them. Security, Human In The Loop, self governance, oversight, limited regulation, ongoing education, and accountability all have a role.
After more than three decades as a professional IT trainer and educator, perhaps it should not surprise anyone that I return to education as part of the answer. I trully believe that education is the answer to all problems.
I have watched technology transform organizations throughout my career, and I firmly believe the individuals and organizations that benefit most from AI will be those that make the effort to understand it rather than fear it.
AI is here. Accept it. Learn it. Understand it. Use it responsibly. Most importantly, educate yourself and others about it!
I hope you enjoyed the series. I’d love to hear your thoughts.
By Frank Pannacchione - President of Metropolitan Computer Services, Inc.
