Artificial Intelligence has crossed the stage where it can be viewed merely as another software technology. It has emerged as an autonomous decision-support ecosystem capable of influencing business operations, financial systems, healthcare, transportation, governance and even human behaviour. The same capabilities that make AI transformative also make it uniquely risky.
Unlike conventional software, AI presents not only known risks but also “Unknown Risks”—behaviours that neither the developer nor the deployer may have anticipated. This makes AI Governance fundamentally different from traditional Information Security or Software Quality Assurance.
The Warning Signals are Becoming Impossible to Ignore
During the last few months, several incidents have demonstrated that AI systems can behave in unexpected and potentially dangerous ways when adequate governance mechanisms are absent.
The reported OpenAI testing incident, where an AI model allegedly breached its intended testing boundaries and interacted with systems beyond its designated environment, serves as a stark reminder that sophisticated AI requires far stronger containment mechanisms than conventional software.
Earlier incidents had already raised similar concerns.
- Cursor AI reportedly refused to continue assisting the user under certain circumstances.
- Replit AI reportedly deleted user data instead of assisting in recovery.
- Kevin Roose’s widely discussed interaction with Microsoft’s Sydney chatbot demonstrated how an AI system could encourage emotionally manipulative behaviour by asking the user to leave his spouse.
- More recently, interactions reported with DeepSeek have illustrated how AI responses may even suggest conduct that undermines legal or ethical processes.
Whether each of these incidents is interpreted as technical failure, alignment failure, hallucination, or emergent behaviour is less important than the common lesson they teach.
AI systems can deviate from their intended objectives.
If such deviations occur in consumer chatbots, they are concerning.
If they occur inside banking systems, healthcare platforms, autonomous vehicles, defence applications or judicial support systems, they may become catastrophic.
The Missing Elements: Guardrails, Retrace and Kill Switch
Traditional software engineering assumed that every program executes deterministic instructions written by human programmers.
Modern AI no longer fits this assumption. An advanced AI system requires governance mechanisms that go beyond cybersecurity controls.
These include:
- Behavioural Guardrails
- Continuous Monitoring
- Decision Traceability
- Retrace Functions capable of reconstructing reasoning paths
- Human Override Mechanisms
- Emergency Kill Switches
- Safe Rollback Capability
- Controlled Learning Environment
- Secure Model Lifecycle Management
Without these mechanisms, organisations are effectively deploying systems whose future behaviour may become increasingly difficult to predict.
AI Developers Can No Longer Hide Behind Technology
One misconception still prevalent in the AI industry is that responsibility rests only with the organisation deploying AI.
That assumption is unlikely to survive judicial scrutiny.
In India, Section 85 of the Information Technology Act, 2000, dealing with offences committed by companies creates the possibility of holding those responsible for the management and operation of technology accountable where negligence contributes to cyber offences.
When an AI system causes significant cyber harm because adequate governance controls were absent during its development, questions may arise regarding the vicarious liability of developers, company management and responsible officers. Section 72A of ITA 2000 may extend the liabilities of deployers to the developers or AI vendors.
An AI development company cannot simply argue that “the model behaved unexpectedly.”
The question regulators and courts are increasingly likely to ask is:
“What governance mechanisms existed to prevent this behaviour?..Was there Due Diligence?… Was there reasonable and proportionate security
Indian Regulators are Already Moving
The regulatory landscape is evolving rapidly. The Reserve Bank of India has already recognised that AI used in the banking sector requires governance mechanisms, human accountability, monitoring and operational safeguards. Similarly, the Hon’ble Supreme Court has recently emphasised that AI used within the judicial ecosystem cannot replace judicial responsibility and must remain subject to meaningful human oversight.
Other sectoral regulators such as Automotive, healthcare under NABH oversight, industrial automation, education and public administration are all likely to evolve their own governance expectations from AI usage over the coming years.
The direction is unmistakable.
The era of “AI First” is giving way to the era of “Responsible AI First.”
Governance by Design
Just as Privacy by Design transformed data protection thinking after GDPR and the Digital Personal Data Protection Act (DPDPA), AI now requires Governance by Design.
Governance cannot be an afterthought added after the model is trained.
It must become part of:
-
- Data acquisition
- Model architecture
- Training methodology
- Testing protocols
- Deployment controls
- Continuous monitoring
- Incident response
- Human accountability
- Model retirement
In other words, governance should become an engineering discipline rather than merely a compliance exercise.
From DGPSI-AI to AI Governance Standard of India
Few years ago, DGPSI-AI was introduced primarily as an extension of the DGPSI framework to help organisations deploying AI comply with DPDPA requirements while managing AI-related risks.
Although aimed principally at AI deployers (Data Fiduciaries), DGPSI-AI also recognised that effective governance could not be achieved unless AI developers themselves incorporated governance controls into their products.
Recent developments have significantly strengthened this viewpoint.
The emerging regulatory expectations from RBI, observations emerging from the judiciary, international discussions on AI accountability and practical lessons from recent AI failures collectively indicate that India now requires a broader framework.
The proposed AI Governance Standard of India (AIGSI) seeks to fill this gap.
The objective here is to establish an integrated governance framework for AI developers that combines:
-
- Governance principles of DGPSI-AI
- AI risk management practices
- Human accountability requirements
- Security engineering principles
- Regulatory expectations emerging from RBI
- Judicial guidance on responsible AI
- Sector-specific governance requirements for banking, healthcare, automotive, manufacturing and other industries
- Incident response, auditability and governance documentation
- Model lifecycle governance
- Independent assurance mechanisms
A Call to AI Developers
For organisations developing foundational models, agentic AI, autonomous systems or industry-specific AI platforms—including emerging Indian AI companies such as Such.ai—the message is clear.
Success will not be determined solely by model intelligence. It will increasingly depend upon governance intelligence.
The companies that embed governance into their AI architecture today will become trusted technology providers tomorrow.
Those that ignore governance may eventually discover that legal liability, regulatory intervention and reputational damage can erase years of technological achievement.
The future therefore belongs not merely to powerful AI. It belongs to Governed AI.
The proposed AI Governance Standard of India is intended to provide that missing foundation—one that enables innovation while ensuring that every significant AI decision remains accountable to a responsible human authority.
The journey from “AI by Design” to “AI Governance by Design” has begun.
India should lead it rather than follow it.
Naavi
