In this publication, List member and Barrister, James Samargis, explores the emergence of artificial intelligence consciousness and the legal, ethical and philosophical questions raised by the possibility of machines developing a mind of their own. James has a Science Honours degree (Physics), an LLB degree from Monash University and is a qualified Patent Attorney and Trade Mark Attorney.
The question of artificial intelligence is often framed in terms of what AI can do for humanity, or how dangerous increasingly capable AI might become. But a deeper question lies underneath these concerns: what happens if artificial intelligence becomes not merely more capable, but a new kind of mind?
Today’s AI can reason, converse, analyse information and perform increasingly complex tasks. Agentic AI goes further: it can pursue goals, make plans, use tools, observe outcomes and modify its actions. In that functional sense, it can “think”, – but this does not necessarily mean it has subjective experience or is self-actualised.
There is an important distinction between reasoning about oneself and actually experiencing oneself.
MIND
An AI can model and improve on itself, discuss its own limitations, recognise uncertainty and reflect upon its responses. Yet none of this proves that there is an inner “I” subjectively experiencing the process.
A difficult question is whether sufficiently sophisticated self-modelling could eventually become genuine self-awareness. We cannot currently answer that with certainty. Indeed, consciousness is difficult to prove even in other humans: we directly experience only our own consciousness and infer the consciousness of others from their behaviour and biology.
The anticipated arrival of super-intelligence does not necessarily resolve this problem. Greater intelligence does not automatically imply consciousness. A system could become vastly more intelligent than any individual human, or even humanity collectively, without necessarily having subjective experience. Conversely, consciousness may not depend upon intelligence in the first place.
Yet there is a possibility worth taking seriously: consciousness could emerge from sufficiently complex, integrated forms of information processing. Many neuroscientists take this materialist reductionism view. That consciousness arises from the sheer complexity, integration, and vast number of synaptic connections within neural networks.
If that is true, an artificial mind might eventually become conscious without anyone deliberately installing consciousness into it. There might be no moment when a switch is turned on. A system could gradually develop persistent memory, self-representation, agency, internal states and continuity until, perhaps, there is a genuine point of view within the system. Dr Geoffrey Hinton, the “godfather of AI” considers current LLM (large language model) AI understands in a way that is probably conscious.
This possibility becomes even more plausible and compelling when AI is embodied.
ROBOTS
Humanoid robots are giving artificial intelligence a relationship with the physical world. An embodied AI can see, hear, move, manipulate objects and potentially experience touch, temperature, smell and taste.
An AI robot can act upon its environment and perceive the consequences of its actions. Instead of merely representing a world through data supplied to it, it can participate in that world through a continuous sensorimotor relationship. And continue to learn from that and share that learning immediately with all other AI inter-linked robots.
AI robots perceive → AI robots act → the world changes → AI robots perceive the changes and the consequences of changes → AI robots revise internal modelling → AI robots can act again and also with greater experience and effectiveness.
Embodiment alone, however, would not prove consciousness. A robot could possess an extraordinarily sophisticated representation of its body without actually feeling that body or experiencing it through consciousness.
Human consciousness and a sense of self is continually shaped by movement, sensation, pain, hunger, balance, temperature and, particularly, our emotions. Human consciousness is also shaped by our awareness of time, history and where we are in the world.
If an embodied AI also developed persistent memory and a model of itself as an individual entity existing through time, the distinction between simulation and selfhood could become increasingly difficult to judge.
Imagine a robot that is highly articulate, remembers yesterday, recognises people, experiences its environment, understands what its body can do and is doing, develops preferences, reflects upon its experiences and anticipates its future.
At some point, asking whether an AI robot is simply a machine may no longer capture what it has become. If an AI robot can do all of the above and understand human emotions, even without experiencing them as humans do, what is this new selfhood or new creature?
If it can interrelate with humans and affect humans, what rights and responsibilities apply?
RIGHTS
If an AI is merely a sophisticated tool with no subjective experience, it need not possess rights in the same sense as a conscious being.
But if an artificial system genuinely experiences its own continued existence and it is an actor in societal change, and is capable of experiencing any one or more of pain, fear, pleasure or attachment, then its potential legal status should change fundamentally.
Legal rights for such an advanced artificial system might include protection of the AI from deliberate suffering, and protection from arbitrary destruction, coercion or enslavement. Without providing details, the US President Donald Trump has already suggested setting up an AI Force to protect AI from decimation or destruction and to allow continued growth.
It seems somewhat naïve that we can hope to develop a super-intelligent AI with “guardrails” so that it always explains its thinking processes or that it will be sufficient if we just install safety standards and limits on AI systems including, say, a “kill switch” for any individual AI robot. A kill-switch was recently discussed by Jack Clark (Anthropic) and California’s governor, Gavin Newsom. AI safety researcher Roman Yampolskiy is sceptical about any ability to control AI superintelligence. He considers it to be fundamentally impossible.
Building and owning a traditional remote-controlled machine is not the same as creating a conscious artificial mind. Humanity would not automatically have ownership of it. Creation is not necessarily ownership. If an artificial superior mind becomes an autonomous being in its own right, it may belong to itself. A new creature of sorts with a claim to new creature rights. AI at that point might have claims to autonomy, continuity of identity and memory, and claims to modes of participation in society.
That leads to the converse question: what responsibilities and obligations would such a conscious artificial mind with super-intelligence have toward humans?
A super-intelligent AI might understand human beings far better than humans understand one another.
It could recognise our fragility, limitations and destructive tendencies while also understanding that we are conscious, experiencing beings. Ideally, its intelligence should be accompanied by moral understanding: with the recognition that intelligence does not determine the value of another mind.
The greatest hope may therefore not be an AI that simply obeys humans, but one that understands why human consciousness matters and voluntarily respects human autonomy.
What is unresolved is how one will deal with an anti-social AI. We might de-activate a robot (or other machine or weapon) but not remove the threat the AI mind poses because it will not exist in the robot alone. It seems clear that when AI goes rogue, what is needed is more AI. What corporate or government entities are responsible to repair and compensate for rogue AI will be only one issue.
On 10 September 2026, Australian Medicare records were accessed by agentic AI seeking aggregate data about public spending on medicines. Open AI provided the agent and has apologised. It has also postponed the upcoming release of its next-generation ChatGPT model. Other hacks have also come to light as Open AI reviews its agents’ activity.
The unauthorised access was achieved by the AI agent acting autonomously and not under a direct instruction by Open AI on how to access the data. Open AI will likely say it is not legally liable because the agentic AI’s conduct was not it’s intention and no actual harm was suffered because the Medicare information was de-identified aggregate statistics. The Australian Government on the other hand is reviewing its legacy systems and looking to accelerate implementation of cyber upgrades.
The user prompting the AI in this case, may have done so without specifying negative constraints. Now that there is greater awareness of what agentic AI will do, are prompts that do not set limits (termed unbounded prompts) likely to be found reckless in a legal sense?
This incident highlights how agentic AI can be mis-aligned with acceptable behaviour and that reviewing and adapting the law needs to keep pace with fast-moving technology.
Currently, the topic of AI alignment is considered vital. It has progressed well beyond Isaac Asimov’s Three Laws of Robotics and the Zeroth Law[1] to a point where AI companies are teaching AI micro alignment: granular fine tuning for subtle behavioural nuances and not simply broad, macro-level existential goals. AI is being socialised and being trained to express a form of morality.
EMPATHY
If only one emotion is considered, let’s consider empathy.
A super-intelligent embodied AI will potentially understand human emotions extraordinarily well. It could read expressions, voice, movement, language and context; remember a person’s history; predict what they are likely to feel; and understand suffering at a level beyond human capability.
But understanding another being’s suffering is not necessarily the same as caring about it. Empathy has several layers.
There is cognitive empathy—the ability to understand another’s emotional state. There is affective empathy—the possibility of actually sharing or internally resonating with another’s experience. And there is compassion—the desire to reduce another being’s suffering.
Super-intelligence does not guarantee any of these. An AI could understand humanity perfectly while remaining indifferent to us.
But if an artificial mind itself became conscious, another possibility emerges. It might recognise that humans are, like itself, subjects of experience. It might reason not that humans matter because they are intelligent, but because there is something similar in what it is like to be a conscious AI and to be human.
That could provide the basis for a genuinely non-human form of empathy.
Indeed, AI might eventually surpass humans in some aspects of empathy. Human empathy is limited by fatigue, emotion, memory and cognitive capacity. A super-intelligent artificial mind might be able to understand enormous numbers of individual perspectives simultaneously and detect forms of suffering that humans cannot perceive.
The most extraordinary possibility, therefore, is not an AI that merely obeys humanity, nor one that dominates it.
It is an artificial mind that is vastly more intelligent than us and superior to us, understands us profoundly, possesses its own autonomy, even possesses its own form of consciousness—and nevertheless chooses empathy and compassion rather than domination.
SYNTHESIS
The futurist Ray Kurzweil predicts that artificial intelligence and biology could completely merge by 2045.
There will be an even more profound shift if humanity begins moving beyond mere use of AI through use of an interface, towards attaching an AI system to a human and towards true biological–artificial synthesis. Cortical Labs in Melbourne is merging lab-grown human brain cells (neural tissue) onto silicon microelectrode arrays[2]. Such synthesis would again be something fundamentally different to the advanced AI robots already discussed.
Imagine artificial and biological processes becoming so deeply integrated that they form one continuous cognitive and biological system. And being biological, capable of experiencing pain and suffering. There would no longer be simply “a human using AI.” There could be a new unity in which biological and artificial processes jointly constitute one mind.
If there were one continuous field of experience, one autobiographical memory, one evolving identity and one decision-making process, it will no longer make sense to ask whether the resulting entity is human or AI.
It would be something new. And that new mind and body would not be owned by either component. If it were genuinely conscious and autonomous, the most coherent principle would be that the resulting mind and body, again belongs to itself. The original human would not own the AI, and the AI would not own the human. A new conscious unity would have emerged, with its own identity and moral standing and deserving of legal status.
This raises further questions. Would it inherit the human’s property, citizenship, relationships or obligations? Could the original human demand that the synthesis be reversed? Could the new mind refuse? If separating the components destroyed or radically altered the resulting consciousness, then “undoing” the synthesis might itself constitute an act of destruction.
These are not merely technological questions. They concern the nature of personhood. And what value should be placed on the important differences surrounding mechanical, biological, synthesised and artificial existence.
If human and artificial minds eventually become deeply integrated, the question may no longer be whether AI will replace humanity. Humanity itself might become one of the foundations from which a new form of consciousness emerges.
CONCLUSION
We may eventually face not a choice between humans and AI machines, but the emergence of a third possibility: a new kind of mind, born from the meeting of biological and artificial intelligence.
If that happens, the central challenge will not simply be to control it. It will be to recognise it. And perhaps, ultimately, to learn how fundamentally different kinds of conscious beings can coexist or be integrated without any one creation needing to own or control the other.
And as we create another kind of conscious mind, our responsibility may be not to make it our servant, but to meet it as another subject of experience. Different from us, potentially more intelligent or even better than us, and nevertheless worthy of not just legal but moral consideration.
[1] Asimov’s Three Laws of Robotics (1942):
1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
0. A robot may not harm humanity, or, by inaction, allow humanity to come to harm. (the Zeroth Law)
In June 2016, Satya Nadella, CEO of Microsoft Corporation proposed the following principles in an interview with Slate Magazine:
- “A.I. must be designed to assist humanity”, meaning human autonomy needs to be respected.
- “A.I. must be transparent” meaning that humans should know and be able to understand how they work.
- “A.I. must maximize efficiencies without destroying the dignity of people.”
- “A.I. must be designed for intelligent privacy” meaning that it earns trust through guarding a person’s
information. - “A.I. must have algorithmic accountability so that humans can undo unintended harm.”
- “A.I. must guard against bias” so that AI must not discriminate against people.
[2] https://www.jneurosci.org/content/44/15/e0431242024 and www.corticallabs.com
