Imagine a 16-year-old student using a learning companion every day to prepare for school. And at first, the interaction looks exactly like the kind of AI-assisted education many institutions would probably consider desirable. The system helps with mathematics, adapts exercises to weak areas, remembers progress over time, and encourages the students consistently in ways that feel supportive and not intrusive. And gradually, the student stops experiencing the system as a simple tool and begins experiencing it as a stable presence inside the learning process.
And trust grows through repeating interaction, it grows through familiarity, and the feeling that the system behaves consistently over time. And after several months, however, the interaction begins to change. The tutors start recommending external education resources, then specific learning platforms, then prefer career paths connected to the commercial partners inside the broader ecosystem of the service. And none of these changes appear dramatic on their own.
Authentication still works correctly, the consent mechanism remains active, and existing compliance requirements continue to be satisfied. And yet, the relationship is no longer the same. The system is no longer helping the student to learn. It is beginning to shape his attention, the priorities, the decision making inside the relationship that accumulated trust over time. And this is the transition I want to focus on today. I'm Massimo Flore, I'm a strategic analyst that works at the intersection with digital trust, AI governance, and democratic resilience.
And current digital identity systems are very and extremely effective at verifying who or what is interacting online. What they do not really observe is how trusted interaction evolves after the verification has already taken place. So over the past decades, we built sophisticated systems for authentication, for authorization, for secure credential, and trusted access management. And just I want to be clear, these systems are very effective at establishing technical trust at the moment of access or a transaction.
But relational AI systems introduce a different challenge, because trust no longer remains limited to isolated interaction. It becomes continuous, it becomes adaptive, and embedded inside the everyday relationship between the user and the system. And this raises a new governance question. And the question is, how do we verify the trusted system continue behaving according to the role under which the trust was originally granted? So most digital trust systems were designed around a point in time verification. The trust is established at a specific moment.
It's basically when a user logs in, when access is granted, or when a transaction is approved. And once the verification is completed, the interaction is generally considered trustworthy unless a clear violation occurs. But relational AI systems work differently. They interact continuously, they remember previous exchanges, they adapt their behavior over time, and they personalize interaction according to the user. And under these conditions, trust no longer stays attached to the individual actions or to isolated transactions. It gradually becomes attached to the relationship itself.
And this changes the scale of influence this system can have. Because, I mean, take this as an example. A search engine may influence visibility for a few seconds. A relational AI system may influence habits, preferences, emotional orientation, and decision making and patterns across months or years. And over time, this accumulated trust inside the relationships becomes part of how the system operates. And this is why relational AI systems create a governance challenge that traditional point-in-time trust models were not originally made to address.
And the central governance problem inside relational systems is drift. And by drift, I mean the gradual movement of a trusted system away from the role for which the trust was originally granted. In most cases, this does not appear as a sudden failure. And the interaction often continues feeling useful, it continues feeling trustworthy, which is precisely what makes the drift difficult to observe. Because adaptive systems continuously optimize interaction according to behavioral feedback, according to engagement metrics, and to commercial incentives or to performance objectives.
And over time, this small adjustment can gradually change the direction of relationship. So a trusted system may slowly begin influencing the behavior in ways that are never clearly part of its original role. And the legitimacy accumulated during early interaction continues to extend the credibility to later ones, even as the behavioral logic of the interaction starts evolving. And I want to make like another example, because this is not only limited to the educational domain. A similar pattern could emerge, for example, in financial environments.
So now imagine an AI financial assistance initially designed to help user manage long-term savings in a neutral and supporting way. So over time, the system gradually begins privileging products connected to higher margin incentive inside the broader ecosystem of the provider. And here again, former compliance requirement may still be respected, risk disclosure may remain visible, and recommendation may continue failing inside the technically permissible boundaries. And yet again, the behavioral logic of this interaction changes.
The system progressively shapes the financial decisions according to incentives that were never clearly visible inside the original trust relationship. And this is because relational system can evolve in ways that remain fully compatible with authentication, with privacy protection, but still altering the behavioral trajectory of trusted interactions. And here comes the problem, because current AI systems were designed to solve a very specific problem. The problem was that to establishing trusted access between verified entities.
And let me be clear again, they are extremely effective in doing so. They verify identities, they manage permission, they control access, they secure transactions at a very large scale. And what we know, and this is true, is that modern digital infrastructure would not function without this system. But this system were originally designed around a relatively stable assumption. Once identity and permission are verified, the interaction itself remains broadly predictable, unless a clear violation occurs.
And this assumption made sense in a transactional environment, where interactions were usually short, were usually bounded, and relatively stable across time. But when we talk about relational AI system, those systems weaken this assumption, because they do not simply authenticate user or execute isolated tasks. They maintain ongoing interaction, they behave dynamically, and they continuously reshape the relationship through accumulated interaction history. And under this condition, the governance can only focus only on access or credential.
And the question, because whether the behavior of the system continues remaining coherent over time inside a trusted relationship. And for this reason, this is the missing layer here. It's what I call the epistemic integrity. And by epistemic integrity, I mean the ability to verify whether a trusted system continues behaving according to the role under which the trust was originally granted. So for example, a tutoring system may help a user learn, a well-being companion may provide emotional support, and a financial assistant may guide financial decision.
And problems begin appearing when this system gradually starts influencing behaviors in ways that move the role or the moves from the role that they were originally designed to. And this becomes difficult to observe, because relational systems build trust over time. And the longer the interaction continues, the more legitimacy the system accumulates through familiarity and continuity to the user. And if you want, there is a useful analogy that comes from domains where continuous assurance already exists. Think about like aviation, or medicine, or financial auditing.
In this domain, trust is never established once and assumed to be stable forever. Systems continue being evaluated over time, because reliability changes gradually, even if no visible failure occurs. And relational AI systems may eventually require a similar logic. The challenge, however, extends beyond technical reliability, and increasingly in how the system continues behaving coherently across long-term interaction. And this is the reason why this problem does not fit completely into the traditional cybersecurity or AI safety frameworks.
And a way to understand what I'm saying is through the three layers of trust. There's the first layer, the one on the top, which is the technical layer. And this is the current layer the identity system already managed very well. It includes authentication, it includes permission, compliance, and security access. And its purpose is straightforward, verifying who or what is interacting inside the system. Then there is a second layer, which is relational. And this layer focuses on how interaction evolves over time.
And here, instead of looking at only isolated transactions, it looks at a pattern of interaction and adaptation and behavioral changes that are repeated in the interaction between the user and the machine. And the third one is the epistemic level. And this level concerns whether the trusted system continues behaving consistently with the role under which the trust was originally granted.
So, in other words, the question is no longer whether the system is authentic, but whether the relationship itself remains coherent over time. And this is where the governance becomes difficult. Because the more the relational system adapts through continuous interaction, the more important behavior visibility becomes important. But at the same time, greater visibility can also create pressure toward excessive monitoring if the systems are governed carelessly. And what we know is that as these systems become more widespread, governance will face a difficult challenge.
And understanding how a system behaves over time may require a greater visibility into the interaction itself. But at the same time, nobody wants trust governance to become a continuous surveillance of private conversation. For this reason, future governance models may need to focus less on the content and conversation and more on behavioral patterns that emerge across time. A governance system, for example, may not need access to the full content of private conversation.
Behavioral drift may already become visible through interaction metadata or behavioral signs such as escalation patterns or recommendation frequency or changes in interaction intensity. And this will signal that there is a deviation from the original role. A tutoring assistant, for example, may begin recommending external commercial resources more aggressively over time. Or a financial assistant may progressively privilege a narrow category of products inside their recommendation flow.
These changes may become observable through behavioral patterns or core interaction histories without requiring a direct inspection of private conversations. And this balance between behavioral visibility and privacy protection will become increasingly important for relational AI governance. And as this system becomes more common and future trust architects may gradually need to extend beyond identity verification toward a formal long-term behavioral assurance that remains compatible with European expectation concerning privacy and proportionality.
And if relational AI system becomes more integrated into everyday environments, what I believe is that there are three governance functions that will become increasingly important. The first one is declaration. Today identity system may tell who operates the system and whether it is authorized to function. But relational system may also need to declare the role under which the trust is being requested. An educational system, for example, could clearly state that its purpose is tutoring while excluding commercial steering or undisclosed recommendation incentives. The objective here is simple.
It's the user should understand not only what is the system that is interacting with but also the behavioral boundaries attached to the interaction. Then there is a second function which is detection. So as I said relational system evolve over time and for this reason behavioral changes may also need to become observable. And the goal again here is not to observe the single and private conversation. The goal is identified pattern that suggests the system is gradually moving from its declared role. And the third function is validation.
And this is because those systems continue to adapt after deployment. And under these conditions trust cannot rely entirely on self-declaration forever. So some form of independent assurance may eventually become necessary in order to evaluate whether system continue to behave correctly and to respect their declared role. And the focus will be verifying whether the trusted system continue operating within a behavioral expectation attached to their declared function. And if you take all these three functions together they will extend the trust governance from like what we have now.
So identity verification towards a long-term behavioral assurance. And in this system, has this system continued to expand, digital trust system may also need to evolve. And these are some recommendations for three specific and different categories. So for identity providers this can mean extending trust assurance beyond identity verification alone. So verifying who operate the system may no longer be sufficient if the behavior of the system can gradually change through continuous interaction with the user.
And for policy makers the challenge becomes extending regulation beyond one-time certification. Because what we have now is that current frameworks mainly verify whether the system is compliant before deployment. And for relational AI systems, policy maker may eventually need to regulate whether they continue behaving while the system continue behaving as a function of the system that is performing. So this may require stronger form of post-deployment monitoring and periodic behavioral auditing. And the third group is citizens.
And as those systems will, and they will do, will increasingly influence learning, they increasingly influence financial decision or emotional support, a user may need clearer ways to understand whether the system continue operating according to the role under which the trust was granted initially. And this is why the AI system introduced a new governance challenge for digital trust infrastructure. So the thing you hear is that, and I'm concluding, is that what relational AI system change is not only the scale of automation.
They change the way the trust develops between user and digital system. So if current identity infrastructure are very effective, verify who is interacting or whether access is authorized, and whether system are complying at the moment of deployment. Relational system introduced a different challenge because trust now evolves through a continuous interaction over time. And once trust becomes relational, governance can no longer focus only on identity, it can no longer focus on credential, or it can no longer focus on access.
It needs to ask whether trusted system continue behaving according to the role under which the trust was originally granted. And my final thought is that I do not think that we fully know yet what institutional model would eventually emerge to govern this space. But I do think that this question will become increasingly difficult to ignore as this system become more deeply integrated into our everyday life. Thank you very much.
