Heather Fricke is the founder of Frick-E Energy™, where her work focuses on the Human-AI Understanding Gap: what happens to human meaning, context and judgment when machines interpret information before people or organizations act on it. She is the creator of Promptology™ and Human Algorithm™.

AI systems are becoming extraordinarily good at finding patterns in what already happened. That is useful right up until the past starts quietly functioning as permission for the future.
Historical data can be accurate and still carry the imprint of old policy, unequal access, institutional habits, earlier automation, missing choices and feedback loops. A model can learn those patterns faithfully without understanding why they existed. Once that happens, “what happened before” can slide into “what should happen next” without anybody ever making that decision explicitly.
That is the governance problem hiding inside many conversations about responsible AI.
The problem is not history itself. Historical evidence matters. The problem is the authority we give it.
A useful AI system should be able to learn from the past without treating the past as destiny.
The distinction matters most when AI starts supporting consequential decisions in health care, lending, hiring, policing, insurance, public services, education and other environments where prior outcomes were shaped by more than neutral demand and supply.
A historical pattern can be statistically real while still being a poor prescription.
The classic health care example is instructive. Researchers found that a widely used population-health algorithm underestimated the needs of Black patients because it used historical health care spending as a proxy for health need. The spending data were real. The proxy was measurable. The failure came from the meaning attached to the number.
The system did not need fabricated data to create a distorted decision. It only needed a historical measure whose relationship to human need had already been shaped by unequal access.
That is why provenance has to mean more than “where did this data come from?”
For consequential AI, provenance also has to include: what conditions produced this outcome, what incentives shaped it, what policies were active, what alternatives were unavailable, what prior systems influenced it, and whether the same conditions still exist.
Without that context, historical data can become institutional memory with amnesia.
The second problem appears after deployment.
Once AI begins influencing decisions, its outputs can help create the next generation of data. A risk score changes who receives scrutiny. A recommendation engine changes who receives attention. A triage system changes who receives follow-up. A hiring filter changes who reaches an interview. Those altered outcomes then return as “new data.”
At that point, the system is no longer merely predicting reality. It is participating in the production of the reality that later becomes its evidence.
That creates a feedback-loop problem.
If organizations do not measure that loop, a system can gradually validate itself against consequences it helped produce.
Predictive-policing research has illustrated the danger clearly: when deployment decisions influence where incidents are observed, future models can interpret that concentrated observation as evidence that the original deployment choice was correct. The loop becomes self-reinforcing even if the underlying distribution of events was different.  The same logic can appear in less dramatic settings.
An AI system recommends one supplier more often, so that supplier accumulates more transaction history. A talent system routes certain candidates forward, so the future “successful employee” dataset increasingly resembles the people the system already preferred. A customer-support agent escalates some issues and resolves others automatically, changing which cases later appear to require human intervention.
The output becomes part of the input.

Responsible AI needs a simple but demanding rule

Responsible AI, therefore, needs a simple but demanding rule: evidence created after system deployment should not automatically be treated as independent confirmation of the system’s quality.
Organizations need to know which outcomes were observed and which outcomes were partly system-produced.
That requires measurement beyond aggregate accuracy.
A model can perform well overall while error is concentrated in a specific population, geography, use case or decision boundary.
Average performance can hide the people carrying the cost.
This is why distribution-sensitive evaluation matters.
Instead of asking only, “How accurate is the model?” teams also need to ask: “Where are the errors concentrated?”
Which groups experience materially different false-positive or false-negative rates?
Did performance change after the system began influencing the data-generating process?
Are historical patterns still valid under current conditions?
What contradictory evidence would be strong enough to force the system or the organization to revise an inherited assumption?
That last question is especially important.
Many AI systems are excellent at accumulating evidence in favor of an existing pattern. Far fewer governance processes are designed to ask what would count as enough contrary evidence to change the pattern.
A responsible system needs room for new evidence to exist.
That means historical priors should not become permanent ceilings. Organizations need explicit evidence-revision rules: when current conditions materially contradict inherited patterns, the system must allow those assumptions to be challenged rather than quietly absorbing the contradiction as noise.
This is not an argument for throwing out historical data or forcing mathematically identical outcomes.
It is an argument for treating history as evidence with provenance, not authority without limits.
The practical governance boundary is straightforward.
AI can retrieve the past. It can summarize it. It can compare patterns. It can estimate probabilities.
But when those patterns begin to shape what should happen next, somebody has to own the transition from description to prescription.
That is where human accountability belongs.
Not as a ceremonial “human in the loop” checkbox after the system has already framed the decision, but as a real authorization point where the organization can ask whether the historical evidence still deserves the authority the model is giving it.
The strongest AI systems will not be the ones that remember the most.
They will be the ones that know the difference between history, evidence and permission.

R$