To the extent that an agent is predictable, it must be:
- observable, and
- have a knowable internal structure
The first implies that the predictor has collected data emitted by the agent.
The second implies that the agent has internal structure and that the predictor has the capacity to represent the internal structure of the other agent.
In general, we can say that people do not have the capacity to explicitly represent other people very well. People are unpredictable to each other. This is what makes us free. When somebody is utterly predictable to us, their rigidity is a sign of weakness or stupidity. They are following a simple algorithm.
We are able to model the internal structure of worms with available computing power.
As we build more and more powerful predictive systems, we can ask: is our internal structure in principle knowable by this powerful machine?
This is different from the question of whether or not the predictive machine has data from which to draw inferences. Though of course the questions are related in their implications.