Here's a variant system that might be easier to analyze, because it lends itself to experiment. Suppose the observer, still seeing only zeroes and ones, has a button he can press. The effect of the button, perhaps unbeknownst to him, is to deflect the ball through an angle of 0.01 degrees. Pressing the button once will affect the sequence but usually only after some time. For a while the ball will be bouncing off the same surfaces.
An observer who has been told or has formed the hypothesis that he is facing a roofs-and-boxes problem can locate the roofs and boxes simply but tediously in the case when the sides of the roofs an boxes are parallel to edges of the arena. He presses his button and waits a long time to see if the zeroes and ones form an approximately periodic pattern. If so he has a measure of the distance of a box from the edge and the amount of overhang. If not he moves on until he has such a pattern. After analyzing one such pattern he moves on till he finds another. Eventually he will get the pattern of roofs and boxes and can predict the future.
How clever must one be to hypothesize that it is a roofs-and-boxes problem? It is a problem of scientific creativity. A scientist might fiddle for a long time before coming up with the hypothesis. Donald Michie said that a good cryptanalyst would probably succeed.
How hard would it be to write a program that could honestly discover the roofs-and-boxes theory?
Roofs-and-boxes illustrates the idea that even to extrapolate experience a robot must know or learn about phenomena in the world. Learning programs have to discover phenomena in the world and not just patterns in the data. To put the matter in another way, important patterns in the data usually take the form of observations of phenomena in the world.