Noga's Bogus

Prelude.

It is obvious that I do not know anything. That statement is probabilistically true, given the knowledge that is to be is a set ( obviously uncountable ) and my knowledge is obviously is countable (finite). I do not even have any degree to talk about computation in general, clearly a B.E. does not count. My job is typing in software like a clerk and thus, none should actually take me seriously.  This post is not my job, it is just a lament.

Theory of Epicycles.

There was an era, where people do actually believe that the earth is fixed, and sun is rotating around the earth. So was all the other planets. This is known as geocentric model. And, it beautifully worked. Till, people figured out there are issues. There were motions which could not be described, mars would go to a direction and then apparently for no reason changed it’s direction in the sky and change it again. Thus the theory of epicycles was born to the most scholastic mind. Now, after thousands of years, we know, for certain, they were plain wrong ( is it? Not entirely sure – see more on Occam’s Razor ).

To the Point : Onto Computation & Machine Learning

Is there a point to it? Yes. Einstein disliked Quantum Physics. Because, there is no mechanism to it, and it is mambo jumbo. It does not really ask what is happening inside, and replace it with observables. The same pattern exhibited by Ptolemians, thousands of years ago. But, a bigger badder folks are into it now. The folk of Machine Learning. Apart from the very front runners and the very starters of the machine learning ( then called A.I) none bothered to check what computers can actually do or rather they are doing.  For those, who are late into the game of understanding, by Church Turing thesis , a standard computer can not exceed  the disabilities of a Turing Machine. That invariably means, nothing running on a computer or computer alike ( any combination of  Neural Networks ) will suffer from the same disability as that of a Turing Machine. Hyercomputation  is not possible with NN(Neural Network)s. They would suffer the same fate as that is a TM (Turing Machine). But, none of the machine learning folks seem to be bothered about the TM nature of actual computation. They are much bothered about applying non applicable continuum mathematics ( read arithmetic on Real ) when they don’t even care that the algorithm  they are so much glorifying can not even run on a real computer. TMs work only on Natural Numbers.

The Halting Problem

When a computation ends in a Turing machine? When given an input, it halts. It is well known that the set of inputs for which a Turing machine halt is a null set. Worse, even for all integers, the set on which a Turing machine would halt still has no density. The summary, TM’s almost surely, never halts. So, when it does not halt, if you inspect the tape of the turing machine, from computation to computation, how does it look like?

A Tape of a Turing machine comprise of symbols, drawn from a finite set. That is called the alphabet of the Turing Machine, loosely. Now, given finite symbols exists, each symbol can be taken to be as a digit, and immediately Godelization becomes possible. Thus, at each step of computation, the tape of the TM can be taken as nothing but representing a number. Hence, a TM going through computation looks like :

T1 -> T2 -> T3 ->T4 …

And that generates the sequence of numbers. That is the sequence.  They are all natural numbers, or rather, if you use our formulation, they are all Rational numbers between 0 to 1 ( imagine there is a decimal point in the left end of the Tape of the Turing Machine ).

Thus, the question is, when TM does not halt, what is the behaviour that is exhibited by the sequence of numbers, appearing on the tape of the TM?  We have proven that the sequence of numbers showcases chaos.  But where machine learning jumps in?

The Machine Learning Problem

Can a program can learn? *Learn* here means, can a program, given input, produce another program, that satisfactorily generates solution to some very specific problem?

Sorry, this is too vague. But two things are certain. A program, trying to create another program which is the result of the computation. Thus, like any other computation, the TM’s tape has the result as the program. Thus, the program we are so much after, gets generated as nothing but a string of symbols or rather a rational number on the tape of the *learning TM*.  Hence, the step by step running of even any machine learning algorithm would generate similar sequences like the earlier :

L1 -> L2 -> L3 -> L4 …

So, when we say, the learning algorithm *converge* what do we mean? The convergence can mean, any of the following :

1. On this space, the algorithm is producing sequences which are converging like an ordinary metric space with a distance metric |Li – Lj| treating L’s like rational numbers.
2. On the recognition space. This is non trivial, and axiomatically one needs to define a metric for closeness criterion. A suitable metric can be Noga metric, which says ratio of successful classification to total sample size. That is a joke. The point is, the success criterion on the recognition space has no real meaning on the TM’s tape space and vice versa.

But any error minimisation would now directly hurts the sequence in the L (TM’s tape ) space. Distant points in the L space can be very close in the recognition space. Distant point in the recognition space can be very close in the L space. That, and this, is a typical hallmark of chaos, or rather what is called, the mapping between L to Recognition space, is chaotic. But let’s drop that. That is too much impractical math. Let’s do some borderline practical and then very practical basic computation.

Cantor Set and Deep Learning

There is this nice thing called Cantor Set.  You take any line segment, and then select the first 1/3 rd, then reject the next 1/3rd and then select the last 1/3rd again.

Now think like this, Cantor Set is actually doing primitive deep, very deeeeeeeep learning. The basic computation blocks are 2, accept ([0,1/3] , [2/3,1]  )  and reject (1/3,2/3) . Purely dumb, but then it recursively performs the same computation again and again over partial inputs. It is wonderful, that it took thousands of years to imagine this for mankind, and worse that even after looking at it, thousands of smart guys, none saw the similarity between this and computation done till infinity. That is a shame for the Computer Science. The space, or rather the computed result of the Cantor Set Computation is Cantor Set, and is a Fractal.  It converges, to a …. news flash : Fractal.

The initial condition are : of course, 1/3 , and tri partition. If you change it, the resulting fractal would be absolutely different than the Cantor Set, albeit having same criterion, same characteristics.  It can be easy shown that given a finite set of selection and rejection logic, when recursively applied to infinity, the set isolated would be Cantor like, a fractal.

Thus, arbitrary back propagation, which is equivalent to a loop running and fixing itself again and again – generates and isolates, a fractal like structure, if the day is bad for it. By invoking density and measure theory, one can easily argue and prove that almost surely any recursive, till infinity computation would only isolate fractals. There are no fixes but stopping at whim. But there is no guarantee that by that time, the system was already in the path to the chaos or not. By increasing more layers, by fixing more errors, only chaos they are ever going to get. The system thus learned, would catastrophically fail to recognise  simple other inputs, and only a local fix would remove that failure. Because you are not at continuum any more, you are in the realm of discrete, and no matter how much one likes continuum mathematics, those are not computable, ever.  They are not counter intuitive, they are what they are. Like every function is almost surely discrete, like every continuous function is almost surely non differentiable, almost every machine learning method must isolate fractals, deep learning, more so due to it’s nature.  That is why those counter intuitive properties were shown long back in the field of vision, and is appearing more and more in the field of deep learning.

Epilogue

I am not going to defend the viewpoint, ever again, nor when it emerges victorious I would claim that we said so. Human achievements are meaningless in the temple of the Truth. There is a proverb: It is fine to make a person awake when they are actually sleeping. It is impossible to make a person awake when he is simply posing as sleeping. Accepting this viewpoint, at the heart of the computation would be taken as computational catastrophe, the simple, step by step, deterministic approach to reality is breaking down into chaos, literally. Thus, none would accept it, ever. But then, as the final line of Count of Monte Cristo goes: “The only thing humans can do, is to wait.” Einstein said “there is no reason to save the truth using sword.” We, from the second oldest civilisation of  the world taught from the very first days of our life : “Truth Will be Victorious” . And the same taught us : “There is no lie, only lesser Truth.” In that note, I would conclude: “Jato dharmah, tatoh jayah.” as Gandhari did not grant Duyodhana’s boon.