Turns out it was a bug. Using the nomenclature of the book I was feeding h into the gradient function when I should have fed a into it. With the >= comparison this made all the gradients 1 and thus it acted like the linear activation function. (The linear activation function does produce about 94% accuracy.) Properly using the gradient function produces the expected results. It doesn’t matter if one uses > or >=.
I am happy I found this bug. But this is also part of why your book is so great. Programming it yourself forces one to understand the little details and allows one to change and modify the algorithms at the very core, which leads to much deeper understanding of how this all works.
Here is an insight that my experimentation produced. I tested a bunch of different activation functions including weird piecewise linear ones, periodic ones with sin and cos, combinations thereof etc. It surprised me that many work just as good as ReLU or sigmoid with a single hidden layer. (I intend to extend this experimentation to multiple hidden layers.) For example, it is kind of shocking at first that the absolute-value-function works just as good as ReLU. This kind of makes sense in the biological case. A neuron being a cell would not be completely identical to its neighbor neuron. Neurons in nature would certainly have different activation functions. Perhaps not as different as I experimented with but they would perhaps be noisy and distorted versions of sigmoid or ReLU. It doesn’t matter, it still works fine.
Further, this makes me wonder if perhaps that variation in activation functions in nature is a benefit. I am wondering if folks have tried to make nets where each activation function of each neuron is different. Perhaps that confers a training advantage to the network because not everything behaves in exactly the same way? I will try to explore this question. But first I need to extend the code to allow for multiple hidden layers.
This is one critique I have to make. In my opinion, it would have been better to go further with the code and extend it to multiple hidden layers than to switch to libraries. The point of the book is programming it yourself to allow full unmitigated experimentation. I would have added one or two chapters to extend the code further even if that would have meant leaving out libraries altogether. Numpy should be fast enough to explore multilayer networks on a single average computer.