Why back propagation goes backward

[delayed]

The real reason is that backprop is basically matrix multiplication and multiplying from left to right is way cheaper from right to left. Since on the left side you will have a scalar loss term and you keep vector - matrix multiplication through the network instead of doing matrix by matrix multiplication from the right side.

So basically it's for the same reason that tup is fast: if the arrows go up, you need far less arrows!