I'd argue that deep networks, seen as recursive generative linear models, owe much of their success to increased computational power but also to the transformations (RKHS) and optimization developments. On the flip-side, GPU makers are rewiring their hardware to make deep nets go. Exciting times ahead!