Deep learning (DL) is transforming whole industries as complicated decision-making processes are being automated by Deep Neural Networks (DNNs) trained on real-world data. Driven by an expanding literature on DNN approximation theory showing that DNNs can approximate a rich variety of functions, these tools are increasingly being applied to scientific computing. Yet, unlike traditional algorithms, little is known about DL from the principles of numerical analysis, e.g., stability, accuracy, efficiency, and sample complexity. This work introduces a computational framework for examining DNNs with regard to these issues. We study DNN performance on a variety of smooth and piecewise smooth test functions in various dimensions. We also compare DL against best-in-class methods based on compressed sensing. Our main conclusion is that there is a crucial gap between the approximation theory of DNNs and their practical performance, with trained DNNs performing relatively poorly on functions for which there are strong approximation results (e.g. smooth functions), yet performing well in comparison to best-in-class methods for other functions. Finally, we present a novel practical existence theorem, asserting the existence of a DNN architecture and training procedure offering the same performance as current best-in-class schemes. This result indicates the potential for practical DNN approximation, and the need for future research into architecture design and training strategies.