Demystifying AI: The Myth of the Black Box
How progress in interpretability is revealing the inner workings of artificial intelligence
The Misconception of AI as “Black Box”
The term “black box” has been used in AI to describe the opaque nature of neural networks. The idea here is: while we know what we feed the neural network (“input”) an observe what comes out of the neural network (“output”), we seemingly have no idea what’s happening in between the two — in other words, we can’t observe by which rules the input gets transformed to an output.
This opacity wasn’t always the case. In the infancy of AI, systems were simple enough that researchers could trace their decision-making processes easily. But, as my dear readers know, neural networks grew from thousands to millions to billions of parameters (a parameter being a single “node”, “neuron”, or “weight”, or whatever you want to call it — the smallest unit of data processing inside the network, notoriously inspired by the human brain). Thus, complexity skyrocketed, and interpreting the way the network works became drastically obscured.
