As scientists push the boundaries of machine learning, the amount of time, energy and money needed to train increasingly complex neural network models is skyrocketing. A new area of artificial intelligence called analog deep learning promises faster computation with a fraction of the power consumption.
Programmable resistors are the key elements of analog deep learning, just as transistors are the central elements of digital processors. By repeating networks of programmable resistors in complex layers, researchers can create a network of analog artificial “neurons” and “synapses” that perform calculations like a digital neural network. This network can then be trained to perform complex AI tasks like image recognition and natural language processing.
A multidisciplinary team of MIT researchers set out to push the speed limits of a type of artificial analog synapse they had previously developed. They used a handy inorganic material in the manufacturing process that allows their devices to run 1 million times faster than previous versions, which is also about 1 million times faster than human brain synapses.
Additionally, this inorganic material also makes the coil extremely energy efficient. Unlike the materials used in the previous version of their device, the new material is compatible with silicon manufacturing techniques. This change has made it possible to fabricate nanoscale devices and could pave the way for integration into commercial computing hardware for deep learning applications.
“With this key idea and the very powerful nanofabrication techniques that we have at MIT.nano, we were able to put these parts together and demonstrate that these devices are inherently very fast and work with reasonable voltages,” says lead author Jesús A. del Alamo, Donner Professor in the Department of Electrical Engineering and Computer Science (EECS) at MIT. . “This work has really brought these devices to a point where they now look really promising for future applications.”
“The device’s operating mechanism is the electrochemical insertion of the smallest ion, the proton, into an insulating oxide to modulate its electronic conductivity. Because we are working with very thin devices, we could accelerate the movement of this ion using a strong electric field and push these ion devices to the nanosecond operating regime,” says lead author Bilge Yildiz, Professor Breene M. Kerr at the Departments of Nuclear Science and Engineering and Materials Science and Engineering.
“The action potential in biological cells rises and falls with a time scale of milliseconds, because the voltage difference of about 0.1 volts is limited by the stability of water,” explains the lead author. Ju Li, Battelle Energy Alliance Professor of Nuclear Science and Engineering and Professor of Materials Science and Engineering, “Here we apply up to 10 volts through a special nanometer-thick solid glass film which conducts protons, without permanently damaging it, and the stronger the field, the faster the ionic devices.
These programmable resistors dramatically increase the rate at which a neural network is trained, while dramatically reducing the cost and energy required to perform that training. This could help scientists develop deep learning models much faster, which could then be applied to uses such as self-driving cars, fraud detection or medical image analysis.
“Once you have an analog processor, you won’t be training the networks that everyone is working on. You will train networks with unprecedented complexities that no one else can afford, and thus vastly outperform them all. In other words, it’s not a faster car, it’s a spaceship,” adds Murat Onen, lead author and post-doctoral fellow at MIT.
Co-authors include Frances M. Ross, Ellen Swallow Richards Professor in the Department of Materials Science and Engineering; post-docs Nicolas Emond and Baoming Wang; and Difei Zhang, an EECS graduate student. The research is published today in Science.
Accelerate deep learning
Analog deep learning is faster and more power efficient than its digital counterpart for two main reasons. “First, the computation is done in memory, so huge data loads aren’t transferred from memory to a processor.” Analog processors also perform operations in parallel. If the size of the matrix increases, an analog processor does not need more time to perform new operations because all calculations occur simultaneously.
The key element of MIT’s new analog processor technology is known as the programmable proton resistor. These resistors, which are measured in nanometers (one nanometer equals one billionth of a meter), are arranged in a network, like a chessboard.
In the human brain, learning occurs due to the strengthening and weakening of connections between neurons, called synapses. Deep neural networks have long embraced this strategy, where network weights are programmed through training algorithms. In the case of this new processor, increasing and decreasing the electrical conductance of the proton resistors enables analog machine learning.
Conductance is controlled by the movement of protons. To increase conductance, more protons are pushed into a channel of the resistor, while to decrease conductance, protons are removed. This is accomplished by using an electrolyte (similar to that in a battery) which conducts protons but blocks electrons.
To develop an ultra-fast and highly energy-efficient programmable proton resistor, researchers turned to different materials for the electrolyte. While other devices used organic compounds, Onen focused on inorganic phosphosilicate glass (PSG).
PSG is basically silicon dioxide, which is the powdered desiccant material found in tiny bags that come in the box with new furniture to remove moisture. It is studied as a proton conductor under humidified conditions for fuel cells. It is also the best known oxide used in silicon processing. To make PSG, a tiny bit of phosphorus is added to silicon to give it special characteristics for proton conduction.
Onen hypothesized that an optimized PSG could have high proton conductivity at room temperature without the need for water, making it an ideal solid electrolyte for this application. He was right.
PSG enables ultrafast proton movement because it contains a multitude of nanometer-sized pores whose surfaces provide proton scattering pathways. It can also withstand very strong pulsed electric fields. This is critical, says Onen, because applying more voltage to the device allows the protons to travel at lightning speeds.
“The speed was certainly surprising. Normally, we wouldn’t apply such extreme fields to devices, so as not to turn them to ashes. But instead, the protons ended up shuttling at immense speeds through the stack of devices, specifically a million times faster than what we had before. And this movement does not damage anything, thanks to the small size and low mass of the protons. It’s almost like teleporting,” he says.
“The nanosecond time scale means that we are close to the ballistic or even quantum tunneling regime for the proton, in such an extreme field,” Li adds.
Because protons do not damage the material, the resistor can operate for millions of cycles without failing. This new electrolyte enabled a programmable proton resistor that is a million times faster than their previous device and can operate efficiently at room temperature, which is important for integrating it into computer hardware.
Thanks to the insulating properties of PSG, almost no electric current passes through the material when the protons move. This makes the device extremely energy efficient, adds Onen.
Now that they’ve demonstrated the effectiveness of these programmable resistors, the researchers plan to redesign them for high-volume manufacturing, del Alamo says. Then they can study the properties of resistor networks and scale them so they can be integrated into systems.
At the same time, they plan to study materials to eliminate bottlenecks that limit the voltage needed to efficiently transfer protons to, through, and from the electrolyte.
“Another exciting direction that these ion devices can enable is energy-efficient hardware to emulate neural circuits and synaptic plasticity rules that are inferred in neuroscience beyond analog deep neural networks. We have already started such a collaboration with the neurosciences, supported by the MIT quest for intelligenceadds Yildiz.
“The collaboration we have is going to be essential to innovate in the future. The road ahead will always be very difficult, but at the same time it is very exciting,” del Alamo said.
“Intercalation reactions such as those found in lithium-ion batteries have been widely explored for memory devices. This work demonstrates that proton-based memory devices offer impressive and surprising switching speed and endurance” “, says William Chueh, associate professor of materials science and engineering at Stanford University, who was not involved in this research. “It lays the foundation for a new class of memory devices to power algorithms deep learning.”
“This work demonstrates a significant breakthrough in bio-inspired resistive memory devices. These all-solid-state proton devices are based on exquisite control of atomic-scale protons, similar to biological synapses but at speeds an order of magnitude faster,” says Teddy & Wilton Emeritus Professor Elizabeth Dickey. Hawkins and head of the Materials Department. Science and Engineering at Carnegie Mellon University, which was not involved in this work. “I commend the interdisciplinary team at MIT for this exciting development, which will enable next-generation computing devices.”
This research is funded, in part, by the MIT-IBM Watson AI Lab.