RUDN University mathematicians reduced neural network size six times without post-training

RUDN University mathematicians reduced neural network size six times without post-training

A team of mathematicians from RUDN University found a way to reduce the size of a trained neural network six times without spending additional resources on re-training it. The approach is based on finding a correlation between the weights of neural connections in the initial system and its simplified version.

The structures of artificial neural networks and neurons in a living organism are based on the same principles. Nodes in a network are interconnected; some of them receive a signal, and some transmit it by activating or suppressing the next element in the chain. The processing of any signal (for example, an image or a sound) requires a lot of network elements and connections coming from them. However, computer models have limited capacity and storage volume. To work with large data volumes, specialists have to invent different ways to lower capacity requirements, including the so-called quantization. It helps reduce the consumption of resources but requires system re-training. A team of mathematicians from RUDN University found out that the latter step could be avoided.

“Several years ago we carried out efficient and cost-effective quantization of weights in a Hopfield network. It is an associative memory network with symmetrical connections between elements that are formed following Hebb’s rule. In the course of its operation, the activity of the network is reduced to a certain equilibrium state, and when it is reached, a task is considered solved. The insights obtained in that study were later applied to feedforward deep learning networks that are very popular in image recognition today. As a rule, these networks require re-training after quantization, but we found a way to avoid it,” said Iakov Karandashev, PhD, an Assistant Professor at the Nikolskii Mathematical Institute, RUDN University.

The main idea behind the simplification of artificial neural networks is the so-called quantization of weights, i.e. reducing the number of bits per each weight. Quantization provides for the averaging of signal: for example, if it is applied to an image, all pixels representing different shades of the same color will become identical. Mathematically, it means that neural connections that are similar by certain parameters should have the same weight (or importance) expressed as a number.

A team of mathematicians from RUDN University carried out calculations and created formulae that effectively establish correlations between the weights in a neural network before and after quantization. Based on them, the scientists developed algorithms using which a trained neural network could classify images. In their experiment, the mathematicians used a text package of 50 thousand photos that could be divided into 1,000 groups. After training, the network was quantized using the new method and not re-trained. Then, the results were compared to other quantization algorithms.

“After quantization, the classification accuracy decreased by only 1%, but the required storage volume was reduced six times. Experiments show that our network doesn’t need re-training due to a strong correlation between initial and quantized weights. This approach could help save resources when completing time-sensitive tasks or working on mobile devices,” added Iakov Karandashev from RUDN University.

The results of the work were published in the Optical Memory and Neural Networks journal

All news
23 Jun
Scientists Reveal the Water Column of the Yamal Lakes Can Be a Microbial Filter

Scientists from the Winogradsky Institute of Microbiology RAS, RUDN University, St. Petersburg State University and the Tyumen Scientific Centre SB RAS studied the microbial communities from several lakes of the Yamal Peninsula. It turned out that methanotrophs (bacteria that use methane as a source of energy) consume methane more actively in the deep mature lakes of the peninsula than in small thermokarst lakes. In this regard, methane emissions into the atmosphere from the surface of deep lakes are low, and only small (relatively younger thermokarst lakes with constitutional ground ice) can make a significant contribution to methane emissions in the north of Western Siberia. Thus, bacteria perform an important function for the climate balance — they reduce the emission of methane into the atmosphere.

20 Jun
RUDN University Physicists Determine the Optimal Conditions for Holding High-Energy Plasma Clouds in Pyrotron

RUDN University physicists have described the conditions for the most efficient operation of long mirror-based variant of cyclotron in the autoresonance mode. These data will bring better understanding of plasma processes in magnetic traps.

16 Jun
The new protocol will allow to obtain bioactive compounds bypassing by-products

Chemists from Russia and the USA have improved the method of creating bioactive indole-acetonitrile compounds. Previously, in the course of their synthesis, by-products were obtained — a new method allows them to be avoided. This increases the yield of the final product to 81%.