ReLU-based activations: analysis and experimental study for deep learning

Áreas de investigación:

• Sin categoría

Año:

2021

Tipo de publicación:

Artículo en conferencia

Palabras clave:

analysis activations, RELU, RELU activations, deep learning

Autores:

• Vargas, Víctor Manuel
• Guijo-Rubio, David
• Gutiérrez, Pedro Antonio
• Hervás-Martínez, César

Volumen:

12882

Título del libro:

Proceedings of the XIX Conference of the Spanish Association for Artificial Intelligence (CAEPIA)

Serie:

Lecture Notes in Artificial Intelligence (LNAI)

Páginas:

33-43

Organización:

Malaga, Spain

Mes:

22nd-24th September

ISBN:

978-3-030-85712-7

ISSN:

0302-9743

BibTex:

@conference{ReluVictorCAEPIA2021, author = "V{\'i}ctor Manuel Vargas and David Guijo-Rubio and Pedro Antonio Guti{\'e}rrez and C{\'e}sar Herv{\'a}s-Mart{\'i}nez", abstract = "Activation functions are used in neural networks as a tool to introduce non-linear transformations into the model and, thus, enhance its representation capabilities. They also determine the output range of the hidden layers and the final output. Traditionally, artificial neural networks mainly used the sigmoid activation function as the depth of the network was limited. Nevertheless, this function tends to saturate the gradients when the number of hidden layers increases. For that reason, in the last years, most of the works published related to deep learning and convolutional networks use the Rectified Linear Unit (ReLU), given that it provides good convergence properties and speeds up the training process thanks to the simplicity of its derivative. However, this function has some known drawbacks that gave rise to new proposals of alternatives activation functions based on ReLU. In this work, we describe, analyse and compare different recently proposed alternatives to test whether these functions improve the performance of deep learning models regarding the standard ReLU.", booktitle = "Proceedings of the XIX Conference of the Spanish Association for Artificial Intelligence (CAEPIA)", doi = "10.1007/978-3-030-85713-4_4", isbn = "978-3-030-85712-7", issn = "0302-9743", keywords = "analysis activations, RELU, RELU activations, deep learning", month = "22nd-24th September", organization = "Malaga, Spain", pages = "33-43", publisher = "Springer", series = " Lecture Notes in Artificial Intelligence (LNAI)", title = "{R}e{LU}-based activations: analysis and experimental study for deep learning", url = "doi.org/10.1007/978-3-030-85713-4_4", volume = "12882", year = "2021", }

Abstract:

Activation functions are used in neural networks as a tool to introduce non-linear transformations into the model and, thus, enhance its representation capabilities. They also determine the output range of the hidden layers and the final output. Traditionally, artificial neural networks mainly used the sigmoid activation function as the depth of the network was limited. Nevertheless, this function tends to saturate the gradients when the number of hidden layers increases. For that reason, in the last years, most of the works published related to deep learning and convolutional networks use the Rectified Linear Unit (ReLU), given that it provides good convergence properties and speeds up the training process thanks to the simplicity of its derivative. However, this function has some known drawbacks that gave rise to new proposals of alternatives activation functions based on ReLU. In this work, we describe, analyse and compare different recently proposed alternatives to test whether these functions improve the performance of deep learning models regarding the standard ReLU.