earticle

논문검색

Convergence of Internet, Broadcasting and Communication

Beta and Alpha Regularizers of Mish Activation Functions for Machine Learning Applications in Deep Neural Networks

초록

영어

A very complex task in deep learning such as image classification must be solved with the help of neural networks and activation functions. The backpropagation algorithm advances backward from the output layer towards the input layer, the gradients often get smaller and smaller and approach zero which eventually leaves the weights of the initial or lower layers nearly unchanged, as a result, the gradient descent never converges to the optimum. We propose a two-factor non-saturating activation functions known as Bea-Mish for machine learning applications in deep neural networks. Our method uses two factors, beta (β) and alpha (α), to normalize the area below the boundary in the Mish activation function and we regard these elements as Bea. Bea-Mish provide a clear understanding of the behaviors and conditions governing this regularization term can lead to a more principled approach for constructing better performing activation functions. We evaluate Bea-Mish results against Mish and Swish activation functions in various models and data sets. Empirical results show that our approach (Bea-Mish) outperforms native Mish using SqueezeNet backbone with an average precision (AP50val) of 2.51% in CIFAR-10 and top-1accuracy in ResNet-50 on ImageNet-1k. shows an improvement of 1.20%.

목차

Abstract
1. Introduction
2. Theory
3. Experiment Settings
4. Results and Discussion
4.1 CIFAR-10 Dataset on Various Baseline Activation Functions
4.2 CIFAR-10 Dataset on Various Standard Neural Network Architectures
4.3 ImageNet-1k Dataset on Various Standard Neural Network Architectures
4.4 Ablation Study of α and β on CIFAR-10
5. Conclusion
Acknowledgement
References

저자정보

  • Peter Beatus Mathayo Master Student, Department of Computer Engineering, Graduate School, Dongseo University, Korea
  • Dae-Ki Kang Professor, Department of Computer Engineering, Dongseo University, Korea

참고문헌

자료제공 : 네이버학술정보

    함께 이용한 논문

      ※ 기관로그인 시 무료 이용이 가능합니다.

      • 4,000원

      0개의 논문이 장바구니에 담겼습니다.