Abstract

Generative adversarial networks (GANs) have turned up as the most widely used approaches for creating realistic samples. They're the effective latent variable models for learning complex real distributions. However, despite their enormous success and popularity, the process of training GANs remains challenging and suffers from a number of failures. These failures include mode collapse where the generator generates the same set of output for different inputs which finally leads to loss of diversity; non-convergence because of the diverging and oscillatory behaviors of both generator and discriminator while being trained; and vanishing or exploding gradients due to which either no learning or extremely slow learning takes place. In the past years, a variety of strategies for stabilizing GAN training have been explored which includes modified architectures, loss functions, and other methods. The choice of loss function has been found to be the most crucial part of the GAN model because it influences the vanishing gradient and model collapse directly. Viewing these loss functions as divergence minimization has provided a rich avenue of development. All of these factors make GAN training inherently unstable, and this instability is difficult to comprehend mathematically. This paper intends to provide a thorough mathematical explanation of these divergence minimization functions. It illustrates in great detail the two variants of the loss functions of the original GAN, their optimization to Kullback-Leibler (KL) divergence and Jensen-Shannon (JS) divergence along with their shortcomings. It also describes the loss functions of the different loss function GAN variants that have been proposed to mitigate these shortcomings as well as their minimization. The original GAN and its loss function variants have also been implemented using the standard MNIST, Fashion-MNIST, and CIFAR-10 datasets.

Keyword

Generative adversarial networks, Divergence minimization, Loss functions, Stable training, Mode collapse, Non-convergence.

Cite this article

Mehmood R, Bashir R, Giri KJ

Refference

[1][1]Simon A, Singh M, S. Venkatesan S, Babu DRR. An overview of M learning and its application. International Journal of Electrical Sciences Electrical Sciences & Engineering. 2015; 1(1): 22-4.

[2][2]Jebara T. Machine learning: discriminative and generative. Springer Science & Business Media; 2012.

[3][3]Harshvardhan GM, Gourisaria MK, Pandey M, Rautaray SS. A comprehensive survey and analysis of generative models in machine learning. Computer Science Review. 2020.

[4][4]Jebara T. Discriminative, generative and imitative learning (Doctoral dissertation, PhD thesis, Media laboratory, MIT).

[5][5]https://iq.opengenus.org/discriminative-model/. Accessed 4 March 2022.

[6][6]https://towardsdatascience.com/what-is-deep-learning-and-how-does-it-work-2ce44bb692ac. Accessed 4 March 2022.

[7][7]Bishop CM, Nasrabadi NM. Pattern recognition and machine learning. New York: Springer; 2006.

[8][8]Salakhutdinov R. Learning deep generative models. Annual Review of Statistics and Its Application. 2015; 2:361-85.

[9][9]Ruthotto L, Haber E. An introduction to deep generative modeling. GAMM‐Mitteilungen. 2021; 44(2).

[10][10]Goodfellow I. Nips 2016 tutorial: generative adversarial networks. arXiv preprint arXiv:1701.00160. 2016.

[11][11]Hong Y, Hwang U, Yoo J, Yoon S. How generative adversarial networks and their variants work: an overview. ACM Computing Surveys. 2019; 52(1):1-43.

[12][12]Singh U. Generative adversarial networks: a survey. 2021:1-28.

[13][13]Mescheder L, Geiger A, Nowozin S. Which training methods for GANs do actually converge? In international conference on machine learning 2018 (pp. 3481-90). PMLR.

[14][14]Ratliff LJ, Burden SA, Sastry SS. Characterization and computation of local Nash equilibria in continuous games. In 2013 51st annual Allerton conference on communication, control, and computing 2013 (pp. 917-24). IEEE.

[15][15]Barnett SA. Convergence problems with generative adversarial networks (GANS). arXiv preprint arXiv:1806.11382. 2018.

[16][16]Dutt RK, Premchand P. Generative adversarial networks (GAN) review. CVR Journal of Science and Technology. 2017; 13:1-5.

[17][17]Dong HW, Yang YH. Towards a deeper understanding of adversarial losses under a discriminative adversarial network setting. arXiv preprint arXiv:1901.08753. 2019.

[18][18]Chu C, Minami K, Fukumizu K. Smoothness and stability in GANS. arXiv preprint arXiv:2002.04185. 2020.

[19][19]Park SW, Ko JS, Huh JH, Kim JC. Review on generative adversarial networks: focusing on computer vision and its applications. Electronics. 2021; 10(10):1-40.

[20][20]De RGH, Papa JP. A survey on text generation using generative adversarial networks. Pattern Recognition. 2021.

[21][21]Park J, Kim H, Kim J, Cheon M. A practical application of generative adversarial networks for RNA-seq analysis to predict the molecular progress of Alzheimers disease. PLoS Computational Biology. 2020; 16(7).

[22][22]Dia M, Savary E, Melchior M, Courbin F. Galaxy image simulation using progressive GANs. arXiv preprint arXiv:1909.12160. 2019.

[23][23]Navidan H, Moshiri PF, Nabati M, Shahbazian R, Ghorashi SA, Shah-mansouri V, et al. Generative adversarial networks (GANs) in networking: a comprehensive survey & evaluation. Computer Networks. 2021.

[24][24]Karras T, Aila T, Laine S, Lehtinen J. Progressive growing of GANS for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196. 2017.

[25][25]Liao W, Hu K, Yang MY, Rosenhahn B. Text to image generation with semantic-spatial aware GAN. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition 2022 (pp. 18187-96).

[26][26]Zhu M, Pan P, Chen W, Yang Y. DM-GAN: dynamic memory generative adversarial networks for text-to-image synthesis. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition 2019 (pp. 5802-10).

[27][27]Zhu JY, Park T, Isola P, Efros AA. Unpaired image-to-image translation using cycle-consistent adversarial networks. In proceedings of the IEEE international conference on computer vision 2017 (pp. 2223-32).

[28][28]Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, et al. Photo-realistic single image super-resolution using a generative adversarial network. In proceedings of the IEEE conference on computer vision and pattern recognition 2017 (pp. 4681-90). IEEE.

[29][29]Li W, Zhou K, Qi L, Lu L, Lu J. Best-buddy GANS for highly detailed image super-resolution. In proceedings of the AAAI conference on artificial intelligence 2022 (pp. 1412-20).

[30][30]Quan F, Lang B, Liu Y. ARRPNGAN: text-to-image GAN with attention regularization and region proposal networks. Signal Processing: Image Communication. 2022.

[31][31]Isola P, Zhu JY, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. In proceedings of the IEEE conference on computer vision and pattern recognition 2017 (pp. 1125-34).

[32][32]Zhao J, Lee F, Hu C, Yu H, Chen Q. LDA-GAN: lightweight domain-attention GAN for unpaired image-to-image translation. Neurocomputing. 2022; 506:355-68.

[33][33]Jeong JJ, Tariq A, Adejumo T, Trivedi H, Gichoya JW, Banerjee I. Systematic review of generative adversarial networks (GANS) for medical image classification and segmentation. Journal of Digital Imaging. 2022; 35:137-52.

[34][34]Arora A, Shantanu. A review on application of GANs in cybersecurity domain. IETE Technical Review. 2022; 39(2):433-41.

[35][35]Brophy E, Wang Z, She Q, Ward T. Generative adversarial networks in time series: a survey and taxonomy. arXiv preprint arXiv:2107.11098. 2021.

[36][36]Kong J, Kim J, Bae J. Hifi-gan: generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems. 2020; 33:17022-33.

[37][37]Jin CB, Kim H, Liu M, Jung W, Joo S, Park E, et al. Deep CT to MR synthesis using paired and unpaired data. Sensors. 2019; 19(10):1-19.

[38][38]Repecka D, Jauniskis V, Karpus L, Rembeza E, Rokaitis I, Zrimec J, et al. Expanding functional protein sequence spaces using generative adversarial networks. Nature Machine Intelligence. 2021; 3(4):324-33.

[39][39]Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434. 2015.

[40][40]Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks. In international conference on machine learning 2017 (pp. 214-23). PMLR.

[41][41]Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V, Courville AC. Improved training of Wasserstein GANs. Advances in Neural Information Processing Systems. 2017.

[42][42]Fedus W, Rosca M, Lakshminarayanan B, Dai AM, Mohamed S, Goodfellow I. Many paths to equilibrium: GANs do not need to decrease a divergence at every step. arXiv preprint arXiv:1710.08446. 2017.

[43][43]Kodali N, Abernethy J, Hays J, Kira Z. On convergence and stability of GANs. arXiv preprint arXiv:1705.07215. 2017.

[44][44]Sharma A, Jindal N, Rana PS. Potential of generative adversarial net algorithms in image and video processing applications–a survey. Multimedia Tools and Applications. 2020; 79(37):27407-37.

[45][45]Jin L, Tan F, Jiang S. Generative adversarial network technologies and applications in computer vision. Computational Intelligence and Neuroscience. 2020.

[46][46]Aggarwal A, Mittal M, Battineni G. Generative adversarial network: an overview of theory and applications. International Journal of Information Management Data Insights. 2021; 1(1):1-9.

[47][47]Huang H, Yu PS, Wang C. An introduction to image synthesis with generative adversarial nets. arXiv preprint arXiv:1803.04469. 2018.

[48][48]Hitawala S. Comparative study on generative adversarial networks. arXiv preprint arXiv:1801.04271. 2018.

[49][49]Jabbar A, Li X, Omar B. A survey on generative adversarial networks: variants, applications, and training. ACM Computing Surveys. 2021; 54(8):1-49.

[50][50]Wali A, Alamgir Z, Karim S, Fawaz A, Ali MB, Adan M, et al. Generative adversarial networks for speech processing: a review. Computer Speech & Language. 2022.

[51][51]Jozdani S, Chen D, Pouliot D, Johnson BA. A review and meta-analysis of generative adversarial networks and their applications in remote sensing. International Journal of Applied Earth Observation and Geoinformation. 2022.

[52][52]Shahriar S. GAN computers generate arts? a survey on visual arts, music, and literary text generation using generative adversarial network. Displays. 2022.

[53][53]Saxena D, Cao J. Generative adversarial networks (GANs) challenges, solutions, and future directions. ACM Computing Surveys. 2021; 54(3):1-42.

[54][54]Kurach K, Lucic M, Zhai X, Michalski M, Gelly S. The GAN landscape: losses, architectures, regularization, and normalization. 2018.

[55][55]Pan Z, Yu W, Wang B, Xie H, Sheng VS, Lei J, et al. Loss functions of generative adversarial networks (GANs): opportunities and challenges. IEEE Transactions on Emerging Topics in Computational Intelligence. 2020; 4(4):500-22.

[56][56]Wiatrak M, Albrecht SV, Nystrom A. Stabilizing generative adversarial networks: a survey. arXiv preprint arXiv:1910.00927. 2019.

[57][57]https://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/readings/L19%20GANs.pdf. Accessed 4 March 2022.

[58][58]Berard H, Gidel G, Almahairi A, Vincent P, Lacoste-julien S. A closer look at the optimization landscapes of generative adversarial networks. arXiv preprint arXiv:1906.04848. 2019.

[59][59]Wang Z, She Q, Ward TE. Generative adversarial networks: a survey and taxonomy. arXiv preprint arXiv:1906.01529. 2019.

[60][60]http://www.moreisdifferent.com/assets/science_notes/notes_on_GAN_objective_functions.pdf. Accessed 4 March 2022.

[61][61]Uddin SM. Intuitive approach to understand the mathematics behind GAN. Intuitive Approach Math. 2019.

[62][62]Huszár F. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? arXiv preprint arXiv:1511.05101. 2015.

[63][63]Theis L, Oord AV, Bethge M. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844. 2015.

[64][64]Manisha P, Gujar S. Generative adversarial networks (GANs): the progress so far in image generation. arXiv. 2019.

[65][65]Arjovsky M, Bottou L. Towards principled methods for training generative adversarial networks. arXiv preprint arXiv:1701.04862. 2017.

[66][66]Gui J, Sun Z, Wen Y, Tao D, Ye J. A review on generative adversarial networks: algorithms, theory, and applications. IEEE Transactions on Knowledge and Data Engineering. 2021.

[67][67]Shannon M, Poole B, Mariooryad S, Bagby T, Battenberg E, Kao D, et al. Non-saturating GAN training as divergence minimization. arXiv preprint arXiv:2010.08029. 2020.

[68][68]Brock A, Donahue J, Simonyan K. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096. 2018.

[69][69]Mallick PK, Meher P, Majumder A, Das SK. Electronic systems and intelligent computing: proceedings of ESIC 2020. Springer; 2020.

[70][70]Carneiro G. Why are generative adversarial networks so fascinating and annoying? In 33rd SIBGRAPI conference on graphics, patterns and images 2020 (pp. 1-8). IEEE.

[71][71]Wang Y. A mathematical introduction to generative adversarial nets (GAN). arXiv preprint arXiv:2009.00169. 2020.

[72][72]Weng L. From GAN to WGAN. arXiv preprint arXiv:1904.08994. 2019.

[73][73]Qin Y, Mitra N, Wonka P. How does Lipschitz regularization influence GAN training? In European conference on computer vision 2020 (pp. 310-26). Springer, Cham.

[74][74]Nakamura K, Korman S, Hong BW. Stabilization of generative adversarial networks via noisy scale-space. arXiv preprint arXiv:2105.00220. 2021.

[75][75]Pinetz T, Soukup D, Pock T. On the estimation of the Wasserstein distance in generative models. In German conference on pattern recognition 2019 (pp. 156-70). Springer, Cham.

[76][76]Sampath V, Maurtua I, Aguilar Martín JJ, Gutierrez A. A survey on generative adversarial networks for imbalance problems in computer vision tasks. Journal of Big Data. 2021; 8(1):1-59.

[77][77]Mao X, Li Q, Xie H, Lau RY, Wang Z, Paul SS. Least squares generative adversarial networks. In proceedings of the IEEE international conference on computer vision 2017 (pp. 2794-802).

[78][78]Bhatia H. Generalized loss functions for generative adversarial networks (Doctoral Dissertation, Queens University (Canada)).