https://ieeexplore.ieee.org/abstract/document/9019697
Final Project Amir Sadovnik COSC 424/525: Deep Learning (Spring 2023) 1 Overview The goal of this project is for you to further explore a certain subject that we covered in class or a new subject related to it (based on some previous work). This is an open project in a sense that you will propose the problem you would like to work on and the solution you are planning use. Your main goal should be to explore the characteristics of your problem/solution under the deep learning framework 2 A Few Guidelines 1. Most projects fall into one of these three categories: (taken from http://cs229.stanford.edu/projects.html) Application Project: This is by far the most common: Pick an application that interests you, and explore how best to apply learning algorithms to solve it. Algorithmic project: Pick a problem or family of problems, and develop a new learning algorithm, or a novel variant of an existing algorithm, to solve it. Theoretical project: Prove some interesting/non-trivial properties of a new or an existing learning algorithm. (This is often quite difficult, and so very few, if any, projects will be purely theoretical.) 2. Your work should be novel in some respect. That is, it should either address a new domain or further examine certain solutions in a specific domain. 3. Your project will not be judged by how well it succeeded, but rather by how well it was motivated/thought of/tested. Therefore, feel free to try some far fetched ideas. 4. You should not propose a completely new idea. Rather, you should further develop an issue explored in the course or from a paper, or the combination of two issues. 1 5. Try to find a recent paper which addresses the topic of your choice, and relate your work to it. 6. Try to not introduce multiple novelties - try to focus on one aspect so you can explore it sufficiently. 7. Make sure that the question you bring up in your paper, is answered by your paper. 8. Feel free to integrate things you are already working on into your project. 9. Good previous work can be found at the ICML and NeurIPS conferences. 10. You will be using Google Cloud to run your code. Details will follow. 11. Teams can be 2-3 students. 3 Sample Projects Algorithms: – Exploring Active Learning in Medical NLP Exploring ways to select data for training in a more efficient way. – Merging Lottery Tickets Took a method which prunes neural networks and tried to merge them. Using GANs to supplement data: – Traffic Sign Recognition: A Test of Small Neural Networks and Expanded Datasets Tried to use GANs to generate traffic signs to improve classification rate – Improving Coronavirus Disease Detection fromLung X-Ray Images Using Gener- ative Adversarial Networks Tried to use GANs to generate x ray images to improve classification rate – Mammography Data Augmentation using conditional DCGAN and Classification of different Image Resolutions Tried to use conditional GANs to generate mammography data to improve clas- sification rate Looking at changing input (pre-processing): – Manufacturing Feature Recognition with a 2D Convolutional Neural Network Use a novel 2d feature representation and a use a CNN on that (vs usual 3d). – Classification of COVID-19-Related Pneumonia Mostly tried to add preprocessing + auto-encoder methods before classification. – Measuring the Impact of Unsharp Masking Applied to X-ray Bone Images in Con- volutional Neural Networks Mostly tried to add preprocessing methods before classification 2 https://icml.cc/Conferences/2019/Schedule https://neurips.cc/Conferences/2019/Schedule Applying network on novel application (or a different type of network): – Day-ahead Precipitation Class Prediction via Stacked Denoising LSTM Autoen- coders Used a stacked LSTM auto-encoders on a problem they have not been used before. – Extracting model Hamiltonian for a complex quantum material Used autoencoders for physics application. – Deep Learning Based Forecast for Solar Radiation Storms Use a combination of convolutions and LSTM to forecast radiation storms. – Deep Learning in Classification of Power System Outages Introduce new features to previous work and change the design of the CNN/RNN. – Applying Convolutional Neural Networks to Time-Frequency Responses of WiFi Signals Applied CNN’s on novel problem. – Application of Convolutional Neural Networks to EGG Signals to Classify Mental State in n-back Task Applied CNN’s on novel problem. – Drone-assisted Parcel Delivery Problem: A Deep Reinforcement Learning Ap- proach Use of deep learning with reinforcement learning on novel problem. Generating Images: – Scene Manipulation of Landmark Images Using Deep Neural Style Transfer Net- works Devise a simplified style transfer network based on VAE and experimented with it. – Single Image Super Resolution: Exploring Tried to create a very small super resolution network. Check out this link for more examples of projects: http://cs229.stanford.edu/projects.html 4 Coding(30%) You are required to write your code in python3. You are allowed to use certain external libraries (i.e. Tensorflow, numpy, matplotlib). However, if you are unsure make sure to check with me in order to get my approval. In addition, you may use other publicly available code - given that you cite it, and describe the exact changes you made to it. 5 Report (70%) I will upload a Latex template to Canvas. Your report will need to include the following sections: 3 http://cs229.stanford.edu/projects.html 1. Abstract(5%) Make sure to clearly state the novelty of your work in the abstract. 2. Introduction(5%) In this section you should discuss the question you are planning to investigate. Make sure to describe the problem clearly, and how you are planning to solve it. Also, clarify what your new contribution is (as opposed to previous works). 3. Previous Work(10%) Discuss the previous work you are using as a starting point/reference. If you are working on a different domain include work that works on the same or similar domain. If you are using a new method, discuss other works which use similar methods. Note that you can use both academic papers in addition to online articles. Make sure to cite all sources. 4. Technical Approach (10%) Describe the problem you are working on including the rules, goals, etc. Assume that the reader has not been introduced to this domain problem previously. Describe your network architecture/pre-processing/learning method. If you are examining different options describe them all. In addition, add diagrams to clarify. 5. Dataset and Implementation (10%) Describe the details of your dataset and implementation. This should include things like the size of your dataset, training/testing split, network hyper-parameters value, training time, hardware used, etc. 6. Experiments and Results Analysis(20%) Present the results which answer your hypothesis questions. These should include dif- ferent graphs which clarify the answer and examine different aspects of your solution. If not described in the implementation section, make sure to carefully describe how your experiments were conducted (how many runs, how it was initialized, etc. This section is the most important, and will be evaluated both on presentation and correctness. 7. Conclusion(5%) What have you achieved in this project? What have you learned? 8. References 9. Appendices: Code Design(5%) Describe the general structure of your code including functions, classes, and data structures used. If you are using code written by someone else make sure to cite it and emphasize the parts that you have written yourself. Workload Distribution Describe how work was divided between the team members 4 6 Submission You are required to submit one zip file with both he report and your code. 5 Overview A Few Guidelines Sample Projects Coding(30%) Report (70%) Submission I. Title of the project: II. Team: III. Problem definition: IV. Motivation: I. Why is it interesting? · II. Where do you think it's going to be used, i.e., application area? · V. Literature review: What reading will you examine to provide context and background? Please put citations of the article/blog posts with full citations VI. Dataset: What data will you use? If you are collecting new data, how will you do it? VII. Proposed method: What method or algorithm are you proposing? If there are existing implementations, will you use them and how? How do you plan to improve or modify such implementations? You don't have to have an exact answer at this point, but you should have a general sense of how you will approach the problem you are working on. VIII. Evaluation: How will you evaluate your results? Qualitatively, what kind of results do you expect (e.g. plots or figures)? Quantitatively, what kind of analysis will you use to evaluate and/or compare your results (e.g. what performance metrics or statistical tests)? ByeGlassesGAN: Identity Preserving Eyeglasses Removal for Face Images Yu-Hui Lee1[0000−0002−4162−1597] and Shang-Hong Lai1,2 1 Department of Computer Science, National Tsing Hua University, Taiwan 2 Microsoft AI R&D Center, Taiwan
[email protected],
[email protected] Abstract. In this paper, we propose a novel image-to-image GAN frame- work for eyeglasses removal, called ByeGlassesGAN, which is used to au- tomatically detect the position of eyeglasses and then remove them from face images. Our ByeGlassesGAN consists of an encoder, a face decoder, and a segmentation decoder. The encoder is responsible for extracting information from the source face image, and the face decoder utilizes this information to generate glasses-removed images. The segmentation decoder is included to predict the segmentation mask of eyeglasses and completed face region. The feature vectors generated by the segmentation decoder are shared with the face decoder, which facilitates better recon- struction results. Our experiments show that ByeGlassesGAN can pro- vide visually appealing results in the eyeglasses-removed face images even for semi-transparent color eyeglasses or glasses with glare. Furthermore, we demonstrate significant improvement in face recognition accuracy for face images with glasses by applying our method as a pre-processing step in our face recognition experiment. Keywords: Generative Adversarial Networks, Face Attributes Manip- ulation, Face Recognition 1 Introduction Face recognition has been researched extensively and widely used in our daily lives. Although state-of-the-art face recognition systems are capable of recogniz- ing faces for practical applications, their accuracies are degraded when the face images are partially occluded, such as wearing eyeglasses. An obvious reason causes this problem is that the eyeglasses may occlude some important infor- mation on faces, leading to discrepancies in facial feature values. For example, the thick frame of glasses may block the eyes. Hence, in the past, researchers proposed to apply the PCA-based methods [24,20] to remove eyeglasses from face images. However, the PCA-based method can only provide approximate glasses removal image via face subspace projection. In addition, they did not really evaluate their methods on diverse face recognition tasks. Another reason for the degradation of face recognition accuracy with eye- glasses is that face images with eyeglasses are considerably fewer than glasses- free images. It is hard to make the recognition model learn the feature of various ar X iv :2 00 8. 11 04 2v 1 [ cs .M M ] 2 5 A ug 2 02 0 2 Y. Lee and S. Lai Fig. 1. Examples of glasses removal by ByeGlassesGAN. kinds of eyeglasses. Recently, alongside with the popularity of face attributes manipulation, some GAN based methods, such as [6] and [30], improved the capability of recognizing faces with eyeglasses by synthesizing a large amount of images of faces with eyeglasses for training a face recognition model. Different from the previous works, we aim at improving face recognition accuracy by removing eyeglasses with the proposed GAN model before face recognition. With the proposed GAN-based method, we can not only improve face recognition accuracy, the visually appealing glasses-removed images can also be used for some interesting applications, like applying virtual makeup. The main contributions of this work are listed as follows: 1. We propose a novel glasses removal framework, which can automatically detect and remove eyeglasses from a face image. 2. Our proposed framework combines the mechanisms of the feature sharing between 2 decoders to acquire better visual results, and an identity classifier to make sure the identity in the glasses-removed face image is well preserved. 3. We come up with a new data synthesis method to train a glasses removal network, which effectively simulates color lens, glare of reflection as well as the refraction on eyeglasses. 4. In the experiment, we demonstrate that the face recognition accuracy is significantly improved for faces with eyeglasses after applying the proposed eyeglasses removal method as a pre-processing step. 2 Related Works 2.1 Face Attributes Manipulation Face attributes manipulation is a research topic that attracts a lot of attention. Along with the popularity of GAN, there are many impressive GAN-based meth- ods proposed for editing face attributes. [16] and [30] edit face attributes through an attribute transformation network and a mask network. Both of them preserve the identity of the source images by using the predicted mask to constrain the editing area. AttGAN [7] edits face images through the attribute classification constraint and reconstruction learning. ELEGANT [25] can not only manipulate face images but also manipulate images according to the attributes of reference images. ERGAN [9] removes eyeglasses by switching features extracted from a face appearance encoder and an eye region encoder. Besides, there are sev- eral face attributes editing methods which are not GAN-based. For example, ByeGlassesGAN 3 DFI [23] manipulated face images through linear interpolation of the feature vectors of different attributes. [2] achieved identity preserving face attributes editing by disentangling the identity and attributes vectors of face images with the mechanisms of Variational Autoencoder and GAN. However, these face at- tributes manipulation methods suffer from