The goal of this assignment is to develop your hands-on skills in performing learning from data, as well as further understanding of the practical technical details of the learning procedure. You will...

{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "EE4305_Mini_Project.ipynb",
"provenance": [],
"collapsed_sections": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.6"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "_0SzgnFlkU0L"
},
"source": [
"### Introduction\n",
"\n",
"In this project we implemented a neural network (NN) architecture from scratch. Building the neural network will give a hands-on experience converting mathematical foundations of NN such as feed-forward and backpropagation algorithms into Python code. \n",
"\n",
"The model implemented will be tested on following examples:\n",
"\n",
"#### Classification:\n",
"\n",
"1. AND/OR Logic\n",
"2. XOR Logic\n",
"\n",
"#### Regression:\n",
"\n",
"1. Sinusoidal curve\n",
"2. Gaussian curve\n",
"\n",
"The model will be evaluated on real world data using the Wisconsin Breast Cancer Dataset. \n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TJMvqyqgplyu"
},
"source": [
"### Problem Description\n",
"\n",
"Below is a list of tasks in the Project:\n",
"\n",
"1. Complete the implementation of Feedforward and Backpropagation\n",
"2. Implement functions for train, predict, and evaluate\n",
"3. Generate datasets for AND, XOR logics and Sinusoidal, Gaussian functions\n",
"4. Test your model using the above datasets.\n",
"5. Implement functions to improve model performance. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XV9SOKLRuQcp"
},
"source": [
"### Import Libraries\n",
"\n",
"* numpy, pandas - Data handling and processing\n",
"* matplotlib, seaborn - Visualization\n",
"* tqdm - For implementing progressbars\n",
"* sklearn - datasets\n",
"* TSNE: for visualization\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "bBkrzmUuvM5m"
},
"source": [
"# Imports\n",
"import numpy as np\n",
"from tqdm.notebook import tqdm\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "IM04rippCHBP",
"outputId": "e69cf92f-65b6-41e1-adfa-729b8d82a352",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 92
}
},
"source": [
"print(np.random.random(10))\n",
"print(np.random.random(10))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"[0.24790877 0.80360919 0.40613672 0.65877726 0.31690227 0.7235624\n",
" 0.46984446 0.94761042 0.2068924 0.29758977]\n",
"[0.7340046 0.91642861 0.00531714 0.8775393 0.7058875 0.41874007\n",
" 0.23504671 0.28281132 0.99023306 0.40585885]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "qcSmNrIsCARD",
"outputId": "aa46e7b6-9848-485f-8058-48b0c16063e1",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 92
}
},
"source": [
"print(np.random.RandomState(seed=30).random(10))\n",
"print(np.random.RandomState(seed=20).random(10))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"[0.64414354 0.38074849 0.66304791 0.16365073 0.96260781 0.34666184\n",
" 0.99175099 0.2350579 0.58569427 0.4066901 ]\n",
"[0.5881308 0.89771373 0.89153073 0.81583748 0.03588959 0.69175758\n",
" 0.37868094 0.51851095 0.65795147 0.19385022]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "qj68l3hzCEGk",
"outputId": "10a846e7-bc2f-422d-eda1-d0f4fbf10afe"
},
"source": [
"print(np.random.RandomState(20).random(10))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"[0.5881308 0.89771373 0.89153073 0.81583748 0.03588959 0.69175758\n",
" 0.37868094 0.51851095 0.65795147 0.19385022]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6mJwuofS3ski"
},
"source": [
"### Implement and test a baseline Neural Network"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hdSftjSfKpTM"
},
"source": [
"#### Activation functions and derivatives\n",
"\n",
"In the below cell we define sigmoid activation function and its derivative"
]
},
{
"cell_type": "code",
"metadata": {
"id": "OB_Z7ydoKo7W"
},
"source": [
"# Activation function\n",
"def sigmoid(t):\n",
" return 1/(1+np.exp(-t))\n",
"\n",
"# Derivative of sigmoid\n",
"def sigmoid_derivative(p):\n",
" return p * (1 - p)\n",
"\n",
"# Activation function\n",
"def relu(t):\n",
" return np.maximum(0,t)\n",
"\n",
"# Derivative of relu\n",
"def relu_derivative(p):\n",
" return np.greater(p, 0).astype(int)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ybW08fHHDfaj"
},
"source": [
"#### Loss functions and derivatives\n",
"\n",
"In the below cell we define rmse loss function and its derivative"
]
},
{
"cell_type": "code",
"metadata": {
"id": "a47PIQ8yDfJm"
},
"source": [
"# Loss function - RMSE\n",
"def rmse(y, ypred):\n",
" return np.sqrt(np.mean((y - ypred)**2))\n",
"\n",
"# Loss function derivative - RMSE\n",
"def rmse_derivative(y, ypred):\n",
" return 2*(y - ypred)\n",
"\n",
"# Loss function - Cross Entropy\n",
"# def cross_entropy(X,y):\n",
" # X = X.clip(min=1e-8,max=None)\n",
" # return \n",
"\n",
"# Loss function derivative - Cross Entropy\n",
"# def cross_entropy_derivative(X,y):\n",
" # return "
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "LcF4O0iwCICj"
},
"source": [
"#### Base Class\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "ORNkfSoNvbWy"
},
"source": [
"# Class definition\n",
"class NeuralNetwork:\n",
" def __init__(self, x, y, hidden_nodes, learning_rate):\n",
" \"\"\"\n",
" Class initializer. Loads the value of input data and initializes weights\n",
"\n",
" Parameters:\n",
" x: input data\n",
" y: ground truth label\n",
"\n",
" Sets class parameters:\n",
" self.x = input data\n",
" self.y = ground truth label\n",
" ...