Hi, I have uploaded both the assignment skeleton and the PDF file for scoring and requirement. Are you able to attain at least 90% of the scoring system on the last page of PDF file? 27/30 marks at...

{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "_0SzgnFlkU0L"
},
"source": [
"### Introduction\n",
"\n",
"In this project we will implement a neural network (NN) architecture from scratch. Building the neural network will give a hands-on experience converting mathematical foundations of NN such as feed-forward and backpropagation algorithms into Python code. The project will also include implementation of sub-functions in NN such as loss functions, activation functions, and derivatives.\n",
"\n",
"The model implemented will be tested on following examples:\n",
"\n",
"#### Classification:\n",
"\n",
"1. AND/OR Logic\n",
"2. XOR Logic\n",
"\n",
"#### Regression:\n",
"\n",
"1. Sinusoidal curve\n",
"2. Gaussian curve\n",
"\n",
"The model will be evaluated on real world data using the Wisconsin Breast Cancer Dataset. \n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "TJMvqyqgplyu"
},
"source": [
"### Problem Description\n",
"\n",
"You are provided with a base Class for implementation of a simple Neural Network. You will also be provided with templates for some of the helper functions. You are required to implement functions in the Neural Network Class. You are also required to test your neural network with various datasets and show the effect of modifying parameters. Below is a list of problems that you are expected to solve in this Project:\n",
"\n",
"1. Complete the implementation of Feedforward and Backpropagation\n",
"2. Implement functions for train, predict, and evaluate\n",
"3. Generate datasets for AND, XOR logics and Sinusoidal, Gaussian functions\n",
"4. Test your model using the above datasets. What are you choice of model parameters and how does it affect the performance?\n",
"5. Implement functions to improve model performance. What are your observations\n",
"6. Evaluate model performance on Wisconsin Breast Cancer dataset. What is the highest accuracy score obtained?"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "HBZe_HR5tvba"
},
"source": [
"### Notes\n",
"\n",
"1. Only following Python libraries (and their functions) are allowed: **numpy, pandas, matplotlib, seaborn.** (The code provided include the use of sklearn library for loading data, evaluation, and TSNE encoding. Do not load any other functions from sklearn.)\n",
"2. Following libraries from Python may not be used: **Keras, TensorFlow, pytorch, thean, scikit-learn (except functions that are already included)**\n",
"3. You may refer to online resources but do not copy+paste. All submitted items will be checked for Plagiarism.\n",
"4. The base class provided is for your reference. You are welcome to implement the NN algorithm differently. In that case, ensure that your implementation covers all the sub-problems. Also explain your methodology."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "XV9SOKLRuQcp"
},
"source": [
"### Import Libraries\n",
"\n",
"* numpy, pandas - Data handling and processing\n",
"* matplotlib, seaborn - Visualization\n",
"* tqdm - For implementing progressbars\n",
"* sklearn - datasets: to load breast cancer dataset, TSNE: for visualization\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 75
},
"colab_type": "code",
"id": "bBkrzmUuvM5m",
"outputId": "79faa168-3224-403e-932d-93ea848d76a8"
},
"outputs": [],
"source": [
"# Imports\n",
"import numpy as np\n",
"from tqdm.notebook import tqdm\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 92
},
"colab_type": "code",
"id": "IM04rippCHBP",
"outputId": "e69cf92f-65b6-41e1-adfa-729b8d82a352"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0.24790877 0.80360919 0.40613672 0.65877726 0.31690227 0.7235624\n",
" 0.46984446 0.94761042 0.2068924 0.29758977]\n",
"[0.7340046 0.91642861 0.00531714 0.8775393 0.7058875 0.41874007\n",
" 0.23504671 0.28281132 0.99023306 0.40585885]\n"
]
}
],
"source": [
"print(np.random.random(10))\n",
"print(np.random.random(10))"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 92
},
"colab_type": "code",
"id": "qcSmNrIsCARD",
"outputId": "aa46e7b6-9848-485f-8058-48b0c16063e1"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0.64414354 0.38074849 0.66304791 0.16365073 0.96260781 0.34666184\n",
" 0.99175099 0.2350579 0.58569427 0.4066901 ]\n",
"[0.5881308 0.89771373 0.89153073 0.81583748 0.03588959 0.69175758\n",
" 0.37868094 0.51851095 0.65795147 0.19385022]\n"
]
}
],
"source": [
"print(np.random.RandomState(seed=30).random(10))\n",
"print(np.random.RandomState(seed=20).random(10))"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "qj68l3hzCEGk"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[0.5881308 0.89771373 0.89153073 0.81583748 0.03588959 0.69175758\n",
" 0.37868094 0.51851095 0.65795147 0.19385022]\n"
]
}
],
"source": [
"print(np.random.RandomState(20).random(10))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "6mJwuofS3ski"
},
"source": [
"### Phase 1: Implement and test a baseline Neural Network"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "hdSftjSfKpTM"
},
"source": [
"#### 1.1 Activation functions and derivatives\n",
"\n",
"In the below cell we define sigmoid activation function and its derivative"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "OB_Z7ydoKo7W"
},
"outputs": [],
"source": [
"# Activation function\n",
"def sigmoid(t):\n",
" return 1/(1+np.exp(-t))\n",
"\n",
"# Derivative of sigmoid\n",
"def sigmoid_derivative(p):\n",
" return p * (1 - p)\n",
"\n",
"# Activation function\n",
"def relu(t):\n",
" return np.maximum(0,t)\n",
"\n",
"# Derivative of relu\n",
"def relu_derivative(p):\n",
" return np.greater(p, 0).astype(int)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ybW08fHHDfaj"
},
"source": [
"#### 1.2 Loss functions and derivatives\n",
"\n",
"In the below cell we define rmse loss function and its derivative"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"colab_type": "code",
"id": "a47PIQ8yDfJm",
"outputId": "e15d12f0-318f-43aa-fe38-9602efe7646b"
},
"outputs": [],
"source": [
"# Loss function - RMSE\n",
"def rmse(y, ypred):\n",
" return np.sqrt(np.mean((y - ypred)**2))\n",
"\n",
"# Loss function derivative - RMSE\n",
"def rmse_derivative(y, ypred):\n",
" return 2*(y - ypred)\n",
"\n",
"# Loss function - Cross Entropy\n",
"def cross_entropy(X,y):\n",
" X = X.clip(min=1e-8,max=None)\n",
" return \n",
"\n",
"# Loss function derivative - Cross Entropy\n",
"def cross_entropy_derivative(X,y):\n",
" return -np.log(cross_entropy(X,y)) "
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "LcF4O0iwCICj"
},
"source": [
"#### 1.3 Base Class\n",
"\n",
"Below is the base class implementation for Neural Network. Please add code whereever it reads **Write your code here**"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "ORNkfSoNvbWy"
},
"outputs": [],
"source": [
"# Class definition\n",
"class NeuralNetwork:\n",
" def __init__(self, x, y, hidden_nodes, learning_rate):\n",
" \"\"\"\n",
" Class initializer. Loads the value of input data and initializes weights\n",
"\n",
" Parameters:\n",
" x: input data\n",
" y: ground truth label\n",
"\n",
" Sets class parameters:\n",
" self.x = input data\n",
" self.y = ground truth label\n",
" self.ypred = initialize prediction variable with zeros as the same shape as ground truth\n",
"\n",
" self.weights = weights for each layer\n",
" self.bias = bias for each layer\n",
"\n",
" self.hidden_nodes = number of nodes in the hidden layer\n",
" \"\"\"\n",
"\n",
" self.x = x\n",
" self.y = y\n",
" self.ypred = np.zeros(y.shape)\n",
"\n",
" self.hidden_nodes = hidden_nodes\n",
" \n",
" self.weights1= np.random.rand(self.x.shape[1], self.hidden_nodes)\n",
" self.weights2 = np.random.rand(self.hidden_nodes, 1)\n",
"\n",
" self.bias1 = np.random.rand(self.hidden_nodes)\n",
" self.bias2 = np.random.rand(1)\n",
"\n",
" self.lr = learning_rate\n",
" \n",
" \n",
" def feedforward(self):\n",
" \"\"\"\n",
" Perform feedforward operation: input -> layer1 -> layer2\n",
" \n",
" HINT:\n",
" layer1 = activation(weight1x + bias1)\n",
" layer2 = activation(weight2x + bias2)\n",
"\n",
" For example: self.layer1 = sigmoid(np.dot(self.x, self.weights1) + self.bias1)\n",
"\n",
" Parameters:\n",
" x: input data\n",
" y: ground truth label\n",
"\n",
" Return:\n",
" self.layer2: The output from layer2\n",
" \"\"\"\n",
"\n",
" # Write your code here\n",
" self.layer1 = sigmoid(np.dot(self.x, self.weights1) + self.bias1)\n",
" self.layer2 = sigmoid(np.dot(self.layer1, self.weights2) + self.bias2)\n",
" return self.layer2\n",
" \n",
" def backprop(self):\n",
" \"\"\"\n",
" Perform backpropagation to update weights and bias: loss -> layer2 -> layer1\n",
" \n",
" ...