SIT720 Machine Learning Assessment Task 5: Machine Learning Project. ©Deakin University XXXXXXXXXX1 XXXXXXXXXXSIT720 This document supplies detailed information on Assessment Task 5 for this unit. Key...

1 answer below »
Please find the assignment Attached


SIT720 Machine Learning Assessment Task 5: Machine Learning Project. ©Deakin University 1 SIT720 This document supplies detailed information on Assessment Task 5 for this unit. Key information • Due: Sunday 10 October 2021 by 8.00 pm (AEST), • Weighting: 35% Learning Outcomes This assessment assesses the following Unit Learning Outcomes (ULO) and related Graduate Learning Outcomes (GLO): Unit Learning Outcome (ULO) Graduate Learning Outcome (GLO) ULO6 - Perform model selection and compute relevant evaluation measure for a given problem. ULO7 - Use concepts of machine learning algorithms to design solution and compare multiple solutions. GLO1 - Discipline-specific knowledge and capabilities GLO2 - Communication GLO4 - Critical thinking GLO5 - Problem solving GLO6 - Self-management Purpose This assessment is an extensive machine learning project. The task is open in nature, where students should make all design decisions to solve a problem and justify their decisions. In addition, they have to design and develop solutions that are better than any existing solutions. Assessment 5 Total marks = 35 Submission Instructions a) Submit your solution codes into a notebook file with “.ipynb” extension. Write discussions and explanations including outputs and figures into a separate file and submit as a PDF file. b) Submission other than the above-mentioned file formats will not be assessed and given zero for the entire submission. c) Insert your Python code responses into the cell of your submitted “.ipynb” file followed by the question i.e., copy the question by adding a cell before the solution cell. If you need multiple cells for better presentation of the code, add question only before the first solution cell. d) Your submitted code should be executable. If your code does not generate the submitted solution, then you will get zero for that part of the marks. e) Answers must be relevant and precise. f) No hard coding is allowed. Avoid using specific value that can be calculated from the data provided. g) Use all the topics covered in the unit for answering this assignment. h) Submit your assignment after running each cell individually. i) The submitted notebook file name should be of this form “SIT720_A5_studentID.ipynb”. For example, if your student ID is 1234, then the submitted file name should be “SIT720_A5_1234.ipynb”. ________________________________________________________________________________ SIT720 Machine Learning Assessment Task 5: Machine Learning Project. ©Deakin University 2 SIT720 Questions ________________________________________________________________________________ Background In this project you are given a dataset and an article that uses this dataset. The authors have developed ten ML models for predicting survival of patients with heart failure and compared their performance. You must read the article to understand the problem, the dataset, and the methodology to complete the following tasks. Dataset The dataset contains the medical records of patients who had heart failure, collected during their follow-up period. Each patient profile has 13 clinical features. A detailed description of the dataset can be found in the Dataset section of the provided article (patient_survival_prediction.pdf). Tasks: 1. Read the article and reproduce the results presented in Table-4 using Python modules and packages (including your own script or customised codes). Write a report summarising the dataset, used ML methods, experiment protocol and results including variations, if any. During reproducing the results: (10 marks) i) you should use the same set of features used by the authors. ii) you should use the same classifier with exact parameter values. iii) you should use the same training/test splitting approach as used by the authors. iv) you should use the same pre/post processing, if any, used by the authors. v) you should report the same performance metrics as shown in Table-4. N.B. (i) Some of the ML methods are not covered in the current unit. Consider them as HD tasks i.e., based on the knowledge gained in the unit you should be able to find necessary packages and modules to reproduce the results. (ii) If you find any issue in reproducing results or some subtle variations are found due to implementation differences of packages and modules in Python then appropriate explanation of them will be considered during evaluation of your submission. (iii) Similarly, variation in results due to randomness of data splitting will also be considered during evaluation based on your explanation. (iii) Obtained marks will be proportional to the number of ML methods that you will report in your submission with correctly reproduced results. (iv) Make sure your Python code segment generates the reported results, otherwise you will receive zero marks for this task. Marking criteria: i) Unsatisfactory (x<5): tried="" to="" implement="" the="" methods="" but="" unable="" to="" follow="" the="" approach="" presented="" in="" the="" article.="" variation="" of="" marks="" in="" this="" group="" will="" depend="" on="" the="" quality="" of="" report.="" ii)="" fair=""><><6): appropriately="" implemented="" 50%="" of="" the="" methods="" presented="" in="" the="" article.="" variation="" of="" marks="" in="" this="" group="" will="" depend="" on="" the="" quality="" of="" report.="" iii)="" good=""><><8): appropriately="" implemented="" 70%="" of="" the="" methods="" presented="" in="" the="" article.="" variation="" of="" marks="" in="" this="" group="" will="" depend="" on="" the="" quality="" of="" report.="" iv)="" excellent(x="">=8): appropriately implemented >=90% of the methods presented in the article. Variation of marks in this group will depend on the quality of report. SIT720 Machine Learning Assessment Task 5: Machine Learning Project. ©Deakin University 3 SIT720 2. Design and develop your own ML solution for this problem. The proposed solution should be different from all approaches mentioned in the provided article. This does not mean that you must have to choose a new ML algorithm. You can develop a novel solution by changing the feature selection approach or parameter optimisations process of used ML methods or using different ML methods or different combinations of them. This means, the proposed system should be substantially different from the methods presented in the article but not limited to only change of ML methods. Compare the result with reported methods in the article. Write a technical report summarising your solution design and outcomes. The report should include: (20 marks) i) Motivation behind the proposed solution. ii) How the proposed solution is different from existing ones. iii) Detail description of the model including all parameters so that any reader can implement your model. iv) Description of experimental protocol. v) Evaluation metrics. vi) Present results using tables and graphs. vii) Compare and discuss results with respect to existing literatures. viii) Appropriate references (IEEE numbered). N.B. This is a HD (High Distinction) level question. Those students who target HD grade should answer this question (including answering all the above questions). For others, this question is an option. This question aims to demonstrate your expertise in the subject area and the ability to do your own research in the related area. Marking criteria: (i) Unsatisfactory (<10): an="" appropriate="" solution="" presented="" whose="" performance="" is="" lower="" than="" the="" reported="" performances="" in="" the="" article="" (table="" 11).="" the="" variation="" in="" the="" marking="" in="" this="" group="" will="" depend="" on="" the="" quality="" of="" the="" report.="" (i)="" fair="" (10="" -=""><14): an="" appropriate="" solution="" presented="" whose="" performance="" is="" at="" least="" equal="" with="" the="" lowest="" performance="" reported="" in="" the="" article="" (table="" 11).="" the="" variation="" in="" the="" marking="" in="" this="" group="" will="" depend="" on="" the="" quality="" of="" the="" report.="" (ii)="" good="" (="">=14): an appropriate solution presented whose performance is better than the best reported performances in the article
assessment-5-t2-2021-flwlwuqw.pdf
Answered 8 days AfterOct 02, 2021

Answer To: SIT720 Machine Learning Assessment Task 5: Machine Learning Project. ©Deakin University XXXXXXXXXX1...

Pritam Kumar answered on Oct 10 2021
133 Votes
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"
\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"
ageanaemiacreatinine_phosphokinasediabetesejection_fractionhigh_blood_pressureplateletsserum_creatinineserum_sodiumsexsmokingtimeDEATH_EVENT
075.005820201265000.001.913010 41
155.0078610380263358.031.11361061
265.001460200162000.001.31291171
350.011110200210000.001.91371071
465.011601200327000.002.71160081
\n",
"
"
],
"text/plain": [
" age anaemia creatinine_phosphokinase diabetes ejection_fraction \\\n",
"0 75.0 0 582 0 20 \n",
"1 55.0 0 7861 0 38 \n",
"2 65.0 0 146 0 20 \n",
"3 50.0 1 111 0 20 \n",
"4 65.0 1 160 1 20 \n",
"\n",
" high_blood_pressure platelets serum_creatinine serum_sodium sex \\\n",
"0 1 265000.00 1.9 130 1 \n",
"1 0 263358.03 1.1 136 1 \n",
"2 0 162000.00 1.3 129 1 \n",
"3 0 210000.00 1.9 137 1 \n",
"4 0 327000.00 2.7 116 0 \n",
"\n",
" smoking time DEATH_EVENT \n",
"0 0 4 1 \n",
"1 0 6 1 \n",
"2 1 7 1 \n",
"3 0 7 1 \n",
"4 0 8 1 "
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import pandas as pd\n",
"\n",
"data = pd.read_csv(\"D:\\\\New\\\\heartfailureclinicalrecordsdataset.csv\")\n",
"data.head()"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"age\n",
"anaemia\n",
"creatinine_phosphokinase\n",
"diabetes\n",
"ejection_fraction\n",
"high_blood_pressure\n",
"platelets\n",
"serum_creatinine\n",
"serum_sodium\n",
"sex\n",
"smoking\n",
"time\n",
"DEATH_EVENT\n"
]
}
],
"source": [
"for col in data.columns:\n",
" print(col)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Task 1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Random Forests feature selection through accuracy reduction"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.ensemble import RandomForestClassifier\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.ensemble import RandomForestRegressor\n",
"from sklearn.inspection import permutation_importance\n",
"from matplotlib import pyplot as plt\n",
"\n",
"plt.rcParams.update({'figure.figsize': (12.0, 8.0)})\n",
"plt.rcParams.update({'font.size': 14})"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"#train-test split for random forest feature selection process\n",
"\n",
"X = data.iloc[:, 0:11]\n",
"y = data['DEATH_EVENT']\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=12)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([0.1487753 , 0.01530943, 0.12832902, 0.01560628, 0.15776899,\n",
" 0.01319197, 0.12878872, 0.26402424, 0.09584213, 0.02159089,\n",
" 0.01077302])"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"rf = RandomForestRegressor(n_estimators=100)\n",
"rf.fit(X_train, y_train)\n",
"\n",
"rf.feature_importances_"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Features sorted by their score:\n",
"[(0.264, 'serum_creatinine'), (0.1578, 'ejection_fraction'), (0.1488, 'age'), (0.1288, 'platelets'), (0.1283, 'creatinine_phosphokinase'), (0.0958, 'serum_sodium'), (0.0216, 'sex'), (0.0156, 'diabetes'), (0.0153, 'anaemia'), (0.0132, 'high_blood_pressure'), (0.0108, 'smoking')]\n"
]
}
],
"source": [
"names = X.columns\n",
"print (\"Features sorted by their score:\")\n",
"print (sorted(zip(map(lambda x: round(x, 4), rf.feature_importances_), names), \n",
" reverse=True))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Based on the results, we select \"serum_creatinine\" and \"ejection_fraction\" as the best two features"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Classification tasks with different classifiers"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"#train-test split for classification tasks\n",
"\n",
"X = pd.DataFrame(data['serum_creatinine'])\n",
"X['ejection_fraction'] = data['ejection_fraction']\n",
"y = data['DEATH_EVENT']\n",
"\n",
"X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=12)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
"#random forest classifier\n",
"\n",
"ranfor_clf = RandomForestClassifier(n_estimators=1000, random_state=42)\n",
"ranfor_model = ranfor_clf.fit(X_train,y_train)\n",
"ranfor_model_fit = ranfor_model.predict(X_test)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.metrics import (\n",
" f1_score,\n",
" accuracy_score,\n",
" matthews_corrcoef,\n",
" roc_auc_score,\n",
" average_precision_score, \n",
" confusion_matrix \n",
")"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Random Forest:\n",
"\n",
"MCC is: 0.327\n",
"F1-score is: 0.55\n",
"Accuracy is: 0.7\n",
"TP Rate is: 0.524\n",
"TN Rate is: 0.795\n",
"ROC-AUC Score is: 0.659\n",
"PR-AUC Score is: 0.47\n"
]
}
],
"source": [
"tn, fp, fn, tp = confusion_matrix(y_test, ranfor_model_fit).ravel()\n",
"\n",
"print(\"Random Forest:\\n\")\n",
"print(\"MCC is:\",\"\", matthews_corrcoef(y_test, ranfor_model_fit).round(3))\n",
"print(\"F1-score is:\",\"\", f1_score(y_test, ranfor_model_fit).round(3))\n",
"print(\"Accuracy is:\",\"\", accuracy_score(y_test, ranfor_model_fit).round(3))\n",
"print(\"TP Rate is:\",\"\", (tp/(tp + fn)).round(3))\n",
"print(\"TN Rate is:\",\"\", (tn/(tn + fp)).round(3))\n",
"print(\"ROC-AUC Score is:\",\"\", roc_auc_score(y_test, ranfor_model_fit).round(3))\n",
"print(\"PR-AUC Score is:\",\"\", average_precision_score(y_test, ranfor_model_fit).round(3))"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"#decision tree classifier\n",
"\n",
"from sklearn.tree import DecisionTreeClassifier\n",
"dt_clf = DecisionTreeClassifier(random_state=42)\n",
"dt_model = dt_clf.fit(X_train, y_train)\n",
"dt_model_fit = dt_model.predict(X_test)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Decision Tree:\n",
"\n",
"MCC is: 0.282\n",
"F1-score is: 0.513\n",
"Accuracy is: 0.683\n",
"TP Rate is: 0.476\n",
"TN Rate is: 0.795\n",
"ROC-AUC Score is: 0.636\n",
"PR-AUC Score is: 0.448\n"
]
}
],
"source": [
"tn, fp, fn, tp = confusion_matrix(y_test, dt_model_fit).ravel()\n",
"\n",
"print(\"Decision Tree:\\n\")\n",
"print(\"MCC is:\",\"\", matthews_corrcoef(y_test, dt_model_fit).round(3))\n",
"print(\"F1-score is:\",\"\", f1_score(y_test, dt_model_fit).round(3))\n",
"print(\"Accuracy is:\",\"\", accuracy_score(y_test, dt_model_fit).round(3))\n",
"print(\"TP Rate is:\",\"\", (tp/(tp + fn)).round(3))\n",
"print(\"TN Rate is:\",\"\", (tn/(tn + fp)).round(3))\n",
"print(\"ROC-AUC Score is:\",\"\", roc_auc_score(y_test, dt_model_fit).round(3))\n",
"print(\"PR-AUC Score is:\",\"\", average_precision_score(y_test, dt_model_fit).round(3))"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"#gradient boosting\n",
"\n",
"from sklearn.ensemble import GradientBoostingClassifier\n",
"gradboost_clf = GradientBoostingClassifier()\n",
"gradboost_model = gradboost_clf.fit(X_train,y_train)\n",
"gradboost_model_fit = gradboost_model.predict(X_test)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Gradient Boosting:\n",
"\n",
"MCC is: 0.332\n",
"F1-score is: 0.485\n",
"Accuracy is: 0.717\n",
"TP Rate is: 0.381\n",
"TN Rate is: 0.897\n",
"ROC-AUC Score is: 0.639\n",
"PR-AUC Score is: 0.471\n"
]
}
],
"source": [
"tn, fp, fn, tp = confusion_matrix(y_test, gradboost_model_fit).ravel()\n",
"\n",
"print(\"Gradient Boosting:\\n\")\n",
"print(\"MCC is:\",\"\", matthews_corrcoef(y_test, gradboost_model_fit).round(3))\n",
"print(\"F1-score is:\",\"\", f1_score(y_test, gradboost_model_fit).round(3))\n",
"print(\"Accuracy is:\",\"\", accuracy_score(y_test, gradboost_model_fit).round(3))\n",
"print(\"TP Rate is:\",\"\", (tp/(tp + fn)).round(3))\n",
"print(\"TN Rate is:\",\"\", (tn/(tn + fp)).round(3))\n",
"print(\"ROC-AUC Score is:\",\"\", roc_auc_score(y_test, gradboost_model_fit).round(3))\n",
"print(\"PR-AUC Score is:\",\"\", average_precision_score(y_test, gradboost_model_fit).round(3))"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"#linear regression\n",
"\n",
"from sklearn.linear_model import LogisticRegression\n",
"linreg_clf = LogisticRegression()\n",
"linreg_model = linreg_clf.fit(X_train,y_train)\n",
"linreg_model_fit = linreg_model.predict(X_test)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Linear Regression:\n",
"\n",
"MCC is: 0.475\n",
"F1-score is: 0.533\n",
"Accuracy is: 0.767\n",
"TP Rate is: 0.381\n",
"TN Rate is: 0.974\n",
"ROC-AUC Score is: 0.678\n",
"PR-AUC Score is: 0.555\n"
]
}
],
"source": [
"tn, fp, fn, tp = confusion_matrix(y_test, linreg_model_fit).ravel()\n",
"\n",
"print(\"Linear Regression:\\n\")\n",
"print(\"MCC is:\",\"\", matthews_corrcoef(y_test, linreg_model_fit).round(3))\n",
"print(\"F1-score is:\",\"\", f1_score(y_test, linreg_model_fit).round(3))\n",
"print(\"Accuracy is:\",\"\", accuracy_score(y_test, linreg_model_fit).round(3))\n",
"print(\"TP Rate is:\",\"\", (tp/(tp + fn)).round(3))\n",
"print(\"TN Rate is:\",\"\", (tn/(tn + fp)).round(3))\n",
"print(\"ROC-AUC Score is:\",\"\", roc_auc_score(y_test, linreg_model_fit).round(3))\n",
"print(\"PR-AUC Score is:\",\"\", average_precision_score(y_test, linreg_model_fit).round(3))"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"#One rule clssifier\n",
"\n",
"from sklearn.dummy import DummyClassifier\n",
"dummy_clf = DummyClassifier(strategy=\"stratified\")\n",
"onerule_model = dummy_clf.fit(X_train, y_train)\n",
"onerule_model_fit = dummy_clf.predict(X_test)"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"One Rule Classifier:\n",
"\n",
"MCC is: -0.172\n",
"F1-score is: 0.238\n",
"Accuracy is: 0.467\n",
"TP Rate is: 0.238\n",
"TN Rate is: 0.59\n",
"ROC-AUC Score is: 0.414\n",
"PR-AUC Score is: 0.323\n"
]
}
],
"source": [
"tn, fp, fn, tp = confusion_matrix(y_test, onerule_model_fit).ravel()\n",
"\n",
"print(\"One Rule Classifier:\\n\")\n",
"print(\"MCC is:\",\"\", matthews_corrcoef(y_test, onerule_model_fit).round(3))\n",
"print(\"F1-score...
SOLUTION.PDF
SOLUTION.PDF

Answer To This Question Is Available To Download

Related Questions & Answers

More Questions »

Submit New Assignment

Copy and Paste Your Assignment Here
Have files?