{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "accelerator": "GPU", "colab": { "name": "AnujChaturvedi.ipynb", "provenance": [], "collapsed_sections": [], "toc_visible": true },...

{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"accelerator": "GPU",
"colab": {
"name": "assignment2-pdqaxoij.ipynb",
"provenance": [],
"collapsed_sections": [
"twFQbltnm8da"
],
"machine_shape": "hm"
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.7"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "wjif_5FsdOFz"
},
"source": [
"# SIT744 Assignment 2: Transfer learning and Efficient Training of Deep Learning Models \n",
"\n",
"

\n",
"

Due: 8:00pm 17 May 2021 (Monday)

\n",
"\n",
"This is an individual assignment. It contributes 45% to your final mark. Read the assignment instruction carefully.\n",
"\n",
"

What to submit

\n",
"\n",
"

\n",
"This assignment is to be completed individually and submitted to CloudDeakin. By the due date, you are required to submit the following files to the corresponding Assignment (Dropbox) in CloudDeakin:\n",
"\n",
"

\n",
"
1. \t[YourID]_assignment2_solution.ipynp: This is your Python notebook solution source file.
\n",
"
2. \t[YourID]_assingment2_output.html: This is the output of your Python notebook solution exported in HTML format.
\n",
"
3. \tExtra files needed to complete your assignment, if any (e.g., images used in your answers).
\n",
"

\n",
"

\n",
"\n",
"

\n",
"For example, if your student ID is: 123456, you will then need to submit the following files:\n",
"

\n",
"
• 123456_assignment2_solution.ipynp
\n",
"
• 123456_assignment2_output.html
\n",
"

\n",
"

\n",
"\n",
"

Warning

\n",
"\n",
"Some components of this assignment may involve heavy computation that runs for a long duration. Please start early to avoid missing the assignment due date.\n",
"\n",
"

Marking criteria

\n",
"\n",
"

\n",
"Your submission will be marked using the following criteria.\n",
"\n",
"

\n",
"
• Showing good effort through completed tasks.
\n",
"
• Applying deep learning theory to design suitable deep learning solutions for the tasks.
\n",
"
• Critically evaluating and reflecting on the pros and cons of various design decisions.
\n",
"
• Demonstrating creativity and resourcefulness in providing unique individual solutions.
\n",
"
• Showing attention to details through a good quality assignment report.
\n",
"

\n",
"

\n",
"\n",
"

\n",
"Indicative weights of various tasks are provided, but the assignment will be marked by the overall quality per the above criteria.\n",
"

\n",
"

"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "twFQbltnm8da"
},
"source": [
"## Assignment objective\n",
"\n",
"This assignment is to feedback on your learning in deep learning theory and its application to data analytics or artificial intelligence problems. \n",
"\n",
"It builds on Assignment 1 but requires a higher level of mastery of deep learning theory and programming/engineering skills. In particular, you will experience training a much deeper network on a large-scale dataset. You will encounter practical issues that help you consolidate textbook learning. "
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "3ITc1hw_o7qV"
},
"source": [
"## Task 1 Solving Fashion-MNIST with Convolutional Neural Networks\n",
"\n",
"*(weight ~15%)*\n",
"\n",
"In Assignment 1, you tackled the image classification problem in Fashion-MNIST. There, you used a Densely Connected Neural Network. You should now know that is not an optimal model architecture for the problem. In Assignment 2, you will apply the best practices of deep-learning computer vision to achieve better image classification performance."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "zdHwmgwOpEfx"
},
"source": [
"### Task 1.1 Revisit Fashion-MNIST classification with DNN\n",
"\n",
"*(weight ~1%)*\n",
"\n",
"Review your Assignment 1 solution, and reproduce the experiment here. Try to improve the model without changing the model architecture.\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "EMaj-3OSBKy8"
},
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"import tensorflow as tf\n",
"from keras.datasets import fashion_mnist\n",
"from keras.utils import to_categorical\n",
"from tensorflow.keras.models import Sequential\n",
"from tensorflow.keras.layers import Dense"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ZbANag4oBkd4",
"outputId": "cdecd38c-c023-4765-c9a0-7bf768f179a0"
},
"source": [
"# load dataset\n",
"(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()\n",
"\n",
"# Change from matrix to array of dimension 28x28 to array of dimension 784\n",
"dim_data = np.prod(x_train.shape[1:])\n",
"x_train = x_train.reshape(x_train.shape[0], dim_data)\n",
"x_test = x_test.reshape(x_test.shape[0], dim_data)\n",
"\n",
"# Pre-processing\n",
"x_train = x_train.astype('float32') / 255.\n",
"x_test = x_test.astype('float32') / 255.\n",
"\n",
"# Change the labels from integer to categorical data\n",
"y_train = to_categorical(y_train)\n",
"y_test = to_categorical(y_test)\n",
"\n",
"class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',\n",
" 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']\n",
"\n",
"model = Sequential()\n",
"model.add(Dense(512, activation='relu', input_shape=(dim_data,)))\n",
"model.add(Dense(512, activation='relu'))\n",
"model.add(Dense(10, activation='softmax'))\n",
"model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])\n",
"model.summary()"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Model: \"sequential_3\"\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"dense_6 (Dense) (None, 512) 401920 \n",
"_________________________________________________________________\n",
"dense_7 (Dense) (None, 512) 262656 \n",
"_________________________________________________________________\n",
"dense_8 (Dense) (None, 10) 5130 \n",
"=================================================================\n",
"Total params: 669,706\n",
"Trainable params: 669,706\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "mAgf4aKpDYny",
"outputId": "af5164c2-12de-4133-ed98-65662322bf74"
},
"source": [
"model.fit(x_train, y_train, batch_size=64, epochs=20, verbose=1,\n",
" validation_data=(x_test, y_test))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Epoch 1/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2659 - accuracy: 0.9124 - val_loss: 0.7135 - val_accuracy: 0.8700\n",
"Epoch 2/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2571 - accuracy: 0.9151 - val_loss: 0.6739 - val_accuracy: 0.8680\n",
"Epoch 3/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2504 - accuracy: 0.9165 - val_loss: 0.6280 - val_accuracy: 0.8679\n",
"Epoch 4/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2465 - accuracy: 0.9169 - val_loss: 0.6858 - val_accuracy: 0.8614\n",
"Epoch 5/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2449 - accuracy: 0.9182 - val_loss: 0.6626 - val_accuracy: 0.8608\n",
"Epoch 6/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2354 - accuracy: 0.9208 - val_loss: 0.6124 - val_accuracy: 0.8842\n",
"Epoch 7/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2347 - accuracy: 0.9199 - val_loss: 0.7341 - val_accuracy: 0.8682\n",
"Epoch 8/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2272 - accuracy: 0.9236 - val_loss: 0.7431 - val_accuracy: 0.8824\n",
"Epoch 9/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2292 - accuracy: 0.9243 - val_loss: 0.6920 - val_accuracy: 0.8834\n",
"Epoch 10/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2279 - accuracy: 0.9251 - val_loss: 0.6480 - val_accuracy: 0.8864\n",
"Epoch 11/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2179 - accuracy: 0.9249 - val_loss: 0.7226 - val_accuracy: 0.8817\n",
"Epoch 12/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2185 - accuracy: 0.9271 - val_loss: 0.7541 - val_accuracy: 0.8837\n",
"Epoch 13/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2179 - accuracy: 0.9282 - val_loss: 0.7805 - val_accuracy: 0.8839\n",
"Epoch 14/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2125 - accuracy: 0.9292 - val_loss: 0.7474 - val_accuracy: 0.8846\n",
"Epoch 15/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2109 - accuracy: 0.9289 - val_loss: 0.7409 - val_accuracy: 0.8880\n",
"Epoch 16/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2047 - accuracy: 0.9310 - val_loss: 0.8187 - val_accuracy: 0.8873\n",
"Epoch 17/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2054 - accuracy: 0.9310 - val_loss: 0.8037 - val_accuracy: 0.8799\n",
"Epoch 18/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.2009 - accuracy: 0.9331 - val_loss: 0.8599 - val_accuracy: 0.8767\n",
"Epoch 19/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.1977 - accuracy: 0.9329 - val_loss: 0.8618 - val_accuracy: 0.8848\n",
"Epoch 20/20\n",
"938/938 [==============================] - 3s 3ms/step - loss: 0.1973 - accuracy: 0.9334 - val_loss: 0.7936 - val_accuracy: 0.8837\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "cfTL_5kPsl4j"
},
"source": [
"### Task 1.2 Train a ConvNet from scratch\n",
"\n",
"*(weight ~5%)*\n",
"\n",
"Build a ConvNet to replace the densely connected network in Task 1.1. Report the classification accuracy on the test set. Aim to achieve higher accuracy. \n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "r6RA6d2gJQDQ"
},
"source": [
"# load dataset\n",
"(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()\n",
"\n",
"# Reshape input data from (28, 28) to (28, 28, 1)\n",
"w, h = 28, 28\n",
"x_train = x_train.reshape(x_train.shape[0], w, h, 1)\n",
"x_test = x_test.reshape(x_test.shape[0], w, h, 1)\n",
"\n",
"# Pre-processing\n",
"x_train = x_train.astype('float32') / 255.\n",
"x_test = x_test.astype('float32') / 255.\n",
"\n",
"# Change the labels from integer to categorical data\n",
"y_train = to_categorical(y_train,10)\n",
"y_test = to_categorical(y_test,10)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "twCrAhTuHsH4",
"outputId": "a5c9ae1e-03f4-4693-897b-74e564e965fb"
},
"source": [
"model = tf.keras.Sequential()\n",
"# Must define the input shape in the first layer of the neural network\n",
"model.add(tf.keras.layers.Conv2D(filters=64, kernel_size=2, padding='same', activation='relu', input_shape=(28,28,1))) \n",
"model.add(tf.keras.layers.MaxPooling2D(pool_size=2))\n",
"model.add(tf.keras.layers.Dropout(0.3))\n",
"model.add(tf.keras.layers.Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'))\n",
"model.add(tf.keras.layers.MaxPooling2D(pool_size=2))\n",
"model.add(tf.keras.layers.Dropout(0.3))\n",
"model.add(tf.keras.layers.Flatten())\n",
"model.add(tf.keras.layers.Dense(256, activation='relu'))\n",
"model.add(tf.keras.layers.Dropout(0.5))\n",
"model.add(tf.keras.layers.Dense(10, activation='softmax'))\n",
"# Take a look at the model summary\n",
"model.summary()"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Model: \"sequential\"\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"conv2d (Conv2D) (None, 28, 28, 64) 320 \n",
"_________________________________________________________________\n",
"max_pooling2d (MaxPooling2D) (None, 14, 14, 64) 0 \n",
"_________________________________________________________________\n",
"dropout (Dropout) (None, 14, 14, 64) 0 \n",
"_________________________________________________________________\n",
"conv2d_1 (Conv2D) (None, 14, 14, 32) 8224 \n",
"_________________________________________________________________\n",
"max_pooling2d_1 (MaxPooling2 (None, 7, 7, 32) 0 \n",
"_________________________________________________________________\n",
"dropout_1 (Dropout) (None, 7, 7, 32) 0 \n",
"_________________________________________________________________\n",
"flatten (Flatten) (None, 1568) 0 \n",
"_________________________________________________________________\n",
"dense (Dense) (None, 256) 401664 \n",
"_________________________________________________________________\n",
"dropout_2 (Dropout) (None, 256) 0 \n",
"_________________________________________________________________\n",
"dense_1 (Dense) (None, 10) 2570 \n",
"=================================================================\n",
"Total params: 412,778\n",
"Trainable params: 412,778\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "hWcE_Yo5Igpr"
},
"source": [
"model.compile(loss='categorical_crossentropy',\n",
" optimizer='adam',\n",
" metrics=['accuracy'])"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "HAMUpguYIkok",
"outputId": "adc8d520-e95f-4a1b-a216-de68e71dfe72"
},
"source": [
"history = model.fit(x_train,\n",
" y_train,\n",
" batch_size=64,\n",
" epochs=20,\n",
" validation_data=(x_test, y_test))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Epoch 1/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.2230 - accuracy: 0.9168 - val_loss: 0.2257 - val_accuracy: 0.9163\n",
"Epoch 2/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.2206 - accuracy: 0.9175 - val_loss: 0.2217 - val_accuracy: 0.9220\n",
"Epoch 3/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.2172 - accuracy: 0.9185 - val_loss: 0.2198 - val_accuracy: 0.9203\n",
"Epoch 4/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.2152 - accuracy: 0.9208 - val_loss: 0.2261 - val_accuracy: 0.9220\n",
"Epoch 5/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.2123 - accuracy: 0.9213 - val_loss: 0.2166 - val_accuracy: 0.9225\n",
"Epoch 6/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.2122 - accuracy: 0.9210 - val_loss: 0.2213 - val_accuracy: 0.9237\n",
"Epoch 7/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.2083 - accuracy: 0.9209 - val_loss: 0.2227 - val_accuracy: 0.9205\n",
"Epoch 8/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.2076 - accuracy: 0.9233 - val_loss: 0.2169 - val_accuracy: 0.9231\n",
"Epoch 9/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.2049 - accuracy: 0.9230 - val_loss: 0.2182 - val_accuracy: 0.9238\n",
"Epoch 10/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.2014 - accuracy: 0.9244 - val_loss: 0.2186 - val_accuracy: 0.9216\n",
"Epoch 11/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.2053 - accuracy: 0.9245 - val_loss: 0.2194 - val_accuracy: 0.9246\n",
"Epoch 12/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.1989 - accuracy: 0.9262 - val_loss: 0.2180 - val_accuracy: 0.9225\n",
"Epoch 13/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.1999 - accuracy: 0.9234 - val_loss: 0.2200 - val_accuracy: 0.9239\n",
"Epoch 14/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.1989 - accuracy: 0.9259 - val_loss: 0.2198 - val_accuracy: 0.9223\n",
"Epoch 15/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.1940 - accuracy: 0.9269 - val_loss: 0.2261 - val_accuracy: 0.9208\n",
"Epoch 16/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.1958 - accuracy: 0.9270 - val_loss: 0.2282 - val_accuracy: 0.9191\n",
"Epoch 17/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.1933 - accuracy: 0.9271 - val_loss: 0.2168 - val_accuracy: 0.9245\n",
"Epoch 18/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.1924 - accuracy: 0.9274 - val_loss: 0.2177 - val_accuracy: 0.9222\n",
"Epoch 19/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.1915 - accuracy: 0.9282 - val_loss: 0.2206 - val_accuracy: 0.9229\n",
"Epoch 20/20\n",
"938/938 [==============================] - 4s 5ms/step - loss: 0.1897 - accuracy: 0.9290 - val_loss: 0.2204 - val_accuracy: 0.9225\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "7oOev2ZYPRrK",
"outputId": "29eaddfe-2029-4580-a93b-fc65df2d9732"
},
"source": [
"# Evaluate the model on test set\n",
"score = model.evaluate(x_test, y_test, verbose=0)\n",
"\n",
"# Print test accuracy\n",
"print('\\n', 'Test accuracy:', score[1])"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"\n",
" Test accuracy: 0.9225000143051147\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "qbhr44DnMbKZ"
},
"source": [
"\n",
"### Task 1.3 Build an input pipeline for data augmentation\n",
"\n",
"*(weight ~3%)*\n",
"\n",
"Build a data preprocessing pipeline to perform data augmentation. (You may use Keras ImageDataGenerator or write your own transformations.)\n",
"\n",
"- Report the new classification accuracy. Make sure that you use the same number of training epochs as in Task 1.2.\n",
"\n",
"- (Optional) Profile your input pipeline to identify the most time-consuming operation. What actions have you taken to address that slow operation? (*Hint: You may use the [TensorFlow Profiler](https://github.com/tensorflow/profiler).*)"
]
},
{
"cell_type": "code",
"metadata": {
"id": "ABlA59tc6530"
},
"source": [
"from tensorflow.keras.preprocessing.image import ImageDataGenerator"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "ZP4UgsSf6OqM"
},
"source": [
"\taug = ImageDataGenerator(\n",
"\t\trotation_range=20,\n",
"\t\tzoom_range=0.15,\n",
"\t\twidth_shift_range=0.2,\n",
"\t\theight_shift_range=0.2,\n",
"\t\tshear_range=0.15,\n",
"\t\thorizontal_flip=True,\n",
"\t\tfill_mode=\"nearest\")"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "bj4w1A8x_WAQ",
"outputId": "deb8c27a-f0d9-48dd-a6ea-e783e660d8b9"
},
"source": [
"model.fit(\n",
"\tx=aug.flow(x_train, y_train, batch_size=64),\n",
"\tvalidation_data=(x_test, y_test),\n",
"\tsteps_per_epoch=len(x_train) // 64,\n",
"\tepochs=20)"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Epoch 1/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 1.0696 - accuracy: 0.6122 - val_loss: 0.3945 - val_accuracy: 0.8635\n",
"Epoch 2/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 0.8348 - accuracy: 0.6923 - val_loss: 0.3760 - val_accuracy: 0.8698\n",
"Epoch 3/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 0.7712 - accuracy: 0.7186 - val_loss: 0.3519 - val_accuracy: 0.8754\n",
"Epoch 4/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 0.7259 - accuracy: 0.7350 - val_loss: 0.3629 - val_accuracy: 0.8709\n",
"Epoch 5/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 0.7009 - accuracy: 0.7426 - val_loss: 0.3665 - val_accuracy: 0.8679\n",
"Epoch 6/20\n",
"937/937 [==============================] - 18s 20ms/step - loss: 0.6795 - accuracy: 0.7522 - val_loss: 0.3917 - val_accuracy: 0.8587\n",
"Epoch 7/20\n",
"937/937 [==============================] - 18s 20ms/step - loss: 0.6606 - accuracy: 0.7586 - val_loss: 0.3838 - val_accuracy: 0.8644\n",
"Epoch 8/20\n",
"937/937 [==============================] - 18s 20ms/step - loss: 0.6458 - accuracy: 0.7648 - val_loss: 0.3844 - val_accuracy: 0.8635\n",
"Epoch 9/20\n",
"937/937 [==============================] - 18s 20ms/step - loss: 0.6313 - accuracy: 0.7701 - val_loss: 0.3576 - val_accuracy: 0.8709\n",
"Epoch 10/20\n",
"937/937 [==============================] - 18s 20ms/step - loss: 0.6226 - accuracy: 0.7721 - val_loss: 0.3821 - val_accuracy: 0.8618\n",
"Epoch 11/20\n",
"937/937 [==============================] - 18s 20ms/step - loss: 0.6168 - accuracy: 0.7751 - val_loss: 0.3623 - val_accuracy: 0.8727\n",
"Epoch 12/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 0.6053 - accuracy: 0.7789 - val_loss: 0.3639 - val_accuracy: 0.8724\n",
"Epoch 13/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 0.5976 - accuracy: 0.7808 - val_loss: 0.3653 - val_accuracy: 0.8712\n",
"Epoch 14/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 0.5936 - accuracy: 0.7828 - val_loss: 0.3668 - val_accuracy: 0.8688\n",
"Epoch 15/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 0.5903 - accuracy: 0.7847 - val_loss: 0.3613 - val_accuracy: 0.8724\n",
"Epoch 16/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 0.5841 - accuracy: 0.7859 - val_loss: 0.3672 - val_accuracy: 0.8662\n",
"Epoch 17/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 0.5803 - accuracy: 0.7880 - val_loss: 0.3647 - val_accuracy: 0.8719\n",
"Epoch 18/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 0.5676 - accuracy: 0.7938 - val_loss: 0.3652 - val_accuracy: 0.8699\n",
"Epoch 19/20\n",
"937/937 [==============================] - 19s 20ms/step - loss: 0.5711 - accuracy: 0.7924 - val_loss: 0.3788 - val_accuracy: 0.8660\n",
"Epoch 20/20\n",
"937/937 [==============================] - 18s 20ms/step - loss: 0.5661 - accuracy: 0.7934 - val_loss: 0.3636 - val_accuracy: 0.8730\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
""
]
},
"metadata": {
"tags": []
},
"execution_count": 12
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "utL9XJtuJAtw",
"outputId": "59e4cb1f-12af-4b6b-86b1-19ae752ae57e"
},
"source": [
"# Evaluate the model on test set\n",
"score = model.evaluate(x_test, y_test, verbose=0)\n",
"\n",
"# Print test accuracy\n",
"print('\\n', 'Test accuracy:', score[1])"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"\n",
" Test accuracy: 0.8730000257492065\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "prctXU4BswKK"
},
"source": [
"### Task 1.4 Fashion-MNIST with transfer learning\n",
"\n",
"*(weight ~3%)*\n",
"\n",
"Use a pretrained model as the convolutional base to improve the classification performance. (Hint: You may use models in Keras Applications or those in the TensorFlow Hub.)\n",
"\n",
"- Try both with fine-tuning and without fine-tuning.\n",
"- Report the model performance as before.\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "1Awg4vtGxzlr"
},
"source": [
"import time\n",
"from keras.models import Sequential, Model\n",
"from keras.layers import Dense, Dropout, Activation, Flatten\n",
"from keras.layers import Conv2D, MaxPooling2D"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "77Vqf9aZi1K-"
},
"source": [
"num_classes = 10\n",
"\n",
"(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()\n",
"\n",
"#Pre-process the data\n",
"x_train = preprocess_input(x_train)\n",
"x_test = preprocess_input(x_test)\n",
"\n",
"# Reshape input data from (28, 28) to (28, 28, 1)\n",
"w, h = 28, 28\n",
"x_train = x_train.reshape(x_train.shape[0], w, h, 1)\n",
"x_test = x_test.reshape(x_test.shape[0], w, h, 1)\n",
"\n",
"datagen = ImageDataGenerator(\n",
"\t\trotation_range=20,\n",
"\t\tzoom_range=0.15,\n",
"\t\twidth_shift_range=0.2,\n",
"\t\theight_shift_range=0.2,\n",
"\t\tshear_range=0.15,\n",
"\t\thorizontal_flip=True,\n",
"\t\tfill_mode=\"nearest\")\n",
"datagen.fit(x_train)\n",
"\n",
"y_train = to_categorical(y_train, num_classes)\n",
"y_test = to_categorical(y_test, num_classes)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "JXht9kNphSC_",
"outputId": "170d25b3-192e-44b9-ff8f-da810b422962"
},
"source": [
"model = Sequential()\n",
"\n",
"# (28,28,1)\n",
"\n",
"model.add(Conv2D(32,(5,5),input_shape=(28,28,1),activation='relu'))\n",
"# (24,24,32)\n",
"\n",
"model.add(Conv2D(16,(5,5),activation='relu'))\n",
"# (20,20,16)\n",
"\n",
"model.add(MaxPooling2D(pool_size=(2,2)))\n",
"# (10,10,16)\n",
"\n",
"model.add(Conv2D(8,(3,3),activation='relu'))\n",
"# (8,8,8)\n",
"\n",
"model.add(Flatten())\n",
"\n",
"model.add(Dropout(0.4))\n",
"\n",
"model.add(Dense(128,activation='relu'))\n",
"\n",
"model.add(Dense(64,activation='relu'))\n",
"\n",
"model.add(Dense(10,activation='softmax'))\n",
"\n",
"model.summary()"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Model: \"sequential_5\"\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"conv2d_26 (Conv2D) (None, 24, 24, 32) 832 \n",
"_________________________________________________________________\n",
"conv2d_27 (Conv2D) (None, 20, 20, 16) 12816 \n",
"_________________________________________________________________\n",
"max_pooling2d_2 (MaxPooling2 (None, 10, 10, 16) 0 \n",
"_________________________________________________________________\n",
"conv2d_28 (Conv2D) (None, 8, 8, 8) 1160 \n",
"_________________________________________________________________\n",
"flatten_2 (Flatten) (None, 512) 0 \n",
"_________________________________________________________________\n",
"dropout_4 (Dropout) (None, 512) 0 \n",
"_________________________________________________________________\n",
"dense_12 (Dense) (None, 128) 65664 \n",
"_________________________________________________________________\n",
"dense_13 (Dense) (None, 64) 8256 \n",
"_________________________________________________________________\n",
"dense_14 (Dense) (None, 10) 650 \n",
"=================================================================\n",
"Total params: 89,378\n",
"Trainable params: 89,378\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "VnyM-5sIJJl4"
},
"source": [
"model.compile(optimizer='adam',\n",
" loss='categorical_crossentropy',\n",
" metrics=['accuracy'])"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "EG5Rlm2KyC1i",
"outputId": "d28295b6-7c9b-477f-a2ea-790c67e0640d"
},
"source": [
"model.fit(x_train,y_train,\n",
" validation_data=(x_test,y_test),\n",
" epochs=10,\n",
" batch_size=64,\n",
" verbose=1,\n",
" shuffle=True)"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 1.9489 - accuracy: 0.5068 - val_loss: 0.5166 - val_accuracy: 0.8096\n",
"Epoch 2/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.5449 - accuracy: 0.7952 - val_loss: 0.4515 - val_accuracy: 0.8344\n",
"Epoch 3/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.4783 - accuracy: 0.8209 - val_loss: 0.3987 - val_accuracy: 0.8499\n",
"Epoch 4/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.4200 - accuracy: 0.8404 - val_loss: 0.3773 - val_accuracy: 0.8614\n",
"Epoch 5/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.4015 - accuracy: 0.8516 - val_loss: 0.3604 - val_accuracy: 0.8636\n",
"Epoch 6/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3786 - accuracy: 0.8555 - val_loss: 0.3682 - val_accuracy: 0.8667\n",
"Epoch 7/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3504 - accuracy: 0.8685 - val_loss: 0.3370 - val_accuracy: 0.8792\n",
"Epoch 8/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3366 - accuracy: 0.8757 - val_loss: 0.3278 - val_accuracy: 0.8761\n",
"Epoch 9/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3293 - accuracy: 0.8746 - val_loss: 0.3315 - val_accuracy: 0.8771\n",
"Epoch 10/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3153 - accuracy: 0.8811 - val_loss: 0.3073 - val_accuracy: 0.8868\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
""
]
},
"metadata": {
"tags": []
},
"execution_count": 70
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6vMltfbsz0kR"
},
"source": [
"#### Without Fine Tuning"
]
},
{
"cell_type": "code",
"metadata": {
"id": "RyHSpfGgybqr"
},
"source": [
"for layer in model.layers[:5]:\n",
" layer.trainable = False"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "syf9oeE3ydWF"
},
"source": [
"x = model.layers[4].output\n",
"x = Dropout(0.5)(x)\n",
"x = Dense(32,activation='relu')(x)\n",
"x = Dense(16,activation='relu')(x)\n",
"predictions = Dense(10,activation='softmax')(x)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "yD_FsujEysok",
"outputId": "0a36cf2e-3d3f-4530-93f6-f93ecff8c614"
},
"source": [
"tl_model = Model(model.input,predictions)\n",
"tl_model.summary()"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Model: \"model_2\"\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"conv2d_26_input (InputLayer) [(None, 28, 28, 1)] 0 \n",
"_________________________________________________________________\n",
"conv2d_26 (Conv2D) (None, 24, 24, 32) 832 \n",
"_________________________________________________________________\n",
"conv2d_27 (Conv2D) (None, 20, 20, 16) 12816 \n",
"_________________________________________________________________\n",
"max_pooling2d_2 (MaxPooling2 (None, 10, 10, 16) 0 \n",
"_________________________________________________________________\n",
"conv2d_28 (Conv2D) (None, 8, 8, 8) 1160 \n",
"_________________________________________________________________\n",
"flatten_2 (Flatten) (None, 512) 0 \n",
"_________________________________________________________________\n",
"dropout_5 (Dropout) (None, 512) 0 \n",
"_________________________________________________________________\n",
"dense_15 (Dense) (None, 32) 16416 \n",
"_________________________________________________________________\n",
"dense_16 (Dense) (None, 16) 528 \n",
"_________________________________________________________________\n",
"dense_17 (Dense) (None, 10) 170 \n",
"=================================================================\n",
"Total params: 31,922\n",
"Trainable params: 17,114\n",
"Non-trainable params: 14,808\n",
"_________________________________________________________________\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "49xuYbrQzDT_"
},
"source": [
"tl_model.compile(optimizer='adam',\n",
" loss='categorical_crossentropy',\n",
" metrics=['accuracy'])"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "NJqXBOCdzOxq",
"outputId": "5ce97cf3-8879-4029-b71f-fc1aa5e6f21e"
},
"source": [
"t=time.time()\n",
"tl_model.fit(x_train,y_train,\n",
" validation_data=(x_test,y_test),\n",
" batch_size=64,\n",
" epochs=10,\n",
" verbose=1,\n",
" shuffle=True)\n",
"print('Training time: %s' % (time.time()-t))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 1.0937 - accuracy: 0.6520 - val_loss: 0.3782 - val_accuracy: 0.8647\n",
"Epoch 2/10\n",
"938/938 [==============================] - 3s 4ms/step - loss: 0.4071 - accuracy: 0.8538 - val_loss: 0.3501 - val_accuracy: 0.8736\n",
"Epoch 3/10\n",
"938/938 [==============================] - 3s 4ms/step - loss: 0.3774 - accuracy: 0.8626 - val_loss: 0.3310 - val_accuracy: 0.8801\n",
"Epoch 4/10\n",
"938/938 [==============================] - 3s 4ms/step - loss: 0.3545 - accuracy: 0.8715 - val_loss: 0.3235 - val_accuracy: 0.8814\n",
"Epoch 5/10\n",
"938/938 [==============================] - 3s 4ms/step - loss: 0.3473 - accuracy: 0.8732 - val_loss: 0.3206 - val_accuracy: 0.8838\n",
"Epoch 6/10\n",
"938/938 [==============================] - 3s 4ms/step - loss: 0.3426 - accuracy: 0.8733 - val_loss: 0.3142 - val_accuracy: 0.8831\n",
"Epoch 7/10\n",
"938/938 [==============================] - 3s 4ms/step - loss: 0.3340 - accuracy: 0.8782 - val_loss: 0.3131 - val_accuracy: 0.8842\n",
"Epoch 8/10\n",
"938/938 [==============================] - 3s 4ms/step - loss: 0.3252 - accuracy: 0.8805 - val_loss: 0.3099 - val_accuracy: 0.8879\n",
"Epoch 9/10\n",
"938/938 [==============================] - 3s 4ms/step - loss: 0.3200 - accuracy: 0.8802 - val_loss: 0.3104 - val_accuracy: 0.8860\n",
"Epoch 10/10\n",
"938/938 [==============================] - 3s 4ms/step - loss: 0.3274 - accuracy: 0.8807 - val_loss: 0.3066 - val_accuracy: 0.8869\n",
"Training time: 33.95117425918579\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "QPE3BCrtz5fG",
"outputId": "fa9c5503-a6c6-4b8c-cb98-4fb437049bba"
},
"source": [
"# Evaluate the model on test set\n",
"score = tl_model.evaluate(x_test, y_test, verbose=0)\n",
"\n",
"# Print test accuracy\n",
"print('\\n', 'Test accuracy:', score[1])"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"\n",
" Test accuracy: 0.886900007724762\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2PNbkV0w0Gug"
},
"source": [
"#### With Fine Tuning"
]
},
{
"cell_type": "code",
"metadata": {
"id": "q4nSJ0ez0LmK"
},
"source": [
"for layer in model.layers[:5]:\n",
" layer.trainable = True"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "ft7w-9X70K_S"
},
"source": [
"x = model.layers[4].output\n",
"x = Dropout(0.5)(x)\n",
"x = Dense(32,activation='relu')(x)\n",
"x = Dense(16,activation='relu')(x)\n",
"predictions = Dense(10,activation='softmax')(x)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "gkt-w3_10KUe",
"outputId": "57879d57-1e7f-4258-95e0-5a5d897a7e73"
},
"source": [
"tl_model = Model(model.input,predictions)\n",
"tl_model.summary()"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Model: \"model_3\"\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"conv2d_26_input (InputLayer) [(None, 28, 28, 1)] 0 \n",
"_________________________________________________________________\n",
"conv2d_26 (Conv2D) (None, 24, 24, 32) 832 \n",
"_________________________________________________________________\n",
"conv2d_27 (Conv2D) (None, 20, 20, 16) 12816 \n",
"_________________________________________________________________\n",
"max_pooling2d_2 (MaxPooling2 (None, 10, 10, 16) 0 \n",
"_________________________________________________________________\n",
"conv2d_28 (Conv2D) (None, 8, 8, 8) 1160 \n",
"_________________________________________________________________\n",
"flatten_2 (Flatten) (None, 512) 0 \n",
"_________________________________________________________________\n",
"dropout_6 (Dropout) (None, 512) 0 \n",
"_________________________________________________________________\n",
"dense_18 (Dense) (None, 32) 16416 \n",
"_________________________________________________________________\n",
"dense_19 (Dense) (None, 16) 528 \n",
"_________________________________________________________________\n",
"dense_20 (Dense) (None, 10) 170 \n",
"=================================================================\n",
"Total params: 31,922\n",
"Trainable params: 31,922\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "PWRo9-yY0ify"
},
"source": [
"tl_model.compile(optimizer='adam',\n",
" loss='categorical_crossentropy',\n",
" metrics=['accuracy'])"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "-b4EtlJ60l99",
"outputId": "d74b5b9e-ea66-466e-9fe3-37a3811ea89a"
},
"source": [
"t=time.time()\n",
"tl_model.fit(x_train,y_train,\n",
" validation_data=(x_test,y_test),\n",
" batch_size=64,\n",
" epochs=10,\n",
" verbose=1,\n",
" shuffle=True)\n",
"print('Training time: %s' % (time.time()-t))"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Epoch 1/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 1.1453 - accuracy: 0.6350 - val_loss: 0.3872 - val_accuracy: 0.8607\n",
"Epoch 2/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.4153 - accuracy: 0.8511 - val_loss: 0.3400 - val_accuracy: 0.8747\n",
"Epoch 3/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3702 - accuracy: 0.8645 - val_loss: 0.3392 - val_accuracy: 0.8758\n",
"Epoch 4/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3482 - accuracy: 0.8724 - val_loss: 0.3222 - val_accuracy: 0.8863\n",
"Epoch 5/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3404 - accuracy: 0.8754 - val_loss: 0.3063 - val_accuracy: 0.8868\n",
"Epoch 6/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3277 - accuracy: 0.8797 - val_loss: 0.3038 - val_accuracy: 0.8898\n",
"Epoch 7/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3183 - accuracy: 0.8810 - val_loss: 0.3040 - val_accuracy: 0.8903\n",
"Epoch 8/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3070 - accuracy: 0.8875 - val_loss: 0.2936 - val_accuracy: 0.8966\n",
"Epoch 9/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3020 - accuracy: 0.8865 - val_loss: 0.2946 - val_accuracy: 0.8913\n",
"Epoch 10/10\n",
"938/938 [==============================] - 4s 4ms/step - loss: 0.3030 - accuracy: 0.8873 - val_loss: 0.2930 - val_accuracy: 0.8945\n",
"Training time: 38.041860818862915\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "88AKEKBO0rld",
"outputId": "2a56be76-1c87-4ca1-ad28-3792f10921ae"
},
"source": [
"# Evaluate the model on test set\n",
"score = tl_model.evaluate(x_test, y_test, verbose=0)\n",
"\n",
"# Print test accuracy\n",
"print('\\n', 'Test accuracy:', score[1])"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"\n",
" Test accuracy: 0.8945000171661377\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UaHLKDLas_dF"
},
"source": [
"### Task 1.5 Performance comparison\n",
"\n",
"*(weight ~3%)*\n",
"\n",
"How many parameters are trainable in each of the two settings (with and without fine-tuning)? How does the difference impact the training time?\n",
"\n",
"Trainable parameters with fine-tuning are 31,922 and without fine-tuning are 17,114. Training time of without fine-tuning is lower than with fine-tuning.\n",
"\n",
"\n",
"Which setting achieved higher accuracy? Why did it work better for this problem? \n",
"\n",
"The model with fine-tuning achieve higher accuracy because it has more trainable parameters.\n",
"\n",
"Have we benefitted from using the pretrained model?\n",
"\n",
"Yes we get benefit from using pretrained model.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ouK5NY-_pLDK"
},
"source": [
"## Task 2 Fast training of deep networks\n",
"\n",
"*(weight ~20%)*"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "LgoOE2W1pdfN"
},
"source": [
"### Task 2.1 Train a highly accurate network for CIFAR10\n",
"\n",
"*(weight ~7%)*\n",
"\n",
"In this task, you will train deep neural networks on the [CIFAR10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html). Compared with the datasets that you have worked on so far, CIFAR10 represents a relatively larger multi-class classification problem and presents a great opportunity for you to solve a \"harder\" problem.\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "bLOq2VMZlTy9"
},
"source": [
"from keras.datasets import cifar10"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Lm2ONncZJdGP",
"outputId": "1ab0cbb2-b7c4-4979-8121-a328ba9fc0f4"
},
"source": [
"# load dataset\n",
"(trainX, trainY), (testX, testY) = cifar10.load_data()"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n",
"170500096/170498071 [==============================] - 3s 0us/step\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "wLwLV7XcJchP"
},
"source": [
"# one hot encode target values\n",
"trainY = to_categorical(trainY)\n",
"testY = to_categorical(testY)"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "ygMb80NKJb8s"
},
"source": [
"# Change from matrix to array of dimension 28x28 to array of dimension 784\n",
"dim_data = np.prod(trainX.shape[1:])\n",
"trainX = trainX.reshape(trainX.shape[0], dim_data)\n",
"testX = testX.reshape(testX.shape[0], dim_data)\n",
"\n",
"# convert from integers to floats\n",
"trainX = trainX.astype('float32')\n",
"testX = testX.astype('float32')\n",
"# normalize to range 0-1\n",
"trainX = trainX / 255.0\n",
"testX = testX / 255.0"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "o5kM_HQtl_I3",
"outputId": "9f950a1a-64ba-4bfb-b1c2-3e81511279d8"
},
"source": [
"model = Sequential()\n",
"model.add(Dense(512, activation='relu', input_shape=(dim_data,)))\n",
"model.add(Dense(512, activation='relu'))\n",
"model.add(Dense(512, activation='relu'))\n",
"model.add(Dense(512, activation='relu'))\n",
"model.add(Dense(10, activation='softmax'))\n",
"model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
"model.summary()"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Model: \"sequential_10\"\n",
"_________________________________________________________________\n",
"Layer (type) Output Shape Param # \n",
"=================================================================\n",
"dense_40 (Dense) (None, 512) 1573376 \n",
"_________________________________________________________________\n",
"dense_41 (Dense) (None, 512) 262656 \n",
"_________________________________________________________________\n",
"dense_42 (Dense) (None, 512) 262656 \n",
"_________________________________________________________________\n",
"dense_43 (Dense) (None, 512) 262656 \n",
"_________________________________________________________________\n",
"dense_44 (Dense) (None, 10) 5130 \n",
"=================================================================\n",
"Total params: 2,366,474\n",
"Trainable params: 2,366,474\n",
"Non-trainable params: 0\n",
"_________________________________________________________________\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "FT-XsHaWm_6k",
"outputId": "b1c5d658-604d-4c34-fa02-e60ef9780850"
},
"source": [
"# fit model\n",
"history = model.fit(trainX, trainY, epochs=20, batch_size=64, validation_data=(testX, testY), verbose=1)"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Epoch 1/20\n",
"782/782 [==============================] - 6s 7ms/step - loss: 1.9893 - accuracy: 0.2676 - val_loss: 1.7236 - val_accuracy: 0.3820\n",
"Epoch 2/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.7050 - accuracy: 0.3839 - val_loss: 1.6477 - val_accuracy: 0.4011\n",
"Epoch 3/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.6111 - accuracy: 0.4213 - val_loss: 1.6428 - val_accuracy: 0.4308\n",
"Epoch 4/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.5522 - accuracy: 0.4405 - val_loss: 1.5236 - val_accuracy: 0.4513\n",
"Epoch 5/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.4899 - accuracy: 0.4652 - val_loss: 1.4685 - val_accuracy: 0.4703\n",
"Epoch 6/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.4477 - accuracy: 0.4808 - val_loss: 1.5340 - val_accuracy: 0.4524\n",
"Epoch 7/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.4119 - accuracy: 0.4913 - val_loss: 1.4524 - val_accuracy: 0.4851\n",
"Epoch 8/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.3672 - accuracy: 0.5078 - val_loss: 1.4704 - val_accuracy: 0.4840\n",
"Epoch 9/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.3458 - accuracy: 0.5142 - val_loss: 1.4407 - val_accuracy: 0.4893\n",
"Epoch 10/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.3070 - accuracy: 0.5289 - val_loss: 1.4289 - val_accuracy: 0.4941\n",
"Epoch 11/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.2519 - accuracy: 0.5489 - val_loss: 1.4540 - val_accuracy: 0.4904\n",
"Epoch 12/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.2147 - accuracy: 0.5628 - val_loss: 1.4768 - val_accuracy: 0.4880\n",
"Epoch 13/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.1784 - accuracy: 0.5727 - val_loss: 1.4462 - val_accuracy: 0.5043\n",
"Epoch 14/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.1250 - accuracy: 0.5888 - val_loss: 1.4955 - val_accuracy: 0.4980\n",
"Epoch 15/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.1065 - accuracy: 0.5966 - val_loss: 1.5338 - val_accuracy: 0.5006\n",
"Epoch 16/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 1.0408 - accuracy: 0.6199 - val_loss: 1.5086 - val_accuracy: 0.4942\n",
"Epoch 17/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 0.9999 - accuracy: 0.6356 - val_loss: 1.5769 - val_accuracy: 0.4902\n",
"Epoch 18/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 0.9620 - accuracy: 0.6488 - val_loss: 1.5995 - val_accuracy: 0.4892\n",
"Epoch 19/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 0.9130 - accuracy: 0.6671 - val_loss: 1.6495 - val_accuracy: 0.4987\n",
"Epoch 20/20\n",
"782/782 [==============================] - 5s 7ms/step - loss: 0.8689 - accuracy: 0.6811 - val_loss: 1.6990 - val_accuracy: 0.4933\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "IaD5oqj3lhuI"
},
"source": [
"#### Task 2.1.1 Document the hardware used\n",
"\n",
"Before you start, write down your hardware specifications, including \n",
"\n",
"- the GPU model, the number of GPUs, and the GPU memory\n",
"- the CPU model, the number of CPUs, and the CPU clock speed\n",
"\n",
"(Hint: you may find commands like `nvidia-smi`, `lscpu` or `psutil` useful.)"
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "k3QLHw1jJu0M",
"outputId": "fb458b47-afd4-4f60-85c7-f68409ec4e6d"
},
"source": [
"print('GPU Model:')\n",
"!nvidia-smi -L"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"GPU Model:\n",
"GPU 0: Tesla T4 (UUID: GPU-e38ef71c-f5a9-df29-4789-0b2db54d7fbf)\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "lsDfjOkXJuWW",
"outputId": "6f6e8bfb-e220-4deb-c15b-6a71aa88fd7f"
},
"source": [
"print('Number of GPUs:')\n",
"!nvidia-smi --query-gpu=name --format=csv,noheader | wc -l"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Number of GPUs:\n",
"1\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "k095hmMPJt2W",
"outputId": "92920db2-96ec-4a21-fade-39453b79b0b7"
},
"source": [
"print('GPU Memory:')\n",
"!nvidia-smi"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"GPU Memory:\n",
"Fri May 14 14:33:41 2021 \n",
"+-----------------------------------------------------------------------------+\n",
"| NVIDIA-SMI 465.19.01 Driver Version: 460.32.03 CUDA Version: 11.2 |\n",
"|-------------------------------+----------------------+----------------------+\n",
"| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n",
"| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n",
"| | | MIG M. |\n",
"|===============================+======================+======================|\n",
"| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |\n",
"| N/A 37C P8 9W / 70W | 0MiB / 15109MiB | 0% Default |\n",
"| | | N/A |\n",
"+-------------------------------+----------------------+----------------------+\n",
" \n",
"+-----------------------------------------------------------------------------+\n",
"| Processes: |\n",
"| GPU GI CI PID Type Process name GPU Memory |\n",
"| ID ID Usage |\n",
"|=============================================================================|\n",
"| No running processes found |\n",
"+-----------------------------------------------------------------------------+\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "qFJM1BdV9IIv",
"outputId": "740d4d7b-c9d8-4053-f32b-4bfe7c285d05"
},
"source": [
"print('CPU Model:')\n",
"!lscpu |grep 'Model name'"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"CPU Model:\n",
"Model name: Intel(R) Xeon(R) CPU @ 2.20GHz\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "KmKtGSeP9QUg",
"outputId": "81c16b6e-7e75-43b6-bdd2-e365e73f2f12"
},
"source": [
"import psutil\n",
"\n",
"print('Number of CPUs:',psutil.cpu_count())"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Number of CPUs: 2\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "HYIdvwEdT1DO",
"outputId": "f77347a4-3f9d-4ee0-bdad-7a106799cc0c"
},
"source": [
"print('CPU Clock Speed:',psutil.cpu_freq())"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"CPU Clock Speed: None\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "adN9Tq-6lyG-"
},
"source": [
"#### Task 2.1.2 Train a \"shallow\" ConvNet\n",
"\n",
"Build a ConvNet with fewer than 10 layers. Train the network until it converges. You will use this network as a baseline for the later experiments. \n",
"\n",
"- Plot the training and validation history. \n",
"- Report the testing accuracy. "
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 302
},
"id": "_pNTozhXX_fa",
"outputId": "331710a5-f6f7-42eb-fbc6-815e4e4021c9"
},
"source": [
"# example of loading the cifar10 dataset\n",
"from matplotlib import pyplot\n",
"from keras.datasets import cifar10\n",
"# load dataset\n",
"(trainX, trainy), (testX, testy) = cifar10.load_data()\n",
"# summarize loaded dataset\n",
"print('Train: X=%s, y=%s' % (trainX.shape, trainy.shape))\n",
"print('Test: X=%s, y=%s' % (testX.shape, testy.shape))\n",
"# plot first few images\n",
"for i in range(9):\n",
"\t# define subplot\n",
"\tpyplot.subplot(330 + 1 + i)\n",
"\t# plot raw pixel data\n",
"\tpyplot.imshow(trainX[i])\n",
"# show the figure\n",
"pyplot.show()"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"Train: X=(50000, 32, 32, 3), y=(50000, 1)\n",
"Test: X=(10000, 32, 32, 3), y=(10000, 1)\n"
],
"name": "stdout"
},
{
"output_type": "display_data",
"data": {
"image/png":...