Objective Investigate and assess Data Engineering approaches to integrate handwriting recognition datasets Instructions Complete this assignment individually Use an accepted style (e.g., APA, Chicago,...

Objective

Investigate and assess Data Engineering approaches to integrate handwriting recognition datasets

Instructions

• Complete this assignment individually


• Use an accepted style (e.g., APA, Chicago, etc.) for citing any materials


• Submit a PDF file containing your submission for this assignment


• Read the Notes section below for additional information


• Find 2 additional datasets to augment the MNIST dataset used in Assignments 2-4


• For each of the 3 datasets (MNIST and the 2 from the previous step), document the following:
  
  
  • The source for the data
  
  
  • The frequency (if any) with which the dataset is updated
  
  
  • Any metadata associated with the dataset
  
  
  
  


• Design a unified model to consolidates the datasets into one form


• Map the datasets to the aforementioned unified model


• Identify processes for extracting the datasets from their respective sources


• Evaluate the need for a data warehouse and/or a data lake to serve as the data repository for the datasets


• Evaluate one of the following technologies/services to determine if it fits our needs
  
  
  • Snowflake
  
  
  • Amazon Web Services
  
  
  • Microsoft Azure
  
  
  • Google Cloud
  
  
  
  


• Include any related literature to support your responses

Notes

As indicated in previous course materials, the MNIST dataset is a mature and often used dataset for automated handwriting recognition. More recently, other datasets have been released that aim to improve on MNIST.

Data Engineering Assignment (Assignment 5) Objective Investigate and assess Data Engineering approaches to integrate handwriting recognition datasets Instructions · Complete this assignment individually · Use an accepted style (e.g., APA, Chicago, etc.) for citing any materials · Submit a PDF file containing your submission for this assignment · Read the Notes section below for additional information · Find 2 additional datasets to augment the MNIST dataset used in Assignments 2-4 · For each of the 3 datasets (MNIST and the 2 from the previous step), document the following: · The source for the data · The frequency (if any) with which the dataset is updated · Any metadata associated with the dataset · Design a unified model to consolidates the datasets into one form · Map the datasets to the aforementioned unified model · Identify processes for extracting the datasets from their respective sources · Evaluate the need for a data warehouse and/or a data lake to serve as the data repository for the datasets · Evaluate one of the following technologies/services to determine if it fits our needs · Snowflake · Amazon Web Services · Microsoft Azure · Google Cloud · Include any related literature to support your responses Notes As indicated in previous course materials, the MNIST dataset is a mature and often used dataset for automated handwriting recognition. More recently, other datasets have been released that aim to improve on MNIST. Rubric { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### GPU benchmark on MNIST" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Using TensorFlow backend.\n" ] } ], "source": [ "import tensorflow as tf\n", "import numpy as np \n", "import matplotlib.pyplot as plt\n", "import keras as k\n", "from tensorflow.examples.tutorials.mnist import input_data\n", "from keras.datasets import mnist\n", "from keras.models import Sequential\n", "from keras.layers import Dense, Dropout, Flatten\n", "from keras.layers import Conv2D, MaxPooling2D, BatchNormalization\n", "from keras.optimizers import SGD, Adam\n", "from keras.models import load_model\n", "from keras import backend as K\n", "\n", "# data preprocessing\n", "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n", "img_rows, img_cols = 28,28\n", "x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n", "x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n", "input_shape = (img_rows, img_cols, 1)\n", "x_test=x_test.astype('float32')\n", "x_train=x_train.astype('float32')\n", "mean=np.mean(x_train)\n", "std=np.std(x_train)\n", "x_test = (x_test-mean)/std\n", "x_train = (x_train-mean)/std\n", "\n", "# labels\n", "num_classes=10\n", "y_train = k.utils.to_categorical(y_train, num_classes)\n", "y_test = k.utils.to_categorical(y_test, num_classes)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "number of filters 1\n", "batch size 8\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 89s 1ms/step - loss: 0.6453 - acc: 0.8034 - val_loss: 0.1984 - val_acc: 0.9360\n", "number of filters 1\n", "batch size 16\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 43s 709us/step - loss: 0.3025 - acc: 0.9064 - val_loss: 0.2010 - val_acc: 0.9379\n", "number of filters 1\n", "batch size 32\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 21s 354us/step - loss: 0.2247 - acc: 0.9294 - val_loss: 0.1230 - val_acc: 0.9600\n", "number of filters 1\n", "batch size 64\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 11s 178us/step - loss: 0.1830 - acc: 0.9418 - val_loss: 0.1164 - val_acc: 0.9614\n", "number of filters 1\n", "batch size 128\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 5s 90us/step - loss: 0.1615 - acc: 0.9488 - val_loss: 0.1066 - val_acc: 0.9652\n", "number of filters 1\n", "batch size 256\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 3s 46us/step - loss: 0.1496 - acc: 0.9527 - val_loss: 0.1051 - val_acc: 0.9657\n", "number of filters 1\n", "batch size 512\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 2s 25us/step - loss: 0.1416 - acc: 0.9560 - val_loss: 0.1041 - val_acc: 0.9656\n", "number of filters 1\n", "batch size 1024\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 1s 18us/step - loss: 0.1392 - acc: 0.9554 - val_loss: 0.0998 - val_acc: 0.9674\n", "number of filters 2\n", "batch size 8\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 86s 1ms/step - loss: 0.4295 - acc: 0.8730 - val_loss: 0.1104 - val_acc: 0.9636\n", "number of filters 2\n", "batch size 16\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 43s 714us/step - loss: 0.1736 - acc: 0.9469 - val_loss: 0.0883 - val_acc: 0.9717\n", "number of filters 2\n", "batch size 32\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 21s 356us/step - loss: 0.1213 - acc: 0.9630 - val_loss: 0.0655 - val_acc: 0.9798\n", "number of filters 2\n", "batch size 64\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 11s 179us/step - loss: 0.0877 - acc: 0.9722 - val_loss: 0.0679 - val_acc: 0.9770\n", "number of filters 2\n", "batch size 128\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 6s 92us/step - loss: 0.0736 - acc: 0.9770 - val_loss: 0.0573 - val_acc: 0.9822\n", "number of filters 2\n", "batch size 256\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 3s 48us/step - loss: 0.0668 - acc: 0.9786 - val_loss: 0.0520 - val_acc: 0.9844\n", "number of filters 2\n", "batch size 512\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 2s 27us/step - loss: 0.0607 - acc: 0.9806 - val_loss: 0.0524 - val_acc: 0.9842\n", "number of filters 2\n", "batch size 1024\n", "Train on 60000 samples, validate on 10000 samples\n", "Epoch 1/1\n", "60000/60000 [==============================] - 1s 22us/step - loss: 0.0591 - acc: 0.9813 - val_loss: 0.0524 - val_acc: 0.9834\n", "number of filters 4\n", "batch size 8\n", "Train on 60000 samples, validate on 10000 samples\n",