diff --git a/tutorial_notebook.ipynb b/tutorial_notebook.ipynb new file mode 100644 index 00000000..79f7eff9 --- /dev/null +++ b/tutorial_notebook.ipynb @@ -0,0 +1,407 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Tensorflow, Word embedding, and Text Analytics" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This notebook is about practical machine learning knowledge and skills in tensorflow, word embedding and text analytics. Some sections have been partially completed to help you get\n", + "started. \n", + "\n", + "* Before you start, read the entire notebook carefully once to understand what you need to do.

\n", + "\n", + "* For each cell marked with **#YOU ARE REQUIRED TO INSERT YOUR CODES IN THIS CELL**, there will be places where you **must** supply your own codes when instructed.
" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Instruction\n", + "\n", + "This tutorial contains **two** parts \n", + "\n", + "* Part 1: Deep Feedforward Neural Network\n", + "* Part 2: Word2Vec, text analytics and application\n", + "\n", + "**Hint**: You are strongly recommended to go through these lectures and practical lab sessions covered from Week 5 to 9 thoroughly." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Part 1: Deep Feedforward Neural Network \n", + "\n", + "Demonstrate the knowledge in deep learning that you have acquired from the lectures. Most of the content are drawn from the materials in week 5, 6 and 7 for deep neural networks. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "*Run the following cell to create necessary subfolders. You must **not** modify these codes and **must** run it first*." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "# Create necessary subfolders to store immediate files.\n", + "\n", + "import os\n", + "if not os.path.exists(\"./models/dnn0\"):\n", + " os.makedirs(\"models/dnn0\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The first part is to apply DNN to recognize letters from A-Z. You have played with MNIST dataset in your pracs and this should have given a good sense of how to apply DNN on images for recognition task. \n", + "\n", + "You are going to work with the **notMNIST** dataset for *letter recognition task*. The dataset contains 10 classes of letters A-J taken from different fonts. You will see some examples at the visualization task in the next part. A short blog about the data can be found [here](http://yaroslavvb.blogspot.com.au/2011/09/notmnist-dataset.html).\n", + "\n", + "Here we only consider a small subset which can be found at [this link](http://yaroslavvb.com/upload/notMNIST/notMNIST_small.mat). This file has been already downloaded and stored in subfolder `datasets` of this folder. The file is in *Matlab* format, thus our first task is to:" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### **Question 1.1**. Load the data into *`numpy array`* format of two variables:\n", + "* *`x`*: storing features with dimension `[num_samples, width, height]` (`num_samples`: number of samples, `width`: image width, `height`: image height), and\n", + "* *`y`*: storing labels with dimension `num_samples`. \n", + "\n", + "Enter the missing codes in the following cell to complete this question." + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "(18724, 28, 28)\n" + ] + } + ], + "source": [ + "# YOU ARE REQUIRED TO INSERT YOUR CODES IN THIS CELL\n", + "\n", + "import numpy as np\n", + "import scipy.io as sio\n", + "data = sio.matlab.loadmat(\"datasets/notMNIST_small.mat\")\n", + "x, y = data['images'] , data['labels']\n", + "x = np.rollaxis(x, axis=2)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### **Question 1.2**. Print out the total number of data points, and the *unique* labels in this dataset.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Total data points are : 18724\n", + "Unique Labels: 10\n" + ] + } + ], + "source": [ + "#YOU ARE REQUIRED TO INSERT YOUR CODES IN THIS CELL\n", + "print('Total data points are :',len(y))\n", + "print('Unique Labels:',len(np.unique(y)))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### **Question 1.3**. Display 100 images in the form of `10x10` matrix, each row showing 10 *random* images of a label. \n", + "\n", + "#### You might decide to use the function `display_images` provided at the beginning of this tutorial, or you can write your own codes.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "metadata": {}, + "outputs": [], + "source": [ + "# this function is a utility to display images from the dataset\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "\n", + "def display_images(images, shape):\n", + " fig = plt.figure(figsize=shape)\n", + " fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.05, wspace=0.05)\n", + " for i in range(np.prod(shape)):\n", + " p = fig.add_subplot(shape[0], shape[1], i+1, xticks=[], yticks=[])\n", + " p.imshow(images[i], cmap=plt.cm.bone) " + ] + }, + { + "cell_type": "code", + "execution_count": 52, + "metadata": {}, + "outputs": [ + { + "ename": "ValueError", + "evalue": "operands could not be broadcast together with shapes (0,) (28,28) ", + "output_type": "error", + "traceback": [ + "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[1;31mValueError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m\u001b[0m in \u001b[0;36m\u001b[1;34m()\u001b[0m\n\u001b[0;32m 7\u001b[0m \u001b[0midx\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0midx\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mnp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrandom\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpermutation\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0midx\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 8\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[1;32min\u001b[0m \u001b[0midx\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 9\u001b[1;33m \u001b[0mimages\u001b[0m \u001b[1;33m+=\u001b[0m \u001b[0mx\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mi\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 10\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 11\u001b[0m \u001b[0mdisplay_images\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mimages\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mshape\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m10\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", + "\u001b[1;31mValueError\u001b[0m: operands could not be broadcast together with shapes (0,) (28,28) " + ] + } + ], + "source": [ + "# YOU ARE REQUIRED TO INSERT YOUR CODES IN THIS CELL\n", + "\n", + "unique_labels = np.unique(y)\n", + "images = []\n", + "for l in unique_labels: \n", + " idx = np.where(y == l)[0]\n", + " idx = idx[np.random.permutation(len(idx))[:10]] \n", + " for i in idx:\n", + " images += # INSERT YOUR CODE HERE\n", + "\n", + "display_images(images, shape=(10, 10))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### **Question 1.4**. Use the *deep feedforward neural network* as the classifier to perform images classification task in a *single split training and testing*." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# YOU ARE REQUIRED TO INSERT YOUR CODES IN THIS CELL\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In training the DNN, scaling data is important. The pixel intensities of images are in the range of [0, 255], which makes the neural network difficult to learn." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true, + "scrolled": true + }, + "outputs": [], + "source": [ + "# YOU ARE REQUIRED TO INSERT YOUR CODES IN THIS CELL\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# YOU ARE REQUIRED TO INSERT YOUR CODES IN THIS CELL\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# YOU ARE REQUIRED TO INSERT YOUR CODES IN THIS CELL [5 marks]\n", + "\n", + "import tensorflow as tf\n", + "\n", + "tf.reset_default_graph()\n", + "\n", + "num_inputs = # INSERT YOUR CODE HERE\n", + "num_hidden1 = # INSERT YOUR CODE HERE\n", + "num_hidden2 = # INSERT YOUR CODE HERE\n", + "num_outputs = len(np.unique(y))\n", + "\n", + "inputs = # INSERT YOUR CODE HERE\n", + "labels = # INSERT YOUR CODE HERE" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# YOU ARE REQUIRED TO INSERT YOUR CODES IN THIS CELL [3 marks]\n", + "\n", + "def neuron_layer(x, num_neurons, name, activation=None):\n", + " with tf.name_scope(name):\n", + " # INSERT YOUR CODE HERE\n", + " if activation == \"sigmoid\":\n", + " # INSERT YOUR CODE HERE\n", + " elif activation == \"relu\":\n", + " # INSERT YOUR CODE HERE\n", + " else:\n", + " return z" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# YOU ARE REQUIRED TO INSERT YOUR CODES IN THIS CELL [7 marks]\n", + "\n", + "with tf.name_scope(\"dnn\"):\n", + " hidden1 = # INSERT YOUR CODE HERE\n", + " hidden2 = # INSERT YOUR CODE HERE\n", + " logits = # INSERT YOUR CODE HERE\n", + "with tf.name_scope(\"loss\"):\n", + " xentropy = # INSERT YOUR CODE HERE\n", + " loss = tf.reduce_mean(xentropy, name=\"loss\")\n", + " \n", + "with tf.name_scope(\"evaluation\"):\n", + " # INSERT YOUR CODE HERE\n", + " \n", + "with tf.name_scope(\"train\"):\n", + " # INSERT YOUR CODE HERE\n", + " \n", + " for var in tf.trainable_variables():\n", + " tf.summary.histogram(var.op.name + \"/values\", var)\n", + " \n", + " for grad, var in grads:\n", + " if grad is not None:\n", + " tf.summary.histogram(var.op.name + \"/gradients\", grad)\n", + "\n", + "# summary\n", + "accuracy_summary = # INSERT YOUR CODE HERE\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# YOU ARE REQUIRED TO INSERT YOUR CODES IN THIS CELL [5 marks]\n", + "\n", + "# merge all summary\n", + "tf.summary.histogram('hidden1/activations', hidden1)\n", + "tf.summary.histogram('hidden2/activations', hidden2)\n", + "merged = # INSERT YOUR CODE HERE\n", + "\n", + "init = # INSERT YOUR CODE HERE\n", + "saver = # INSERT YOUR CODE HERE\n", + "\n", + "train_writer = tf.summary.FileWriter(\"models/dnn0/train\", tf.get_default_graph())\n", + "test_writer = tf.summary.FileWriter(\"models/dnn0/test\", tf.get_default_graph())\n", + "\n", + "num_epochs = # INSERT YOUR CODE HERE\n", + "batch_size = # INSERT YOUR CODE HERE" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**(d)** **You are now required write code to train the DNN.** Write codes in the following cell. **[5 points]** " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# YOU ARE REQUIRED TO INSERT YOUR CODES IN THIS CELL\n", + "\n", + "with tf.Session() as sess:\n", + " init.run()\n", + " print(\"Epoch\\tTrain accuracy\\tTest accuracy\")\n", + " for epoch in range(num_epochs):\n", + " for idx_start in range(0, x_train.shape[0], batch_size):\n", + " idx_end = # INSERT YOUR CODE HERE\n", + " x_batch, y_batch = # INSERT YOUR CODE HERE\n", + " sess.run(training_op, feed_dict={inputs: x_batch, labels: y_batch})\n", + " \n", + " summary_train, acc_train = # INSERT YOUR CODE HERE\n", + " summary_test, acc_test = # INSERT YOUR CODE HERE\n", + " \n", + " train_writer.add_summary(summary_train, epoch)\n", + " test_writer.add_summary(summary_test, epoch)\n", + " \n", + " print(\"{}\\t{}\\t{}\".format(epoch, acc_train, acc_test))\n", + "\n", + " save_path = saver.save(sess, \"models/dnn0.ckpt\")" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.6.4" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +}