{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "\n",
    "drive.mount('/content/drive', force_remount=True)\n",
    "\n",
    "# 输入daseCV所在的路径\n",
    "# 'daseCV' 文件夹包括 '.py', 'classifiers' 和'datasets'文件夹\n",
    "# 例如 'CV/assignments/assignment1/daseCV/'\n",
    "FOLDERNAME = None\n",
    "\n",
    "assert FOLDERNAME is not None, \"[!] Enter the foldername.\"\n",
    "\n",
    "%cd drive/My\\ Drive\n",
    "%cp -r $FOLDERNAME ../../\n",
    "%cd ../../\n",
    "%cd daseCV/datasets/\n",
    "!bash get_datasets.sh\n",
    "%cd ../../"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# 实现一个神经网络\n",
    "\n",
    "在这个练习中，我们将开发一个具有全连接层的神经网络来进行分类任务，并在CIFAR-10数据集上进行测试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "# 一些初始化设置\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from daseCV.classifiers.neural_net import TwoLayerNet\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # 设置默认绘图大小\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# 自动重载外部模块的详细资料可以查看下面链接\n",
    "# http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "    \"\"\" returns relative error \"\"\"\n",
    "    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "source": [
    "在文件`daseCV/classifiers/neural_net`中使用一个类`TwoLayerNet`表示我们的网络实例。网络参数存储在实例变量`self.params`中, 其中键是参数名，值是numpy数组。\n",
    "下面，我们初始化玩具数据和一个玩具模型，我们将使用它来开发具体代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "# 创建一个小网络和一些玩具数据\n",
    "# 注意，我们设置了可重复实验的随机种子。\n",
    "\n",
    "input_size = 4\n",
    "hidden_size = 10\n",
    "num_classes = 3\n",
    "num_inputs = 5\n",
    "\n",
    "def init_toy_model():\n",
    "    np.random.seed(0)\n",
    "    return TwoLayerNet(input_size, hidden_size, num_classes, std=1e-1)\n",
    "\n",
    "def init_toy_data():\n",
    "    np.random.seed(1)\n",
    "    X = 10 * np.random.randn(num_inputs, input_size)\n",
    "    y = np.array([0, 1, 2, 2, 1])\n",
    "    return X, y\n",
    "\n",
    "net = init_toy_model()\n",
    "X, y = init_toy_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 前向传播：计算scores\n",
    "\n",
    "打开文件`daseCV/classifiers/neural_net`，查看`TwoLayerNet.loss`函数。这个函数与你之前在SVM和Softmax写的损失函数非常相似：输入数据和权重，计算类别的scores、loss和参数上的梯度。\n",
    "\n",
    "实现前向传播的第一部分：使用权重和偏差来计算所有输入的scores。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = net.loss(X)\n",
    "print('Your scores:')\n",
    "print(scores)\n",
    "print()\n",
    "print('correct scores:')\n",
    "correct_scores = np.asarray([\n",
    "  [-0.81233741, -1.27654624, -0.70335995],\n",
    "  [-0.17129677, -1.18803311, -0.47310444],\n",
    "  [-0.51590475, -1.01354314, -0.8504215 ],\n",
    "  [-0.15419291, -0.48629638, -0.52901952],\n",
    "  [-0.00618733, -0.12435261, -0.15226949]])\n",
    "print(correct_scores)\n",
    "print()\n",
    "\n",
    "# The difference should be very small. We get < 1e-7\n",
    "print('Difference between your scores and correct scores:')\n",
    "print(np.sum(np.abs(scores - correct_scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 反向传播: 计算损失\n",
    "\n",
    "在同一个函数中，编码实现第二个部分，计算损失值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss, _ = net.loss(X, y, reg=0.05) #reg为0.1\n",
    "correct_loss = 1.30378789133\n",
    "\n",
    "# should be very small, we get < 1e-12\n",
    "print('Difference between your loss and correct loss:')\n",
    "print(np.sum(np.abs(loss - correct_loss)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 反向传播\n",
    "\n",
    "实现函数的其余部分。计算关于变量`W1`, `b1`, `W2`, `b2`的梯度。当你正确实现了前向传播的代码后(hopefully!)，你可以用数值梯度检查debug你的反向传播:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from daseCV.gradient_check import eval_numerical_gradient\n",
    "\n",
    "# 使用数值梯度检查反向传播的代码。\n",
    "# 如果你的代码是正确的，那么对于W1、W2、b1和b2，\n",
    "# 数值梯度和解析梯度之间的差异应该小于1e-8。\n",
    "\n",
    "loss, grads = net.loss(X, y, reg=0.05)\n",
    "\n",
    "# these should all be less than 1e-8 or so\n",
    "for param_name in grads:\n",
    "    f = lambda W: net.loss(X, y, reg=0.05)[0]\n",
    "    param_grad_num = eval_numerical_gradient(f, net.params[param_name], verbose=False)\n",
    "    print('%s max relative error: %e' % (param_name, rel_error(param_grad_num, grads[param_name])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 训练网络\n",
    "\n",
    "我们使用随机梯度下降(SGD)训练网络，类似于SVM和Softmax。查看`TwoLayerNet.train`函数并填写训练代码中缺失的部分。这与SVM和Softmax分类器的训练过程非常相似。您还必须实现`TwoLayerNet.predict`，即在网络训练过程中周期性地进行预测，以持续追踪网络的准确率\n",
    "\n",
    "当你完成了这个函数吼，运行下面的代码，在玩具数据上训练一个两层网络。你的训练损失应该少于0.02。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = init_toy_model()\n",
    "stats = net.train(X, y, X, y,\n",
    "            learning_rate=1e-1, reg=5e-6,\n",
    "            num_iters=100, verbose=False)\n",
    "\n",
    "print('Final training loss: ', stats['loss_history'][-1])\n",
    "\n",
    "# plot the loss history\n",
    "plt.plot(stats['loss_history'])\n",
    "plt.xlabel('iteration')\n",
    "plt.ylabel('training loss')\n",
    "plt.title('Training Loss history')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 加载数据\n",
    "\n",
    "现在你已经实现了一个两层的神经网络，通过了梯度检查,并且在玩具数据有效工作，现在可以加载我们喜欢的CIFAR-10数据了(我不喜欢（╯‵□′）╯︵┴─┴ )，这样就可以训练真实数据集上的分类器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "from daseCV.data_utils import load_CIFAR10\n",
    "\n",
    "def get_CIFAR10_data(num_training=49000, num_validation=1000, num_test=1000):\n",
    "    \"\"\"\n",
    "    Load the CIFAR-10 dataset from disk and perform preprocessing to prepare\n",
    "    it for the two-layer neural net classifier. These are the same steps as\n",
    "    we used for the SVM, but condensed to a single function.  \n",
    "    \"\"\"\n",
    "    # Load the raw CIFAR-10 data\n",
    "    cifar10_dir = 'daseCV/datasets/cifar-10-batches-py'\n",
    "    \n",
    "    # 清除变量,防止多次加载数据(这可能会导致内存问题)\n",
    "    try:\n",
    "       del X_train, y_train\n",
    "       del X_test, y_test\n",
    "       print('Clear previously loaded data.')\n",
    "    except:\n",
    "       pass\n",
    "\n",
    "    X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)\n",
    "        \n",
    "    # Subsample the data\n",
    "    mask = list(range(num_training, num_training + num_validation))\n",
    "    X_val = X_train[mask]\n",
    "    y_val = y_train[mask]\n",
    "    mask = list(range(num_training))\n",
    "    X_train = X_train[mask]\n",
    "    y_train = y_train[mask]\n",
    "    mask = list(range(num_test))\n",
    "    X_test = X_test[mask]\n",
    "    y_test = y_test[mask]\n",
    "\n",
    "    # Normalize the data: subtract the mean image\n",
    "    mean_image = np.mean(X_train, axis=0)\n",
    "    X_train -= mean_image\n",
    "    X_val -= mean_image\n",
    "    X_test -= mean_image\n",
    "\n",
    "    # Reshape data to rows\n",
    "    X_train = X_train.reshape(num_training, -1)\n",
    "    X_val = X_val.reshape(num_validation, -1)\n",
    "    X_test = X_test.reshape(num_test, -1)\n",
    "\n",
    "    return X_train, y_train, X_val, y_val, X_test, y_test\n",
    "\n",
    "\n",
    "# Invoke the above function to get our data.\n",
    "X_train, y_train, X_val, y_val, X_test, y_test = get_CIFAR10_data()\n",
    "print('Train data shape: ', X_train.shape)\n",
    "print('Train labels shape: ', y_train.shape)\n",
    "print('Validation data shape: ', X_val.shape)\n",
    "print('Validation labels shape: ', y_val.shape)\n",
    "print('Test data shape: ', X_test.shape)\n",
    "print('Test labels shape: ', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 训练网络\n",
    "\n",
    "我们使用SGD训练网络。此外，在训练过程中,我们采用指数学习率衰减计划,把学习率乘以衰减率来降低学习率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "input_size = 32 * 32 * 3\n",
    "hidden_size = 50\n",
    "num_classes = 10\n",
    "net = TwoLayerNet(input_size, hidden_size, num_classes)\n",
    "\n",
    "# Train the network\n",
    "stats = net.train(X_train, y_train, X_val, y_val,\n",
    "            num_iters=1000, batch_size=200,\n",
    "            learning_rate=1e-4, learning_rate_decay=0.95,\n",
    "            reg=0.25, verbose=True)\n",
    "\n",
    "# Predict on the validation set\n",
    "val_acc = (net.predict(X_val) == y_val).mean()\n",
    "print('Validation accuracy: ', val_acc)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Debug 训练过程\n",
    "\n",
    "使用默认参数，验证集的验证精度应该在0.29左右。太差了\n",
    "\n",
    "解决这个问题的一种策略是在训练过程中绘制损失函数, 以及训练集和验证集的准确度。\n",
    "\n",
    "另一种策略是把网络的第一层权重可视化。在大多数以视觉数据为训练对象的神经网络中，第一层的权值在可视化时通常会显示有趣的结构。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the loss function and train / validation accuracies\n",
    "plt.subplot(2, 1, 1)\n",
    "plt.plot(stats['loss_history'])\n",
    "plt.title('Loss history')\n",
    "plt.xlabel('Iteration')\n",
    "plt.ylabel('Loss')\n",
    "\n",
    "plt.subplot(2, 1, 2)\n",
    "plt.plot(stats['train_acc_history'], label='train')\n",
    "plt.plot(stats['val_acc_history'], label='val')\n",
    "plt.title('Classification accuracy history')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Classification accuracy')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from daseCV.vis_utils import visualize_grid\n",
    "\n",
    "# Visualize the weights of the network\n",
    "\n",
    "def show_net_weights(net):\n",
    "    W1 = net.params['W1']\n",
    "    W1 = W1.reshape(32, 32, 3, -1).transpose(3, 0, 1, 2)\n",
    "    plt.imshow(visualize_grid(W1, padding=3).astype('uint8'))\n",
    "    plt.gca().axis('off')\n",
    "    plt.show()\n",
    "\n",
    "show_net_weights(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 调整超参数\n",
    "\n",
    "**What's wrong?**. 查看上面的可视化，我们可以看到损失或多或少是线性下降的，这似乎表明学习率可能太小了。此外，训练的准确度和验证的准确度之间没有差距，这说明我们使用的模型容量较小，我们应该增加模型的大小。另一方面，对于一个非常大的模型，我们期望看到更多的过拟合，这表现为训练和验证准确度之间有非常大的差距。\n",
    "\n",
    "**Tuning**. 调整超参数并了解它们如何影响最终的性能是使用神经网络的一个重要部分，因此我们希望你进行大量实践。下面，你应该试验各种超参数的不同值，包括隐层大小、学习率、训练周期数和正则化强度。你也可以考虑调整学习速率衰减，但是这个实验中默认值应该能够获得良好的性能。\n",
    "\n",
    "**Approximate results**. 你应该在验证集上获得超过48%的分类准确率。我们最好的模型在验证集上获得超过52%的准确率。\n",
    "\n",
    "**Experiment**: 在这个练习中，你的任务是使用一个全连接的神经网络，在CIFAR-10上获得尽可能好的结果(52%可以作为参考)。您可以自由地实现自己的技术(例如，使用PCA来降低维度，或添加dropout，或添加特征，等等)。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "**在下面说明你的超参数搜索过程**\n",
    "\n",
    "$\\color{blue}{你的回答: }$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "best_net = None # store the best model into this \n",
    "\n",
    "#################################################################################\n",
    "# TODO：使用验证集调整超参数。 将您的最佳模型存储在best_net中。\n",
    "# 使用上面用过的可视化手段可能能够帮助你调试网络。\n",
    "# 可视化结果与上面比较差的网络有明显的差别。\n",
    "# 手工调整超参数可能很有趣，但是你会发现编写代码自动扫描超参数的可能组合会很有用。     \n",
    "#################################################################################\n",
    "# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****\n",
    "\n",
    "pass\n",
    "\n",
    "# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# visualize the weights of the best network\n",
    "show_net_weights(best_net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 在测试集上面测试\n",
    "\n",
    "当你完成实验时，你可以在测试集上评估你最终的模型；你应该得到48%以上的准确度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_acc = (best_net.predict(X_test) == y_test).mean()\n",
    "print('Test accuracy: ', test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "**问题 2**\n",
    "\n",
    "\n",
    "现在您已经完成训练了一个神经网络分类器，您可能会发现您的测试精度远远低于训练精度。我们可以用什么方法来缩小这种差距?选出下列正确的选项\n",
    "\n",
    "1. 在更大的数据集上训练\n",
    "2. 增加更多的隐藏单元\n",
    "3. 增加正则化强度\n",
    "4. 其他\n",
    "\n",
    "$\\color{blue}{\\textit Your Answer:}$\n",
    "\n",
    "$\\color{blue}{\\textit Your Explanation:}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# 重要\n",
    "\n",
    "这里是作业的结尾处，请执行以下步骤:\n",
    "\n",
    "1. 点击`File -> Save`或者用`control+s`组合键，确保你最新的的notebook的作业已经保存到谷歌云。\n",
    "2. 执行以下代码确保 `.py` 文件保存回你的谷歌云。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "FOLDER_TO_SAVE = os.path.join('drive/My Drive/', FOLDERNAME)\n",
    "FILES_TO_SAVE = ['daseCV/classifiers/neural_net.py']\n",
    "\n",
    "for files in FILES_TO_SAVE:\n",
    "  with open(os.path.join(FOLDER_TO_SAVE, '/'.join(files.split('/')[1:])), 'w') as f:\n",
    "    f.write(''.join(open(files).readlines()))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}