DaSE-Computer-Vision-2021/assignment1/features.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "\n",
    "drive.mount('/content/drive', force_remount=True)\n",
    "\n",
    "# 输入daseCV所在的路径\n",
    "# 'daseCV' 文件夹包括 '.py', 'classifiers' 和'datasets'文件夹\n",
    "# 例如 'CV/assignments/assignment1/daseCV/'\n",
    "FOLDERNAME = None\n",
    "\n",
    "assert FOLDERNAME is not None, \"[!] Enter the foldername.\"\n",
    "\n",
    "%cd drive/My\\ Drive\n",
    "%cp -r $FOLDERNAME ../../\n",
    "%cd ../../\n",
    "%cd daseCV/datasets/\n",
    "!bash get_datasets.sh\n",
    "%cd ../../"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# 图像特征练习\n",
    "*补充并完成本练习。*\n",
    "\n",
    "我们已经看到，通过在输入图像的像素上训练线性分类器，从而在图像分类任务上达到一个合理的性能。在本练习中，我们将展示我们可以通过对线性分类器（不是在原始像素上，而是在根据原始像素计算出的特征上）进行训练来改善分类性能。\n",
    "\n",
    "你将在此notebook中完成本练习的所有工作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "from daseCV.data_utils import load_CIFAR10\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading extenrnal modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "source": [
    "## 数据加载\n",
    "与之前的练习类似，我们将从磁盘加载CIFAR-10数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "from daseCV.features import color_histogram_hsv, hog_feature\n",
    "\n",
    "def get_CIFAR10_data(num_training=49000, num_validation=1000, num_test=1000):\n",
    "    # Load the raw CIFAR-10 data\n",
    "    cifar10_dir = 'daseCV/datasets/cifar-10-batches-py'\n",
    "\n",
    "    # Cleaning up variables to prevent loading data multiple times (which may cause memory issue)\n",
    "    try:\n",
    "       del X_train, y_train\n",
    "       del X_test, y_test\n",
    "       print('Clear previously loaded data.')\n",
    "    except:\n",
    "       pass\n",
    "\n",
    "    X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)\n",
    "    \n",
    "    # Subsample the data\n",
    "    mask = list(range(num_training, num_training + num_validation))\n",
    "    X_val = X_train[mask]\n",
    "    y_val = y_train[mask]\n",
    "    mask = list(range(num_training))\n",
    "    X_train = X_train[mask]\n",
    "    y_train = y_train[mask]\n",
    "    mask = list(range(num_test))\n",
    "    X_test = X_test[mask]\n",
    "    y_test = y_test[mask]\n",
    "    \n",
    "    return X_train, y_train, X_val, y_val, X_test, y_test\n",
    "\n",
    "X_train, y_train, X_val, y_val, X_test, y_test = get_CIFAR10_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "source": [
    "## 特征提取\n",
    "对于每一张图片我们将会计算它的方向梯度直方图（英語：Histogram of oriented gradient，简称HOG）以及在HSV颜色空间使用色相通道的颜色直方图。\n",
    "\n",
    "简单来讲，HOG能提取图片的纹理信息而忽略颜色信息，颜色直方图则提取出颜色信息而忽略纹理信息。\n",
    "因此，我们希望将两者结合使用而不是单独使用任一个。去实现这个设想是一个十分有趣的事情。\n",
    "\n",
    "`hog_feature` 和 `color_histogram_hsv`两个函数都可以对单个图像进行运算并返回改图像的一个特征向量。\n",
    "extract_features函数输入一个图像集合和一个特征函数列表然后对每张图片运行每个特征函数，\n",
    "然后将结果存储在一个矩阵中，矩阵的每一列是单个图像的所有特征向量的串联。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "from daseCV.features import *\n",
    "\n",
    "num_color_bins = 10 # Number of bins in the color histogram\n",
    "feature_fns = [hog_feature, lambda img: color_histogram_hsv(img, nbin=num_color_bins)]\n",
    "X_train_feats = extract_features(X_train, feature_fns, verbose=True)\n",
    "X_val_feats = extract_features(X_val, feature_fns)\n",
    "X_test_feats = extract_features(X_test, feature_fns)\n",
    "\n",
    "# Preprocessing: Subtract the mean feature\n",
    "mean_feat = np.mean(X_train_feats, axis=0, keepdims=True)\n",
    "X_train_feats -= mean_feat\n",
    "X_val_feats -= mean_feat\n",
    "X_test_feats -= mean_feat\n",
    "\n",
    "# Preprocessing: Divide by standard deviation. This ensures that each feature\n",
    "# has roughly the same scale.\n",
    "std_feat = np.std(X_train_feats, axis=0, keepdims=True)\n",
    "X_train_feats /= std_feat\n",
    "X_val_feats /= std_feat\n",
    "X_test_feats /= std_feat\n",
    "\n",
    "# Preprocessing: Add a bias dimension\n",
    "X_train_feats = np.hstack([X_train_feats, np.ones((X_train_feats.shape[0], 1))])\n",
    "X_val_feats = np.hstack([X_val_feats, np.ones((X_val_feats.shape[0], 1))])\n",
    "X_test_feats = np.hstack([X_test_feats, np.ones((X_test_feats.shape[0], 1))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用特征训练SVM\n",
    "使用之前作业完成的多分类SVM代码来训练上面提取的特征。这应该比原始数据直接在SVM上训练会去的更好的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "# 使用验证集调整学习率和正则化强度\n",
    "\n",
    "from daseCV.classifiers.linear_classifier import LinearSVM\n",
    "\n",
    "learning_rates = [1e-9, 1e-8, 1e-7]\n",
    "regularization_strengths = [5e4, 5e5, 5e6]\n",
    "\n",
    "results = {}\n",
    "best_val = -1\n",
    "best_svm = None\n",
    "\n",
    "################################################################################\n",
    "# 你需要做的: \n",
    "# 使用验证集设置学习率和正则化强度。\n",
    "# 这应该与你对SVM所做的验证相同；\n",
    "# 将训练最好的的分类器保存在best_svm中。\n",
    "# 你可能还想在颜色直方图中使用不同数量的bins。\n",
    "# 如果你细心一点应该能够在验证集上获得接近0.44的准确性。    \n",
    "################################################################################\n",
    "# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****\n",
    "\n",
    "pass\n",
    "\n",
    "# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****\n",
    "\n",
    "# Print out results.\n",
    "for lr, reg in sorted(results):\n",
    "    train_accuracy, val_accuracy = results[(lr, reg)]\n",
    "    print('lr %e reg %e train accuracy: %f val accuracy: %f' % (\n",
    "                lr, reg, train_accuracy, val_accuracy))\n",
    "    \n",
    "print('best validation accuracy achieved during cross-validation: %f' % best_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate your trained SVM on the test set\n",
    "y_test_pred = best_svm.predict(X_test_feats)\n",
    "test_accuracy = np.mean(y_test == y_test_pred)\n",
    "print(test_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 直观了解算法工作原理的一种重要方法是可视化它所犯的错误。\n",
    "# 在此可视化中，我们显示了当前系统未正确分类的图像示例。\n",
    "# 第一列显示的图像是我们的系统标记为“ plane”，但其真实标记不是“ plane”。\n",
    "\n",
    "examples_per_class = 8\n",
    "classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\n",
    "for cls, cls_name in enumerate(classes):\n",
    "    idxs = np.where((y_test != cls) & (y_test_pred == cls))[0]\n",
    "    idxs = np.random.choice(idxs, examples_per_class, replace=False)\n",
    "    for i, idx in enumerate(idxs):\n",
    "        plt.subplot(examples_per_class, len(classes), i * len(classes) + cls + 1)\n",
    "        plt.imshow(X_test[idx].astype('uint8'))\n",
    "        plt.axis('off')\n",
    "        if i == 0:\n",
    "            plt.title(cls_name)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "**问题 1:**\n",
    "\n",
    "描述你看到的错误分类结果。你认为他们有道理吗？\n",
    "\n",
    "$\\color{blue}{\\textit 答:}$ **在这里写上你的回答**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 图像特征神经网络\n",
    "在之前的练习中，我们看到在原始像素上训练两层神经网络比线性分类器具有更好的分类精度。在这里，我们已经看到使用图像特征的线性分类器优于使用原始像素的线性分类器。\n",
    "为了完整起见，我们还应该尝试在图像特征上训练神经网络。这种方法应优于以前所有的方法：你应该能够轻松地在测试集上达到55％以上的分类精度；我们最好的模型可达到约60％的精度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "# Preprocessing: Remove the bias dimension\n",
    "# Make sure to run this cell only ONCE\n",
    "print(X_train_feats.shape)\n",
    "X_train_feats = X_train_feats[:, :-1]\n",
    "X_val_feats = X_val_feats[:, :-1]\n",
    "X_test_feats = X_test_feats[:, :-1]\n",
    "\n",
    "print(X_train_feats.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "from daseCV.classifiers.neural_net import TwoLayerNet\n",
    "\n",
    "input_dim = X_train_feats.shape[1]\n",
    "hidden_dim = 500\n",
    "num_classes = 10\n",
    "best_acc = 0.0\n",
    "\n",
    "net = TwoLayerNet(input_dim, hidden_dim, num_classes)\n",
    "best_net = None\n",
    "\n",
    "################################################################################\n",
    "# TODO: 使用图像特征训练两层神经网络。\n",
    "# 您可能希望像上一节中那样对各种参数进行交叉验证。\n",
    "# 将最佳的模型存储在best_net变量中。                                             \n",
    "################################################################################\n",
    "# *****START OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****\n",
    "\n",
    "pass\n",
    "\n",
    "# *****END OF YOUR CODE (DO NOT DELETE/MODIFY THIS LINE)*****\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在测试集上运行得到的最好的神经网络分类器，应该能够获得55％以上的准确性。\n",
    "\n",
    "test_acc = (best_net.predict(X_test_feats) == y_test).mean()\n",
    "print(test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# 重要\n",
    "\n",
    "这里是作业的结尾处，请执行以下步骤:\n",
    "\n",
    "1. 点击`File -> Save`或者用`control+s`组合键，确保你最新的的notebook的作业已经保存到谷歌云。\n",
    "2. 执行以下代码确保 `.py` 文件保存回你的谷歌云。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "FOLDER_TO_SAVE = os.path.join('drive/My Drive/', FOLDERNAME)\n",
    "FILES_TO_SAVE = []\n",
    "\n",
    "for files in FILES_TO_SAVE:\n",
    "  with open(os.path.join(FOLDER_TO_SAVE, '/'.join(files.split('/')[1:])), 'w') as f:\n",
    "    f.write(''.join(open(files).readlines()))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}