Gesture_Recognition_Neural_Networks_Vishnu_Vardhan.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "L4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Problem Statement:\n",
        "\n",
        "Imagine you are working as a data scientist at a home electronics company which manufactures state of the art smart televisions. You want to develop a cool feature in the smart-TV that can recognise five different gestures performed by the user which will help users control the TV without using a remote.\n",
        "\n",
        "The gestures are continuously monitored by the webcam mounted on the TV. Each gesture corresponds to a specific command:\n",
        "\n",
        "* Thumbs up:  Increase the volume\n",
        "* Thumbs down: Decrease the volume\n",
        "* Left swipe: 'Jump' backwards 10 seconds\n",
        "* Right swipe: 'Jump' forward 10 seconds  \n",
        "* Stop: Pause the movie"
      ],
      "metadata": {
        "id": "sc3JIkZGpx4l"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Understanding:\n",
        "\n",
        "* The training data consists of a few hundred videos categorised into one of the five classes.\n",
        "* Each video (typically 2-3 seconds long) is divided into a sequence of 30 frames(images).\n",
        "*  videos have two types of dimensions - either 360x360 or 120x160 (depending on the webcam used to record the videos)\n",
        "\n",
        "\n",
        "* Data can be downloaded from: https://drive.google.com/uc?id=1ehyrYBQ5rbQQe6yL4XbLWe3FMvuVUGiL"
      ],
      "metadata": {
        "id": "dqk6aRzoqTgt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 1. Importing Neccessary libraries"
      ],
      "metadata": {
        "id": "BrwBiKBvq4fo"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "Y2XhWiM93-eJ"
      },
      "outputs": [],
      "source": [
        "# Importing the necessary libraries\n",
        "import numpy as np\n",
        "import os\n",
        "#from scipy.misc.pilutil import imread, imresize\n",
        "import datetime\n",
        "import os\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "import abc\n",
        "from sys import getsizeof"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from imageio import imread\n",
        "from skimage.transform import resize\n",
        "#from scipy.misc import imread, imresize\n",
        "from PIL import Image, ImageFilter, ImageEnhance"
      ],
      "metadata": {
        "id": "HpYf60rm4LwE"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tensorflow==2.09"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Dy5JoKWVcnHe",
        "outputId": "a98a56de-ca5d-4294-fb99-d648ccc0cc7c"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: tensorflow==2.09 in /usr/local/lib/python3.10/dist-packages (2.9.0)\n",
            "Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (1.4.0)\n",
            "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (1.6.3)\n",
            "Requirement already satisfied: flatbuffers<2,>=1.12 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (1.12)\n",
            "Requirement already satisfied: gast<=0.4.0,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (0.4.0)\n",
            "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (0.2.0)\n",
            "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (1.64.1)\n",
            "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (3.11.0)\n",
            "Requirement already satisfied: keras<2.10.0,>=2.9.0rc0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (2.9.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (1.1.2)\n",
            "Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (18.1.1)\n",
            "Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (1.26.4)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (3.3.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (24.1)\n",
            "Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (3.19.6)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (71.0.4)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (1.16.0)\n",
            "Requirement already satisfied: tensorboard<2.10,>=2.9 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (2.9.1)\n",
            "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (0.37.1)\n",
            "Requirement already satisfied: tensorflow-estimator<2.10.0,>=2.9.0rc0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (2.9.0)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (2.4.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (4.12.2)\n",
            "Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow==2.09) (1.16.0)\n",
            "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow==2.09) (0.44.0)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.10,>=2.9->tensorflow==2.09) (2.27.0)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.10,>=2.9->tensorflow==2.09) (0.4.6)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.10,>=2.9->tensorflow==2.09) (3.6)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.10,>=2.9->tensorflow==2.09) (2.32.3)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.10,>=2.9->tensorflow==2.09) (0.6.1)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.10,>=2.9->tensorflow==2.09) (1.8.1)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.10,>=2.9->tensorflow==2.09) (3.0.3)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.10,>=2.9->tensorflow==2.09) (5.4.0)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.10,>=2.9->tensorflow==2.09) (0.4.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.10,>=2.9->tensorflow==2.09) (4.9)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.10,>=2.9->tensorflow==2.09) (1.3.1)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.10,>=2.9->tensorflow==2.09) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.10,>=2.9->tensorflow==2.09) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.10,>=2.9->tensorflow==2.09) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.10,>=2.9->tensorflow==2.09) (2024.7.4)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.10,>=2.9->tensorflow==2.09) (2.1.5)\n",
            "Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.10,>=2.9->tensorflow==2.09) (0.6.0)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.10,>=2.9->tensorflow==2.09) (3.2.2)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "np.random.seed(30)\n",
        "import random as rn\n",
        "rn.seed(30)\n",
        "from keras import backend as K\n",
        "import tensorflow as tf\n",
        "tf.random.set_seed(30)"
      ],
      "metadata": {
        "id": "Jwivq1GT4aoM"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import cv2\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline"
      ],
      "metadata": {
        "id": "9ECYKILB4fta"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 2. Importing data"
      ],
      "metadata": {
        "id": "-U5C5ATKrDxG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this block, you read the folder names for training and validation. You also set the `batch_size` here. Note that you set the batch size in such a way that you are able to use the GPU in full capacity. You keep increasing the batch size until the machine throws an error."
      ],
      "metadata": {
        "id": "_tuviOfN4mrg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import gdown"
      ],
      "metadata": {
        "id": "KnLqtzZEFOWp"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "file_id = \"1ehyrYBQ5rbQQe6yL4XbLWe3FMvuVUGiL\"\n",
        "file_name = \"Project_data.zip\"\n",
        "gdown.download(f\"https://drive.google.com/uc?id={file_id}\", file_name)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 145
        },
        "id": "vriZ0VmW4k7U",
        "outputId": "2082d7fa-0124-4ef5-e380-2b3e55bef40b"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Downloading...\n",
            "From (original): https://drive.google.com/uc?id=1ehyrYBQ5rbQQe6yL4XbLWe3FMvuVUGiL\n",
            "From (redirected): https://drive.google.com/uc?id=1ehyrYBQ5rbQQe6yL4XbLWe3FMvuVUGiL&confirm=t&uuid=f16ae1fb-9711-4606-b6e1-182931694057\n",
            "To: /content/Project_data.zip\n",
            "100%|██████████| 1.71G/1.71G [00:44<00:00, 38.1MB/s]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Project_data.zip'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Remove extracted files if already exist\n",
        "!rm -rf 'Project_data'\n",
        "# Unzip file\n",
        "!unzip -qq Project_data.zip"
      ],
      "metadata": {
        "id": "Sqx2uhJoFB_j"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_doc = np.random.permutation(open('/content/Project_data/train.csv').readlines())\n",
        "val_doc = np.random.permutation(open('/content/Project_data/val.csv').readlines())\n",
        "batch_size = 32 #experiment with the batch size"
      ],
      "metadata": {
        "id": "uz2xXs3kF5gA"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_doc.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2eklHjPOGxHo",
        "outputId": "adc8f2b3-b648-451c-f59a-075a3d92aa5d"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(663,)"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "val_doc.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BjB7OzTAHGog",
        "outputId": "63152f14-6a7b-40e8-f413-bf690c274077"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(100,)"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 3. Generator Function\n",
        "\n",
        "This is one of the most important part of the code. The overall structure of the generator has been given. In the generator, you are going to preprocess the images as you have images of 2 different dimensions as well as create a batch of video frames. You have to experiment with `img_idx`, `y`, `z` and normalization such that you get high accuracy."
      ],
      "metadata": {
        "id": "-We7Ehi8HRou"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "img_idx = [0,1,2,4,6,8,10,12,14,16,18,20,22,24,26,27,28,29] #create a list of image numbers you want to use for a particular video\n",
        "frames = len(img_idx)\n",
        "shape_h = 120\n",
        "shape_w = 120"
      ],
      "metadata": {
        "id": "tinVudy8zUOS"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# # generator with augmentation for train data\n",
        "# def generator(source_path, folder_list, batch_size, is_train = False, augmention = False, debug=False):\n",
        "#     # print('\\nSource path = ', source_path, '; batch size =', batch_size)\n",
        "#     x = len(img_idx)\n",
        "#     y, z = shape_h, shape_w\n",
        "#     while True:\n",
        "#         # doubling the data for augmentation\n",
        "#         if is_train and augmention:\n",
        "#             t = np.concatenate((np.random.permutation(folder_list), np.random.permutation(folder_list)))\n",
        "#         else:\n",
        "#             t = np.random.permutation(folder_list)\n",
        "\n",
        "#         if (len(t)%batch_size) == 0:\n",
        "#             num_batches = int(len(t)/batch_size)\n",
        "#         else:\n",
        "#             num_batches = len(t)//batch_size + 1\n",
        "\n",
        "#         for batch in range(num_batches): # we iterate over the number of batches\n",
        "#             batch_data = np.zeros((batch_size,x,y,z,3)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
        "#             batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
        "#             for folder in range(batch_size): # iterate over the batch_size\n",
        "#                 if debug:\n",
        "#                     plt.figure(figsize=(20,5))\n",
        "#                 #handling remaining datapoints\n",
        "#                 folder_idx = folder + (batch*batch_size)\n",
        "#                 if folder_idx >= len(t):\n",
        "#                     break\n",
        "#                 folder_str = t[folder_idx]\n",
        "#                 imgs = os.listdir(source_path+'/'+ folder_str.split(';')[0]) # read all the images in the folder\n",
        "#                 # randomly enabling augmentation and augmentation type\n",
        "#                 aug_type = None\n",
        "#                 if is_train and augmention and rn.randint(0,1) == 1:\n",
        "#                     aug_type = rn.randint(0, 4) #randomly selecting augmentation type\n",
        "#                 for idx,item in enumerate(img_idx): #  Iterate over the frames/images of a folder to read them in\n",
        "#                     image = imread(source_path+'/'+ folder_str.strip().split(';')[0]+'/'+imgs[item]).astype(np.float32)\n",
        "\n",
        "#                     # plotting original images for debugging purpose only\n",
        "#                     if debug:\n",
        "#                         plt.subplot(2, x, idx+1)\n",
        "#                         plt.imshow(image.astype('uint8'))\n",
        "\n",
        "#                     #crop the images and resize them. Note that the images are of 2 different shape\n",
        "#                     #and the conv3D will throw error if the inputs in a batch have different shapes\n",
        "#                     # making the rectangle images into square by cropping sides\n",
        "#                     # so the aspect ration can be mantained while resizing.\n",
        "#                     if image.shape[1] > image.shape[0]:\n",
        "#                         diff_px = image.shape[1] - image.shape[0]\n",
        "#                         crop_start = diff_px//2\n",
        "#                         crop_end = crop_start + image.shape[0]\n",
        "#                         image = image[:, crop_start:crop_end]\n",
        "#                     elif image.shape[0] > image.shape[1]:\n",
        "#                         diff_px = image.shape[0] - image.shape[1]\n",
        "#                         crop_start = diff_px//2\n",
        "#                         crop_end = crop_start + image.shape[1]\n",
        "#                         image = image[:, crop_start:crop_end]\n",
        "\n",
        "#                     resized_im = resize(image, (y,z))\n",
        "\n",
        "#                     if aug_type is not None:\n",
        "#                         if aug_type == 0: # edge Enhancement\n",
        "#                             resized_im = np.array(Image.fromarray(resized_im, 'RGB').filter(ImageFilter.EDGE_ENHANCE))\n",
        "#                         elif aug_type == 1: # adding gaussian blur\n",
        "#                             resized_im = np.array(Image.fromarray(resized_im, 'RGB').filter(ImageFilter.GaussianBlur(1)))\n",
        "#                         elif aug_type == 2: # enchancing image detailing\n",
        "#                             resized_im = np.array(Image.fromarray(resized_im, 'RGB').filter(ImageFilter.DETAIL))\n",
        "#                         elif aug_type == 3: # sharpening image\n",
        "#                             resized_im = np.array(Image.fromarray(resized_im, 'RGB').filter(ImageFilter.SHARPEN))\n",
        "#                         elif aug_type == 4: # Brightness enhancement\n",
        "#                             resized_im = np.array(ImageEnhance.Brightness((Image.fromarray(resized_im, 'RGB'))).enhance(1.5))\n",
        "#                     # plotting rezised images for debugging purpose only\n",
        "#                     if debug:\n",
        "#                         plt.subplot(2, x, idx+x+1)\n",
        "#                         plt.imshow(resized_im)\n",
        "\n",
        "#                     batch_data[folder,idx,:,:,0] = resized_im[:,:,0]/255 #normalise and feed in the image\n",
        "#                     batch_data[folder,idx,:,:,1] = resized_im[:,:,1]/255 #normalise and feed in the image\n",
        "#                     batch_data[folder,idx,:,:,2] = resized_im[:,:,2]/255 #normalise and feed in the image\n",
        "\n",
        "#                 batch_labels[folder, int(folder_str.strip().split(';')[2])] = 1\n",
        "#             yield batch_data, batch_labels #you yield the batch_data and the batch_labels, remember what does yield do\n"
      ],
      "metadata": {
        "id": "wAipAa64WJQq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def generator(source_path, folder_list, batch_size):\n",
        "    print( 'Source path = ', source_path, '; batch size =', batch_size)\n",
        "    while True:\n",
        "        t = np.random.permutation(folder_list)\n",
        "        num_batches = int(len(t)/batch_size) # calculate the number of batches\n",
        "        # left over batches which should be handled separately\n",
        "        leftover_batches = len(t) - num_batches * batch_size\n",
        "\n",
        "        for batch in range(num_batches): # we iterate over the number of batches\n",
        "            batch_data = np.zeros((batch_size, len(img_idx), shape_h, shape_w, 3)) # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
        "            batch_labels = np.zeros((batch_size,5)) # batch_labels is the one hot representation of the output\n",
        "            for folder in range(batch_size): # iterate over the batch_size\n",
        "                imgs = os.listdir(source_path+'/'+ t[folder + (batch*batch_size)].split(';')[0]) # read all the images in the folder\n",
        "                for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
        "                    image = imread(source_path+'/'+ t[folder + (batch*batch_size)].strip().split(';')[0]+'/'+imgs[item]).astype(np.float32)\n",
        "\n",
        "                    #crop the images and resize them. Note that the images are of 2 different shape\n",
        "                    #and the conv3D will throw error if the inputs in a batch have different shapes\n",
        "                    image = resize(image, (shape_h, shape_w))\n",
        "                    batch_data[folder,idx,:,:,0] = (image[:,:,0])/255\n",
        "                    batch_data[folder,idx,:,:,1] = (image[:,:,1])/255\n",
        "                    batch_data[folder,idx,:,:,2] = (image[:,:,2])/255\n",
        "\n",
        "                batch_labels[folder, int(t[folder + (batch*batch_size)].strip().split(';')[2])] = 1\n",
        "            yield batch_data, batch_labels #you yield the batch_data and the batch_labels, remember what does yield do\n",
        "\n",
        "\n",
        "        # write the code for the remaining data points which are left after full batches\n",
        "        # write the code for the remaining data points which are left after full batches\n",
        "        if leftover_batches != 0:\n",
        "            for batch in range(num_batches):\n",
        "                # x is the number of images you use for each video, (y,z) is the final size of the input images and 3 is the number of channels RGB\n",
        "                batch_data = np.zeros((batch_size,len(img_idx),shape_h, shape_w,3))\n",
        "                # batch_labels is the one hot representation of the output: 10 videos with 5 columns as classes\n",
        "                batch_labels = np.zeros((batch_size,5))\n",
        "                for folder in range(batch_size): # iterate over the batch_size\n",
        "                    imgs = os.listdir(source_path +'/'+t[batch * batch_size + folder].split(';')[0])\n",
        "                    for idx,item in enumerate(img_idx): #  Iterate iver the frames/images of a folder to read them in\n",
        "\n",
        "                        image = imread(source_path +'/'+t[batch * batch_size + folder].split(';')[0] +'/'+imgs[item]).astype(np.float32)\n",
        "                        image = resize(image, (shape_h,shape_w))\n",
        "\n",
        "                        batch_data[folder,idx,:,:,0] = (image[:,:,0])/255\n",
        "                        batch_data[folder,idx,:,:,1] = (image[:,:,1])/255\n",
        "                        batch_data[folder,idx,:,:,2] = (image[:,:,2])/255\n",
        "\n",
        "                    #Fill the one hot encoding stuff where we maintain the label\n",
        "                    batch_labels[folder, int(t[batch * batch_size + folder].split(';')[2])] = 1\n",
        "                yield batch_data, batch_labels #you yield the batch_data and the batch_labels, remember what does yield do"
      ],
      "metadata": {
        "id": "OQRyzJlUHIU9"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note here that a video is represented above in the generator as (number of images, height, width, number of channels). Take this into consideration while creating the model architecture."
      ],
      "metadata": {
        "id": "PIrZEdBnHhq5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "curr_dt_time = datetime.datetime.now()\n",
        "train_path = '/content/Project_data/train'\n",
        "val_path = '/content/Project_data/val'\n",
        "num_train_sequences = len(train_doc)\n",
        "print('# training sequences =', num_train_sequences)\n",
        "num_val_sequences = len(val_doc)\n",
        "print('# validation sequences =', num_val_sequences)\n",
        "num_epochs = 20 # choose the number of epochs\n",
        "print ('# epochs =', num_epochs)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Gu1qxq7JHbN9",
        "outputId": "2f13109a-71a8-491b-9453-d060accf7bd3"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "# training sequences = 663\n",
            "# validation sequences = 100\n",
            "# epochs = 20\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 4. Model\n",
        "\n",
        "Here you make the model using different functionalities that Keras provides. Remember to use Conv3D and MaxPooling3D and not Conv2D and Maxpooling2D for a 3D convolution model. You would want to use TimeDistributed while building a Conv2D + RNN model. Also remember that the last layer is the softmax. Design the network in such a way that the model is able to give good accuracy on the least number of parameters so that it can fit in the memory of the webcam."
      ],
      "metadata": {
        "id": "yvaYN0RtHqDN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from keras.models import Sequential, Model\n",
        "from keras.layers import Dense, GRU, Flatten, TimeDistributed, Flatten, BatchNormalization, Activation, Dropout,GlobalAveragePooling2D, GlobalAveragePooling3D, ConvLSTM2D\n",
        "from tensorflow.keras.layers import Conv2D,Conv3D, MaxPooling3D\n",
        "from keras.callbacks import ModelCheckpoint, ReduceLROnPlateau\n",
        "from keras import optimizers"
      ],
      "metadata": {
        "id": "qB6iAozGHk1w"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3D Convolutional Neural Networks (Conv3D) Model"
      ],
      "metadata": {
        "id": "0S94YoXYrd9-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = Sequential()\n",
        "model.add(Conv3D(64, (3,3,3), strides=(1,1,1), padding='same', input_shape=(len(img_idx),120,120,3)))\n",
        "model.add(BatchNormalization())\n",
        "model.add(Activation('elu'))\n",
        "model.add(MaxPooling3D(pool_size=(2,2,1), strides=(2,2,1)))\n",
        "\n",
        "model.add(Conv3D(128, (3,3,3), strides=(1,1,1), padding='same'))\n",
        "model.add(BatchNormalization())\n",
        "model.add(Activation('elu'))\n",
        "model.add(MaxPooling3D(pool_size=(2,2,2), strides=(2,2,2)))\n",
        "\n",
        "# model.add(Dropout(0.25))\n",
        "\n",
        "model.add(Conv3D(256, (3,3,3), strides=(1,1,1), padding='same'))\n",
        "model.add(BatchNormalization())\n",
        "model.add(Activation('elu'))\n",
        "model.add(MaxPooling3D(pool_size=(2,2,2), strides=(2,2,2)))\n",
        "\n",
        "# model.add(Dropout(0.25))\n",
        "\n",
        "model.add(Conv3D(256, (3,3,3), strides=(1,1,1), padding='same'))\n",
        "model.add(BatchNormalization())\n",
        "model.add(Activation('elu'))\n",
        "model.add(MaxPooling3D(pool_size=(2,2,2), strides=(2,2,2)))\n",
        "\n",
        "model.add(Flatten())\n",
        "model.add(Dropout(0.5))\n",
        "model.add(Dense(512, activation='elu'))\n",
        "model.add(Dropout(0.5))\n",
        "model.add(Dense(5, activation='softmax'))"
      ],
      "metadata": {
        "id": "gvBSMl1bHtem"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sgd = optimizers.SGD(learning_rate=0.001, decay=1e-6, momentum=0.7, nesterov=True)\n",
        "model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "print (model.summary())\n",
        "# #optimiser = 'adam' #write your optimizer\n",
        "# #model.compile(optimizer=optimiser, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "# #print (model.summary())"
      ],
      "metadata": {
        "id": "mOAakNthHzIZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6d8fd382-97c2-4b43-ee12-bcbe86ad4314"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " conv3d (Conv3D)             (None, 18, 120, 120, 64)  5248      \n",
            "                                                                 \n",
            " batch_normalization_4 (Batc  (None, 18, 120, 120, 64)  256      \n",
            " hNormalization)                                                 \n",
            "                                                                 \n",
            " activation (Activation)     (None, 18, 120, 120, 64)  0         \n",
            "                                                                 \n",
            " max_pooling3d (MaxPooling3D  (None, 9, 60, 120, 64)   0         \n",
            " )                                                               \n",
            "                                                                 \n",
            " conv3d_1 (Conv3D)           (None, 9, 60, 120, 128)   221312    \n",
            "                                                                 \n",
            " batch_normalization_5 (Batc  (None, 9, 60, 120, 128)  512       \n",
            " hNormalization)                                                 \n",
            "                                                                 \n",
            " activation_1 (Activation)   (None, 9, 60, 120, 128)   0         \n",
            "                                                                 \n",
            " max_pooling3d_1 (MaxPooling  (None, 4, 30, 60, 128)   0         \n",
            " 3D)                                                             \n",
            "                                                                 \n",
            " conv3d_2 (Conv3D)           (None, 4, 30, 60, 256)    884992    \n",
            "                                                                 \n",
            " batch_normalization_6 (Batc  (None, 4, 30, 60, 256)   1024      \n",
            " hNormalization)                                                 \n",
            "                                                                 \n",
            " activation_2 (Activation)   (None, 4, 30, 60, 256)    0         \n",
            "                                                                 \n",
            " max_pooling3d_2 (MaxPooling  (None, 2, 15, 30, 256)   0         \n",
            " 3D)                                                             \n",
            "                                                                 \n",
            " conv3d_3 (Conv3D)           (None, 2, 15, 30, 256)    1769728   \n",
            "                                                                 \n",
            " batch_normalization_7 (Batc  (None, 2, 15, 30, 256)   1024      \n",
            " hNormalization)                                                 \n",
            "                                                                 \n",
            " activation_3 (Activation)   (None, 2, 15, 30, 256)    0         \n",
            "                                                                 \n",
            " max_pooling3d_3 (MaxPooling  (None, 1, 7, 15, 256)    0         \n",
            " 3D)                                                             \n",
            "                                                                 \n",
            " flatten (Flatten)           (None, 26880)             0         \n",
            "                                                                 \n",
            " dropout (Dropout)           (None, 26880)             0         \n",
            "                                                                 \n",
            " dense_3 (Dense)             (None, 512)               13763072  \n",
            "                                                                 \n",
            " dropout_1 (Dropout)         (None, 512)               0         \n",
            "                                                                 \n",
            " dense_4 (Dense)             (None, 5)                 2565      \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 16,649,733\n",
            "Trainable params: 16,648,325\n",
            "Non-trainable params: 1,408\n",
            "_________________________________________________________________\n",
            "None\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let us create the `train_generator` and the `val_generator` which will be used in `.fit_generator`."
      ],
      "metadata": {
        "id": "sn9GXz11IAhR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_generator = generator(train_path, train_doc, batch_size)\n",
        "val_generator = generator(val_path, val_doc, batch_size)"
      ],
      "metadata": {
        "id": "NlivLahtH5aB"
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_name = 'model_Conv3D' + '_' + str(curr_dt_time).replace(' ','').replace(':','_') + '/'\n",
        "\n",
        "if not os.path.exists(model_name):\n",
        "    os.mkdir(model_name)\n",
        "\n",
        "filepath = model_name + 'model-{epoch:05d}-{loss:.5f}-{categorical_accuracy:.5f}-{val_loss:.5f}-{val_categorical_accuracy:.5f}..keras'\n",
        "\n",
        "checkpoint = ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=False, save_weights_only=False, mode='auto')\n",
        "#\n",
        "\n",
        "LR = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=2, verbose=1, mode='min', epsilon=0.0001, cooldown=0, min_lr=0.00001)\n",
        "callbacks_list = [checkpoint, LR]"
      ],
      "metadata": {
        "id": "cT1QM-P-IFRG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4c995df4-c904-4bbd-b242-361e521ec440"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:`epsilon` argument is deprecated and will be removed, use `min_delta` instead.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The `steps_per_epoch` and `validation_steps` are used by `fit_generator` to decide the number of next() calls it need to make."
      ],
      "metadata": {
        "id": "pjpJWR2RJ48o"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "if (num_train_sequences%batch_size) == 0:\n",
        "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
        "else:\n",
        "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
        "\n",
        "if (num_val_sequences%batch_size) == 0:\n",
        "    validation_steps = int(num_val_sequences/batch_size)\n",
        "else:\n",
        "    validation_steps = (num_val_sequences//batch_size) + 1"
      ],
      "metadata": {
        "id": "t_bxZH5gIfUq"
      },
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let us now fit the model. This will start training the model and with the help of the checkpoints, you'll be able to save the model at the end of each epoch."
      ],
      "metadata": {
        "id": "mU5yhA8jKGGP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "num_epochs=25"
      ],
      "metadata": {
        "id": "4ZprOl6VMdw8"
      },
      "execution_count": 36,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1,\n",
        "                     callbacks=callbacks_list, validation_data=val_generator,\n",
        "                     validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
      ],
      "metadata": {
        "id": "CMGYAKMVJ8hz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "505c28dc-33b4-4e46-9470-2de1b381b3ca"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Source path =  /content/Project_data/train ; batch size = 32\n",
            "Epoch 1/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 4.6548 - categorical_accuracy: 0.3661 Source path =  /content/Project_data/val ; batch size = 32\n",
            "\n",
            "Epoch 1: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00001-4.65485-0.36607-1.89518-0.25781..keras\n",
            "21/21 [==============================] - 243s 12s/step - loss: 4.6548 - categorical_accuracy: 0.3661 - val_loss: 1.8952 - val_categorical_accuracy: 0.2578 - lr: 0.0010\n",
            "Epoch 2/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 1.3439 - categorical_accuracy: 0.5536 \n",
            "Epoch 2: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00002-1.34391-0.55357-3.45089-0.30469..keras\n",
            "21/21 [==============================] - 242s 12s/step - loss: 1.3439 - categorical_accuracy: 0.5536 - val_loss: 3.4509 - val_categorical_accuracy: 0.3047 - lr: 0.0010\n",
            "Epoch 3/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 1.2514 - categorical_accuracy: 0.5744 \n",
            "Epoch 3: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00003-1.25141-0.57440-4.88261-0.32812..keras\n",
            "\n",
            "Epoch 3: ReduceLROnPlateau reducing learning rate to 0.0005000000237487257.\n",
            "21/21 [==============================] - 242s 12s/step - loss: 1.2514 - categorical_accuracy: 0.5744 - val_loss: 4.8826 - val_categorical_accuracy: 0.3281 - lr: 0.0010\n",
            "Epoch 4/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 1.0252 - categorical_accuracy: 0.6429 \n",
            "Epoch 4: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00004-1.02520-0.64286-6.26951-0.30469..keras\n",
            "21/21 [==============================] - 242s 12s/step - loss: 1.0252 - categorical_accuracy: 0.6429 - val_loss: 6.2695 - val_categorical_accuracy: 0.3047 - lr: 5.0000e-04\n",
            "Epoch 5/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.8472 - categorical_accuracy: 0.6801 \n",
            "Epoch 5: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00005-0.84721-0.68006-7.43570-0.32812..keras\n",
            "\n",
            "Epoch 5: ReduceLROnPlateau reducing learning rate to 0.0002500000118743628.\n",
            "21/21 [==============================] - 241s 12s/step - loss: 0.8472 - categorical_accuracy: 0.6801 - val_loss: 7.4357 - val_categorical_accuracy: 0.3281 - lr: 5.0000e-04\n",
            "Epoch 6/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.8215 - categorical_accuracy: 0.6964 \n",
            "Epoch 6: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00006-0.82151-0.69643-7.53900-0.33594..keras\n",
            "21/21 [==============================] - 242s 12s/step - loss: 0.8215 - categorical_accuracy: 0.6964 - val_loss: 7.5390 - val_categorical_accuracy: 0.3359 - lr: 2.5000e-04\n",
            "Epoch 7/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.7961 - categorical_accuracy: 0.7068 \n",
            "Epoch 7: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00007-0.79605-0.70685-8.54268-0.22656..keras\n",
            "\n",
            "Epoch 7: ReduceLROnPlateau reducing learning rate to 0.0001250000059371814.\n",
            "21/21 [==============================] - 241s 12s/step - loss: 0.7961 - categorical_accuracy: 0.7068 - val_loss: 8.5427 - val_categorical_accuracy: 0.2266 - lr: 2.5000e-04\n",
            "Epoch 8/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.7067 - categorical_accuracy: 0.7470 \n",
            "Epoch 8: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00008-0.70667-0.74702-8.66516-0.30469..keras\n",
            "21/21 [==============================] - 242s 12s/step - loss: 0.7067 - categorical_accuracy: 0.7470 - val_loss: 8.6652 - val_categorical_accuracy: 0.3047 - lr: 1.2500e-04\n",
            "Epoch 9/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6831 - categorical_accuracy: 0.7262 \n",
            "Epoch 9: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00009-0.68310-0.72619-8.21547-0.36719..keras\n",
            "\n",
            "Epoch 9: ReduceLROnPlateau reducing learning rate to 6.25000029685907e-05.\n",
            "21/21 [==============================] - 242s 12s/step - loss: 0.6831 - categorical_accuracy: 0.7262 - val_loss: 8.2155 - val_categorical_accuracy: 0.3672 - lr: 1.2500e-04\n",
            "Epoch 10/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.7199 - categorical_accuracy: 0.7262 \n",
            "Epoch 10: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00010-0.71994-0.72619-8.78082-0.26562..keras\n",
            "21/21 [==============================] - 241s 12s/step - loss: 0.7199 - categorical_accuracy: 0.7262 - val_loss: 8.7808 - val_categorical_accuracy: 0.2656 - lr: 6.2500e-05\n",
            "Epoch 11/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6801 - categorical_accuracy: 0.7366 \n",
            "Epoch 11: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00011-0.68005-0.73661-8.55029-0.26562..keras\n",
            "\n",
            "Epoch 11: ReduceLROnPlateau reducing learning rate to 3.125000148429535e-05.\n",
            "21/21 [==============================] - 242s 12s/step - loss: 0.6801 - categorical_accuracy: 0.7366 - val_loss: 8.5503 - val_categorical_accuracy: 0.2656 - lr: 6.2500e-05\n",
            "Epoch 12/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6827 - categorical_accuracy: 0.7143 \n",
            "Epoch 12: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00012-0.68270-0.71429-8.17179-0.31250..keras\n",
            "21/21 [==============================] - 243s 12s/step - loss: 0.6827 - categorical_accuracy: 0.7143 - val_loss: 8.1718 - val_categorical_accuracy: 0.3125 - lr: 3.1250e-05\n",
            "Epoch 13/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.7237 - categorical_accuracy: 0.7173 \n",
            "Epoch 13: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00013-0.72368-0.71726-8.73683-0.27344..keras\n",
            "\n",
            "Epoch 13: ReduceLROnPlateau reducing learning rate to 1.5625000742147677e-05.\n",
            "21/21 [==============================] - 241s 12s/step - loss: 0.7237 - categorical_accuracy: 0.7173 - val_loss: 8.7368 - val_categorical_accuracy: 0.2734 - lr: 3.1250e-05\n",
            "Epoch 14/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.7040 - categorical_accuracy: 0.7396 \n",
            "Epoch 14: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00014-0.70404-0.73958-8.73396-0.25000..keras\n",
            "21/21 [==============================] - 241s 12s/step - loss: 0.7040 - categorical_accuracy: 0.7396 - val_loss: 8.7340 - val_categorical_accuracy: 0.2500 - lr: 1.5625e-05\n",
            "Epoch 15/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.7041 - categorical_accuracy: 0.7470 \n",
            "Epoch 15: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00015-0.70410-0.74702-7.95650-0.29688..keras\n",
            "\n",
            "Epoch 15: ReduceLROnPlateau reducing learning rate to 1e-05.\n",
            "21/21 [==============================] - 241s 12s/step - loss: 0.7041 - categorical_accuracy: 0.7470 - val_loss: 7.9565 - val_categorical_accuracy: 0.2969 - lr: 1.5625e-05\n",
            "Epoch 16/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6190 - categorical_accuracy: 0.7693 \n",
            "Epoch 16: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00016-0.61900-0.76935-7.35990-0.31250..keras\n",
            "21/21 [==============================] - 241s 12s/step - loss: 0.6190 - categorical_accuracy: 0.7693 - val_loss: 7.3599 - val_categorical_accuracy: 0.3125 - lr: 1.0000e-05\n",
            "Epoch 17/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6650 - categorical_accuracy: 0.7470 \n",
            "Epoch 17: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00017-0.66500-0.74702-6.82950-0.35156..keras\n",
            "21/21 [==============================] - 242s 12s/step - loss: 0.6650 - categorical_accuracy: 0.7470 - val_loss: 6.8295 - val_categorical_accuracy: 0.3516 - lr: 1.0000e-05\n",
            "Epoch 18/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6212 - categorical_accuracy: 0.7545 \n",
            "Epoch 18: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00018-0.62122-0.75446-6.54465-0.31250..keras\n",
            "21/21 [==============================] - 240s 12s/step - loss: 0.6212 - categorical_accuracy: 0.7545 - val_loss: 6.5447 - val_categorical_accuracy: 0.3125 - lr: 1.0000e-05\n",
            "Epoch 19/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6350 - categorical_accuracy: 0.7604 \n",
            "Epoch 19: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00019-0.63495-0.76042-5.85488-0.32031..keras\n",
            "21/21 [==============================] - 241s 12s/step - loss: 0.6350 - categorical_accuracy: 0.7604 - val_loss: 5.8549 - val_categorical_accuracy: 0.3203 - lr: 1.0000e-05\n",
            "Epoch 20/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6595 - categorical_accuracy: 0.7485 \n",
            "Epoch 20: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00020-0.65950-0.74851-5.10046-0.31250..keras\n",
            "21/21 [==============================] - 241s 12s/step - loss: 0.6595 - categorical_accuracy: 0.7485 - val_loss: 5.1005 - val_categorical_accuracy: 0.3125 - lr: 1.0000e-05\n",
            "Epoch 21/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6807 - categorical_accuracy: 0.7426 \n",
            "Epoch 21: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00021-0.68072-0.74256-4.69201-0.31250..keras\n",
            "21/21 [==============================] - 240s 11s/step - loss: 0.6807 - categorical_accuracy: 0.7426 - val_loss: 4.6920 - val_categorical_accuracy: 0.3125 - lr: 1.0000e-05\n",
            "Epoch 22/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6471 - categorical_accuracy: 0.7485 \n",
            "Epoch 22: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00022-0.64715-0.74851-3.78148-0.35156..keras\n",
            "21/21 [==============================] - 241s 12s/step - loss: 0.6471 - categorical_accuracy: 0.7485 - val_loss: 3.7815 - val_categorical_accuracy: 0.3516 - lr: 1.0000e-05\n",
            "Epoch 23/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6929 - categorical_accuracy: 0.7307 \n",
            "Epoch 23: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00023-0.69294-0.73065-3.55918-0.32031..keras\n",
            "21/21 [==============================] - 240s 11s/step - loss: 0.6929 - categorical_accuracy: 0.7307 - val_loss: 3.5592 - val_categorical_accuracy: 0.3203 - lr: 1.0000e-05\n",
            "Epoch 24/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6921 - categorical_accuracy: 0.7262 \n",
            "Epoch 24: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00024-0.69214-0.72619-2.68367-0.39062..keras\n",
            "21/21 [==============================] - 240s 11s/step - loss: 0.6921 - categorical_accuracy: 0.7262 - val_loss: 2.6837 - val_categorical_accuracy: 0.3906 - lr: 1.0000e-05\n",
            "Epoch 25/25\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6492 - categorical_accuracy: 0.7470 \n",
            "Epoch 25: saving model to model_Conv3D_2024-08-0916_28_11.225274/model-00025-0.64924-0.74702-2.51409-0.40625..keras\n",
            "21/21 [==============================] - 239s 11s/step - loss: 0.6492 - categorical_accuracy: 0.7470 - val_loss: 2.5141 - val_categorical_accuracy: 0.4062 - lr: 1.0000e-05\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7b5e6e63ba00>"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## CNN2D + ConvLSTM2D"
      ],
      "metadata": {
        "id": "REoU_e4MTgmQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = Sequential()\n",
        "model.add(TimeDistributed(\n",
        "    Conv2D(8, (3,3), activation='relu'), input_shape=(len(img_idx),120,120,3))\n",
        ")\n",
        "model.add(BatchNormalization())\n",
        "model.add(TimeDistributed(\n",
        "    Conv2D(16, (3,3), activation='relu'))\n",
        ")\n",
        "model.add(BatchNormalization())\n",
        "model.add(\n",
        "    ConvLSTM2D(8, kernel_size = 3, return_sequences=False)\n",
        ")\n",
        "model.add(BatchNormalization())\n",
        "model.add(TimeDistributed(\n",
        "    Dense(64, activation='relu'))\n",
        ")\n",
        "model.add(BatchNormalization())\n",
        "model.add(GlobalAveragePooling2D())\n",
        "model.add(Dense(64, activation='relu'))\n",
        "model.add(Dense(5, activation='softmax'))"
      ],
      "metadata": {
        "id": "SyZMt-OAkumH"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "optimiser = optimizers.Adam(lr=0.01) #write your optimizer\n",
        "model.compile(optimizer=optimiser, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "print(model.summary())\n",
        "#sgd = optimizers.SGD(learning_rate=0.001, decay=1e-6, momentum=0.7, nesterov=True)\n",
        "#model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['categorical_accuracy'])\n",
        "#print (model.summary())"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dRecSIU2VJtx",
        "outputId": "75622a47-7de9-41e2-dcc4-3a40d3e3b064"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"sequential\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " time_distributed (TimeDistr  (None, 18, 118, 118, 8)  224       \n",
            " ibuted)                                                         \n",
            "                                                                 \n",
            " batch_normalization (BatchN  (None, 18, 118, 118, 8)  32        \n",
            " ormalization)                                                   \n",
            "                                                                 \n",
            " time_distributed_1 (TimeDis  (None, 18, 116, 116, 16)  1168     \n",
            " tributed)                                                       \n",
            "                                                                 \n",
            " batch_normalization_1 (Batc  (None, 18, 116, 116, 16)  64       \n",
            " hNormalization)                                                 \n",
            "                                                                 \n",
            " conv_lstm2d (ConvLSTM2D)    (None, 114, 114, 8)       6944      \n",
            "                                                                 \n",
            " batch_normalization_2 (Batc  (None, 114, 114, 8)      32        \n",
            " hNormalization)                                                 \n",
            "                                                                 \n",
            " time_distributed_2 (TimeDis  (None, 114, 114, 64)     576       \n",
            " tributed)                                                       \n",
            "                                                                 \n",
            " batch_normalization_3 (Batc  (None, 114, 114, 64)     256       \n",
            " hNormalization)                                                 \n",
            "                                                                 \n",
            " global_average_pooling2d (G  (None, 64)               0         \n",
            " lobalAveragePooling2D)                                          \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 64)                4160      \n",
            "                                                                 \n",
            " dense_2 (Dense)             (None, 5)                 325       \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 13,781\n",
            "Trainable params: 13,589\n",
            "Non-trainable params: 192\n",
            "_________________________________________________________________\n",
            "None\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# train_generator = generator(train_path, train_doc, batch_size, is_train = True, augmention = enable_augmentation)\n",
        "# val_generator = generator(val_path, val_doc, batch_size)"
      ],
      "metadata": {
        "id": "bNwhCfJeXk83"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "batch_size=32"
      ],
      "metadata": {
        "id": "nx9OgQFNqDy_"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_generator = generator(train_path, train_doc, batch_size)\n",
        "val_generator = generator(val_path, val_doc, batch_size)"
      ],
      "metadata": {
        "id": "8fN8NiQRVMbH"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model_name = 'model_init_CNN2D+convLSTM' + '_' + str(curr_dt_time).replace(' ','').replace(':','_') + '/'\n",
        "\n",
        "if not os.path.exists(model_name):\n",
        "    os.mkdir(model_name)\n",
        "\n",
        "filepath = model_name + 'CNN2D+convLSTM-{epoch:05d}-{loss:.5f}-{categorical_accuracy:.5f}-{val_loss:.5f}-{val_categorical_accuracy:.5f}..keras'\n",
        "\n",
        "checkpoint = ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=False, save_weights_only=False, mode='auto')\n",
        "#\n",
        "\n",
        "LR = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=5, verbose=1, mode='min', epsilon=0.0001, cooldown=0, min_lr=0.001) #0.00001\n",
        "callbacks_list = [checkpoint, LR]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zM2BvQr2Vg27",
        "outputId": "a7062b9a-f2a8-47f2-c75c-74daf46bb6db"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:tensorflow:`epsilon` argument is deprecated and will be removed, use `min_delta` instead.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "if (num_train_sequences%batch_size) == 0:\n",
        "    steps_per_epoch = int(num_train_sequences/batch_size)\n",
        "else:\n",
        "    steps_per_epoch = (num_train_sequences//batch_size) + 1\n",
        "\n",
        "if (num_val_sequences%batch_size) == 0:\n",
        "    validation_steps = int(num_val_sequences/batch_size)\n",
        "else:\n",
        "    validation_steps = (num_val_sequences//batch_size) + 1"
      ],
      "metadata": {
        "id": "hcbsSP8mVxB-"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "num_epochs=50"
      ],
      "metadata": {
        "id": "l0B97L-ppOYt"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=num_epochs, verbose=1,\n",
        "                    callbacks=callbacks_list, validation_data=val_generator,\n",
        "                    validation_steps=validation_steps, class_weight=None, workers=1, initial_epoch=0)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QkXw3-yaqVK_",
        "outputId": "f605a8cd-a358-438b-eeaa-882fd82e59c4"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Source path =  /content/Project_data/train ; batch size = 32\n",
            "Epoch 1/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 1.3301 - categorical_accuracy: 0.3720Source path =  /content/Project_data/val ; batch size = 32\n",
            "\n",
            "Epoch 1: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00001-1.33005-0.37202-1.80222-0.21875..keras\n",
            "21/21 [==============================] - 140s 7s/step - loss: 1.3301 - categorical_accuracy: 0.3720 - val_loss: 1.8022 - val_categorical_accuracy: 0.2188 - lr: 0.0100\n",
            "Epoch 2/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 1.1155 - categorical_accuracy: 0.4940\n",
            "Epoch 2: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00002-1.11550-0.49405-2.32022-0.20312..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 1.1155 - categorical_accuracy: 0.4940 - val_loss: 2.3202 - val_categorical_accuracy: 0.2031 - lr: 0.0100\n",
            "Epoch 3/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 1.0127 - categorical_accuracy: 0.5253\n",
            "Epoch 3: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00003-1.01266-0.52530-2.18769-0.22656..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 1.0127 - categorical_accuracy: 0.5253 - val_loss: 2.1877 - val_categorical_accuracy: 0.2266 - lr: 0.0100\n",
            "Epoch 4/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.9305 - categorical_accuracy: 0.5625\n",
            "Epoch 4: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00004-0.93047-0.56250-2.04253-0.21094..keras\n",
            "21/21 [==============================] - 139s 7s/step - loss: 0.9305 - categorical_accuracy: 0.5625 - val_loss: 2.0425 - val_categorical_accuracy: 0.2109 - lr: 0.0100\n",
            "Epoch 5/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.9127 - categorical_accuracy: 0.5670\n",
            "Epoch 5: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00005-0.91275-0.56696-1.52998-0.28906..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.9127 - categorical_accuracy: 0.5670 - val_loss: 1.5300 - val_categorical_accuracy: 0.2891 - lr: 0.0100\n",
            "Epoch 6/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.7759 - categorical_accuracy: 0.6562\n",
            "Epoch 6: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00006-0.77585-0.65625-1.88606-0.28906..keras\n",
            "21/21 [==============================] - 135s 7s/step - loss: 0.7759 - categorical_accuracy: 0.6562 - val_loss: 1.8861 - val_categorical_accuracy: 0.2891 - lr: 0.0100\n",
            "Epoch 7/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.7270 - categorical_accuracy: 0.6771\n",
            "Epoch 7: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00007-0.72704-0.67708-5.82723-0.22656..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.7270 - categorical_accuracy: 0.6771 - val_loss: 5.8272 - val_categorical_accuracy: 0.2266 - lr: 0.0100\n",
            "Epoch 8/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6817 - categorical_accuracy: 0.6979\n",
            "Epoch 8: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00008-0.68166-0.69792-1.61509-0.32031..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.6817 - categorical_accuracy: 0.6979 - val_loss: 1.6151 - val_categorical_accuracy: 0.3203 - lr: 0.0100\n",
            "Epoch 9/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.7188 - categorical_accuracy: 0.6786\n",
            "Epoch 9: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00009-0.71879-0.67857-2.79407-0.31250..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.7188 - categorical_accuracy: 0.6786 - val_loss: 2.7941 - val_categorical_accuracy: 0.3125 - lr: 0.0100\n",
            "Epoch 10/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6893 - categorical_accuracy: 0.6905\n",
            "Epoch 10: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00010-0.68932-0.69048-2.44826-0.32031..keras\n",
            "\n",
            "Epoch 10: ReduceLROnPlateau reducing learning rate to 0.004999999888241291.\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.6893 - categorical_accuracy: 0.6905 - val_loss: 2.4483 - val_categorical_accuracy: 0.3203 - lr: 0.0100\n",
            "Epoch 11/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.5831 - categorical_accuracy: 0.7411\n",
            "Epoch 11: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00011-0.58305-0.74107-1.48287-0.42188..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.5831 - categorical_accuracy: 0.7411 - val_loss: 1.4829 - val_categorical_accuracy: 0.4219 - lr: 0.0050\n",
            "Epoch 12/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.5682 - categorical_accuracy: 0.7634\n",
            "Epoch 12: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00012-0.56823-0.76339-2.22728-0.32812..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.5682 - categorical_accuracy: 0.7634 - val_loss: 2.2273 - val_categorical_accuracy: 0.3281 - lr: 0.0050\n",
            "Epoch 13/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.5508 - categorical_accuracy: 0.7649\n",
            "Epoch 13: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00013-0.55079-0.76488-2.02535-0.38281..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.5508 - categorical_accuracy: 0.7649 - val_loss: 2.0254 - val_categorical_accuracy: 0.3828 - lr: 0.0050\n",
            "Epoch 14/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.4917 - categorical_accuracy: 0.7976\n",
            "Epoch 14: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00014-0.49168-0.79762-1.87092-0.53906..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.4917 - categorical_accuracy: 0.7976 - val_loss: 1.8709 - val_categorical_accuracy: 0.5391 - lr: 0.0050\n",
            "Epoch 15/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.6041 - categorical_accuracy: 0.7604\n",
            "Epoch 15: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00015-0.60411-0.76042-1.35862-0.54688..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.6041 - categorical_accuracy: 0.7604 - val_loss: 1.3586 - val_categorical_accuracy: 0.5469 - lr: 0.0050\n",
            "Epoch 16/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.4968 - categorical_accuracy: 0.7723\n",
            "Epoch 16: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00016-0.49683-0.77232-1.31340-0.50781..keras\n",
            "21/21 [==============================] - 135s 7s/step - loss: 0.4968 - categorical_accuracy: 0.7723 - val_loss: 1.3134 - val_categorical_accuracy: 0.5078 - lr: 0.0050\n",
            "Epoch 17/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.4276 - categorical_accuracy: 0.8021\n",
            "Epoch 17: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00017-0.42762-0.80208-2.32322-0.32812..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.4276 - categorical_accuracy: 0.8021 - val_loss: 2.3232 - val_categorical_accuracy: 0.3281 - lr: 0.0050\n",
            "Epoch 18/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.4565 - categorical_accuracy: 0.8065\n",
            "Epoch 18: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00018-0.45654-0.80655-1.48445-0.46875..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.4565 - categorical_accuracy: 0.8065 - val_loss: 1.4845 - val_categorical_accuracy: 0.4688 - lr: 0.0050\n",
            "Epoch 19/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.4547 - categorical_accuracy: 0.7932\n",
            "Epoch 19: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00019-0.45467-0.79315-0.76818-0.71094..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.4547 - categorical_accuracy: 0.7932 - val_loss: 0.7682 - val_categorical_accuracy: 0.7109 - lr: 0.0050\n",
            "Epoch 20/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.4326 - categorical_accuracy: 0.8155\n",
            "Epoch 20: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00020-0.43265-0.81548-0.85922-0.65625..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.4326 - categorical_accuracy: 0.8155 - val_loss: 0.8592 - val_categorical_accuracy: 0.6562 - lr: 0.0050\n",
            "Epoch 21/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.4424 - categorical_accuracy: 0.7961\n",
            "Epoch 21: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00021-0.44242-0.79613-1.11577-0.59375..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.4424 - categorical_accuracy: 0.7961 - val_loss: 1.1158 - val_categorical_accuracy: 0.5938 - lr: 0.0050\n",
            "Epoch 22/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.4575 - categorical_accuracy: 0.8080\n",
            "Epoch 22: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00022-0.45755-0.80804-2.12588-0.46875..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.4575 - categorical_accuracy: 0.8080 - val_loss: 2.1259 - val_categorical_accuracy: 0.4688 - lr: 0.0050\n",
            "Epoch 23/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.4670 - categorical_accuracy: 0.8140\n",
            "Epoch 23: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00023-0.46697-0.81399-1.50799-0.46094..keras\n",
            "21/21 [==============================] - 138s 7s/step - loss: 0.4670 - categorical_accuracy: 0.8140 - val_loss: 1.5080 - val_categorical_accuracy: 0.4609 - lr: 0.0050\n",
            "Epoch 24/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.3935 - categorical_accuracy: 0.8318\n",
            "Epoch 24: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00024-0.39352-0.83185-0.63454-0.72656..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.3935 - categorical_accuracy: 0.8318 - val_loss: 0.6345 - val_categorical_accuracy: 0.7266 - lr: 0.0050\n",
            "Epoch 25/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.3522 - categorical_accuracy: 0.8616\n",
            "Epoch 25: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00025-0.35224-0.86161-1.16370-0.58594..keras\n",
            "21/21 [==============================] - 135s 7s/step - loss: 0.3522 - categorical_accuracy: 0.8616 - val_loss: 1.1637 - val_categorical_accuracy: 0.5859 - lr: 0.0050\n",
            "Epoch 26/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.4062 - categorical_accuracy: 0.8214\n",
            "Epoch 26: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00026-0.40618-0.82143-0.92862-0.76562..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.4062 - categorical_accuracy: 0.8214 - val_loss: 0.9286 - val_categorical_accuracy: 0.7656 - lr: 0.0050\n",
            "Epoch 27/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.3372 - categorical_accuracy: 0.8631\n",
            "Epoch 27: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00027-0.33721-0.86310-1.24346-0.64062..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.3372 - categorical_accuracy: 0.8631 - val_loss: 1.2435 - val_categorical_accuracy: 0.6406 - lr: 0.0050\n",
            "Epoch 28/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.3231 - categorical_accuracy: 0.8735\n",
            "Epoch 28: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00028-0.32313-0.87351-0.80042-0.78125..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.3231 - categorical_accuracy: 0.8735 - val_loss: 0.8004 - val_categorical_accuracy: 0.7812 - lr: 0.0050\n",
            "Epoch 29/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.3019 - categorical_accuracy: 0.8824\n",
            "Epoch 29: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00029-0.30188-0.88244-1.46440-0.62500..keras\n",
            "\n",
            "Epoch 29: ReduceLROnPlateau reducing learning rate to 0.0024999999441206455.\n",
            "21/21 [==============================] - 135s 7s/step - loss: 0.3019 - categorical_accuracy: 0.8824 - val_loss: 1.4644 - val_categorical_accuracy: 0.6250 - lr: 0.0050\n",
            "Epoch 30/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.2408 - categorical_accuracy: 0.9092\n",
            "Epoch 30: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00030-0.24083-0.90923-0.96601-0.68750..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.2408 - categorical_accuracy: 0.9092 - val_loss: 0.9660 - val_categorical_accuracy: 0.6875 - lr: 0.0025\n",
            "Epoch 31/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.2476 - categorical_accuracy: 0.9062\n",
            "Epoch 31: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00031-0.24761-0.90625-0.90743-0.78125..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.2476 - categorical_accuracy: 0.9062 - val_loss: 0.9074 - val_categorical_accuracy: 0.7812 - lr: 0.0025\n",
            "Epoch 32/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.2947 - categorical_accuracy: 0.8795\n",
            "Epoch 32: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00032-0.29475-0.87946-1.34991-0.71094..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.2947 - categorical_accuracy: 0.8795 - val_loss: 1.3499 - val_categorical_accuracy: 0.7109 - lr: 0.0025\n",
            "Epoch 33/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.2302 - categorical_accuracy: 0.9092\n",
            "Epoch 33: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00033-0.23025-0.90923-1.11525-0.75000..keras\n",
            "21/21 [==============================] - 135s 7s/step - loss: 0.2302 - categorical_accuracy: 0.9092 - val_loss: 1.1153 - val_categorical_accuracy: 0.7500 - lr: 0.0025\n",
            "Epoch 34/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.2373 - categorical_accuracy: 0.9092\n",
            "Epoch 34: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00034-0.23727-0.90923-0.78246-0.78906..keras\n",
            "\n",
            "Epoch 34: ReduceLROnPlateau reducing learning rate to 0.0012499999720603228.\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.2373 - categorical_accuracy: 0.9092 - val_loss: 0.7825 - val_categorical_accuracy: 0.7891 - lr: 0.0025\n",
            "Epoch 35/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.1917 - categorical_accuracy: 0.9405\n",
            "Epoch 35: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00035-0.19165-0.94048-0.66819-0.83594..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.1917 - categorical_accuracy: 0.9405 - val_loss: 0.6682 - val_categorical_accuracy: 0.8359 - lr: 0.0012\n",
            "Epoch 36/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.1770 - categorical_accuracy: 0.9449\n",
            "Epoch 36: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00036-0.17699-0.94494-0.82216-0.82031..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.1770 - categorical_accuracy: 0.9449 - val_loss: 0.8222 - val_categorical_accuracy: 0.8203 - lr: 0.0012\n",
            "Epoch 37/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.2098 - categorical_accuracy: 0.9196\n",
            "Epoch 37: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00037-0.20981-0.91964-0.92039-0.77344..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.2098 - categorical_accuracy: 0.9196 - val_loss: 0.9204 - val_categorical_accuracy: 0.7734 - lr: 0.0012\n",
            "Epoch 38/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.1889 - categorical_accuracy: 0.9345\n",
            "Epoch 38: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00038-0.18892-0.93452-0.61226-0.84375..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.1889 - categorical_accuracy: 0.9345 - val_loss: 0.6123 - val_categorical_accuracy: 0.8438 - lr: 0.0012\n",
            "Epoch 39/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.1974 - categorical_accuracy: 0.9226\n",
            "Epoch 39: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00039-0.19743-0.92262-0.67496-0.85156..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.1974 - categorical_accuracy: 0.9226 - val_loss: 0.6750 - val_categorical_accuracy: 0.8516 - lr: 0.0012\n",
            "Epoch 40/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.1707 - categorical_accuracy: 0.9375\n",
            "Epoch 40: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00040-0.17074-0.93750-0.56166-0.82812..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.1707 - categorical_accuracy: 0.9375 - val_loss: 0.5617 - val_categorical_accuracy: 0.8281 - lr: 0.0012\n",
            "Epoch 41/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.2201 - categorical_accuracy: 0.9062\n",
            "Epoch 41: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00041-0.22009-0.90625-0.57854-0.85156..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.2201 - categorical_accuracy: 0.9062 - val_loss: 0.5785 - val_categorical_accuracy: 0.8516 - lr: 0.0012\n",
            "Epoch 42/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.1952 - categorical_accuracy: 0.9286\n",
            "Epoch 42: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00042-0.19518-0.92857-0.61166-0.82812..keras\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.1952 - categorical_accuracy: 0.9286 - val_loss: 0.6117 - val_categorical_accuracy: 0.8281 - lr: 0.0012\n",
            "Epoch 43/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.1881 - categorical_accuracy: 0.9405\n",
            "Epoch 43: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00043-0.18811-0.94048-0.58607-0.84375..keras\n",
            "21/21 [==============================] - 135s 7s/step - loss: 0.1881 - categorical_accuracy: 0.9405 - val_loss: 0.5861 - val_categorical_accuracy: 0.8438 - lr: 0.0012\n",
            "Epoch 44/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.1757 - categorical_accuracy: 0.9390\n",
            "Epoch 44: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00044-0.17572-0.93899-0.77327-0.84375..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.1757 - categorical_accuracy: 0.9390 - val_loss: 0.7733 - val_categorical_accuracy: 0.8438 - lr: 0.0012\n",
            "Epoch 45/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.1773 - categorical_accuracy: 0.9286\n",
            "Epoch 45: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00045-0.17729-0.92857-0.74619-0.83594..keras\n",
            "\n",
            "Epoch 45: ReduceLROnPlateau reducing learning rate to 0.001.\n",
            "21/21 [==============================] - 137s 7s/step - loss: 0.1773 - categorical_accuracy: 0.9286 - val_loss: 0.7462 - val_categorical_accuracy: 0.8359 - lr: 0.0012\n",
            "Epoch 46/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.1500 - categorical_accuracy: 0.9479\n",
            "Epoch 46: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00046-0.15004-0.94792-0.81629-0.82031..keras\n",
            "21/21 [==============================] - 138s 7s/step - loss: 0.1500 - categorical_accuracy: 0.9479 - val_loss: 0.8163 - val_categorical_accuracy: 0.8203 - lr: 0.0010\n",
            "Epoch 47/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.1792 - categorical_accuracy: 0.9449\n",
            "Epoch 47: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00047-0.17920-0.94494-0.82975-0.78125..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.1792 - categorical_accuracy: 0.9449 - val_loss: 0.8298 - val_categorical_accuracy: 0.7812 - lr: 0.0010\n",
            "Epoch 48/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.1724 - categorical_accuracy: 0.9390\n",
            "Epoch 48: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00048-0.17239-0.93899-0.73440-0.79688..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.1724 - categorical_accuracy: 0.9390 - val_loss: 0.7344 - val_categorical_accuracy: 0.7969 - lr: 0.0010\n",
            "Epoch 49/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.2191 - categorical_accuracy: 0.9211\n",
            "Epoch 49: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00049-0.21913-0.92113-0.72107-0.81250..keras\n",
            "21/21 [==============================] - 136s 7s/step - loss: 0.2191 - categorical_accuracy: 0.9211 - val_loss: 0.7211 - val_categorical_accuracy: 0.8125 - lr: 0.0010\n",
            "Epoch 50/50\n",
            "21/21 [==============================] - ETA: 0s - loss: 0.2240 - categorical_accuracy: 0.9196\n",
            "Epoch 50: saving model to model_init_CNN2D+convLSTM_2024-08-1002_21_21.466451/CNN2D+convLSTM-00050-0.22398-0.91964-0.91301-0.80469..keras\n",
            "21/21 [==============================] - 135s 7s/step - loss: 0.2240 - categorical_accuracy: 0.9196 - val_loss: 0.9130 - val_categorical_accuracy: 0.8047 - lr: 0.0010\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7c435d625ed0>"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Conclusions:\n",
        "\n",
        "After iterations with trail and error, two models are considered Conv3D and combination of Conv2D and ConvLSTM. various Iterations of batch_size , Epochs, learning rate are considered to get the best outcome\n",
        "\n",
        "It is observed that, Conv2D + ConvLSTM gives the best results with maximum training accuracy of 94% and validation accuracy of 84%.\n",
        "\n",
        "Hence, Conv2D + ConvLSTM model with batch_size of 32, number of epochs as 50, learning rate of 0.01 with Adam as optimizer is best choice for the problem statement"
      ],
      "metadata": {
        "id": "2BVzZaa9puE8"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "4NTHY3WptCZo"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}