Web-Scraping-Lab.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Hands-on Lab : Web Scraping**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **30 to 45** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab you will perform the following:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Extract information from a given web site \n",
    "* Write the scraped data into a csv file.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract information from the given web site\n",
    "You will extract the data from the below web site: <br> \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#this url contains the data you need to scrape\n",
    "url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/datasets/Programming_Languages.html\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data you need to scrape is the **name of the programming language** and **average annual salary**.<br> It is a good idea to open the url in your web broswer and study the contents of the web page before you start to scrape.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the required libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Your code here\n",
    "from bs4 import BeautifulSoup # this module helps in web scrapping.\n",
    "import requests  # this module helps us to download a web page\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the webpage at the url\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#your code goes here\n",
    "url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/datasets/Programming_Languages.html\"\n",
    "\n",
    "# get the contents of the webpage in text format and store in a variable called data\n",
    "data  = requests.get(url).text "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a soup object\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#your code goes here\n",
    "soup = BeautifulSoup(data,\"html.parser\")  # create a soup object using the variable 'data'\n",
    "#print (soup)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scrape the `Language name` and `annual average salary`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Language--->Average Annual Salary\n",
      "Python--->$114,383\n",
      "Java--->$101,013\n",
      "R--->$92,037\n",
      "Javascript--->$110,981\n",
      "Swift--->$130,801\n",
      "C++--->$113,865\n",
      "C#--->$88,726\n",
      "PHP--->$84,727\n",
      "SQL--->$84,793\n",
      "Go--->$94,082\n"
     ]
    }
   ],
   "source": [
    "#your code goes here\n",
    "#find a html table in the web page\n",
    "table = soup.find('table') # in html table is represented by the tag <table>\n",
    "\n",
    "for row in table.find_all('tr'): # in html table row is represented by the tag <tr>\n",
    "    # Get all columns in each row.\n",
    "    cols = row.find_all('td') # in html a column is represented by the tag <td>\n",
    "    #print(cols)\n",
    "    \n",
    "    cols1= cols[0].getText()\n",
    "    cols2 = cols[1].getText()\n",
    "    cols3 = cols[2].getText() # store the value in column 3 as color_name\n",
    "    cols4 = cols[3].getText() # store the value in column 4 as color_code\n",
    "    #print(\"{}--->{}---->{}---->{}\".format(cols1,cols2,cols3, cols4))\n",
    "    print(\"{}--->{}\".format(cols2,cols4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the scrapped data into a file named *popular-languages.csv*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# your code goes here\n",
    "\n",
    "rows = table.find_all('tr')\n",
    "\n",
    "# Initialize lists to store the data\n",
    "languages = []\n",
    "salary = []\n",
    "\n",
    "# Extract the data from each row\n",
    "for row in rows[1:]:  # Skip the header row\n",
    "    cols = row.find_all('td')\n",
    "    languages.append(cols[0].text.strip())\n",
    "    salary.append(cols[1].text.strip())\n",
    "\n",
    "# Create a DataFrame\n",
    "data = {\n",
    "    'Language': languages,\n",
    "    'Salary': salary\n",
    "}\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# Save the DataFrame to a CSV file\n",
    "output_file = 'popular-languages.csv'\n",
    "df.to_csv(output_file, index=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ramesh Sannareddy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rav Ahuja\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|  Date (YYYY-MM-DD) |  Version | Changed By  |  Change Description |\n",
    "|---|---|---|---|\n",
    "| 2020-10-17  | 0.1  | Ramesh Sannareddy  |  Created initial version of the lab |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Copyright &copy; 2020 IBM Corporation. This notebook and its source code are released under the terms of the [MIT License](https://cognitiveclass.ai/mit-license/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01).\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}