Web Scraping and Crawling with Python: Beautiful Soup, Requests & Selenium
What you'll learn
Python Refresher: Review of Data Structures, Conditionals, File Handling
How Websites are Hosted on Servers; Basic Calls to Server (GET, POST Methods)
Web Scraping with Python Beautiful Soup and Requests
Using Selenium to handle javascript and AJAX
Diverse Web Scraping Exercises
Source codes (*.py files) for all Exercises can be downloaded
Q&A board to send your questions and get them answered quickly
Requirements
Some prior programming experience in Python (e.g. Data Structures and OOP) will help. The course includes a full Python refresher section.
Complete beginners may wish to take a beginner Python course first, and then transition to this course afterwards.
This course adopts a step-by-step approach and requires you to open a Python editor, download available *.py code files, and start applying the provided examples and exercises.
Python 3: Codes of this course are tested on Python 3. It is up to you to adapt them if you want to run them in Python 2.
Description
Web Scraping (also termed Screen Scraping, Web Data Extraction, Web Harvesting, etc.) is a technique for extracting large amounts of data from websites and save the the extracted data to a local file or to a database.
In this course, you will learn how to perform web scraping using Python 3 and the Beautiful Soup, a free open-source library written in Python for parsing HTML.
We will use lxml, which is an extensive library for parsing XML and HTML documents very quickly; it can even handle messed up tags. We will also be using the Requests module instead of the already built-in urllib2 module due to improvements in speed and readability.
Finally, we will use Selenium alongside Beautiful Soup to crawl AJAX & javascript driven pages.
The course cover the following topics: accessing web pages programmatically; scraping web pages to extract the required data using Beautiful Soup to parse web pages; interacting with web pages to do different things with them programmatically; and using Selenium for web scraping and when we need it.
By the end of this course, you will be able to understand how websites and servers function, diverse data extraction techniques, and methods of handling and organizing data.
This Web Scraping course covers the following topics
Review of data structures (Lists, Dictionaries, Tuples, File Handling)How websites are hosted on serversCalls to the server (GET, POST methods)Review of HTML and CSSRequests Module and BeautifulSoup Module overviewParsing HTML using BeautifulSoupFiltering elements using BeautifulSoup and navigating the Parse Treejavascript and AJAX overviewSelenium and the need for itSelecting elements using Selenium CSS selectors XPath selectors Navigating pages using Selenium Practical Projects
Overview
Section 1: Web Scraping Course Overview
Lecture 1 Web Scraping Course Overview
Section 2: Python Refresher: Data Structures (Optional)
Lecture 2 Lists
Lecture 3 Dictionaries
Lecture 4 Tuples
Lecture 5 List Comprehensions - Part 1
Lecture 6 List Comprehensions - Part 2
Lecture 7 Inline - if else and List Comprehensions
Lecture 8 Installing xlrd and XlsxWriter​ to Read/Write to Excel Files
Lecture 9 Wrting to Excel Files
Lecture 10 Reading from Excel Files
Lecture 11 Python Editor & Other Software
Lecture 12 Exercise #1: YOU: Web Scraping Expert
Section 3: How Servers Work
Lecture 13 How Websites are Hosted
Lecture 14 HTML Revision
Section 4: BeautifulSoup Warm-up Exercise
Lecture 15 BeautifulSoup Solved Exercise
Section 5: Installing Required Python Packages
Lecture 16 Installing Required Python Packages
Section 6: Introduction to Requests Python Library
Lecture 17 Requests Get Method
Lecture 18 User Agent
Lecture 19 Installing fake_useragent Package
Section 7: Introduction to Beautiful Soup Python Library
Lecture 20 Web Scraping with Beautiful Soup - Overview
Lecture 21 Web Scraping with Beautiful Soup - Overview P.2
Lecture 22 Accessing Tags
Lecture 23 Navigable Strings
Section 8: Navigating with Beautiful Soup - Going Down
Lecture 24 Navigating through Tag Names
Lecture 25 Contents and Children Methods
Lecture 26 Descendants Method
Section 9: Navigating with Beautiful Soup - Going Up
Lecture 27 Parent Method
Lecture 28 Parents Method
Section 10: Navigating with Beautiful Soup - Going Sideways
Lecture 29 next_sibling
Lecture 30 previous_sibling
Lecture 31 next_siblings & previous_siblings
Section 11: Regular Expressions with Python
Lecture 32 Metacharacters Overview
Lecture 33 Compile Function and Character Class
Lecture 34 Special Sequences
Lecture 35 * Repeating Things
Lecture 36 + Repeating Things
Lecture 37 ? and {m,n} Repeating Things
Lecture 38 Metacharacters part2
Section 12: Searching the Parse Tree Using Beautiful Soup
Lecture 39 Introduction to Searching with BeautifulSoup
Lecture 40 find_all Function
Lecture 41 find_all More Parameters
Lecture 42 find Function
Section 13: Project 1: Scraping CustomerReports Website
Lecture 43 Web Scraping CustomerReports - part 1
Lecture 44 Web Scraping CustomerReports - part 2
Section 14: Project 2: Web Scraping CodingBat Website with Beautiful Soup
Lecture 45 Project 2 Description
Lecture 46 Web Scraping CodingBat - part 1
Lecture 47 Web Scraping CodingBat - part 2
Lecture 48 Web Scraping CodingBat - part 3
Section 15: Using Selenium to Handle AJAX & javascript Driven Web Pages
Lecture 49 javascript, AJAX and Selenium intro
Lecture 50 Installing Selenium
Lecture 51 Installing ChromeDriver
Lecture 52 Introduction to Selenium
Lecture 53 Searching Elements and Inputting Data
Lecture 54 Clicking Elements
Lecture 55 XPath Introduction
Lecture 56 XPath Examples
Section 16: Project 3: Web Scraping Your Instagram Account
Lecture 57 Project 3 Description
Lecture 58 Logging in to Instagram
Lecture 59 Settings Tab!
Lecture 60 Opening Target Profile (NEW)
Lecture 61 Scrolling Down v.1 (NEW)
Lecture 62 Scrolling Down v.2 (NEW)
Lecture 63 Exception Handling (NEW)
Lecture 64 Making Folders (NEW)
Lecture 65 Downloading Images v.1 (NEW)
Lecture 66 Downloading Images v.2 (NEW)
Lecture 67 Downloading Captions (NEW)
Lecture 68 Writing Captions to Excel File (NEW)
Lecture 69 Instagram Final Code - Updated: 2018-06-22
Section 17: Web Scraping Best Practices
Lecture 70 Web Scraping Best Practices
Section 18: Bonus: Data Extraction with APIs
Lecture 71 Data Extraction with APIs (Free Tutorial)
Section 19: Bonus: Scrapy: Powerful Web Scraping and Crawling Framework in Python
Lecture 72 Coupon for "Scrapy: Powerful Web Scraping & Crawling with Python" Course
Those who want to learn how to use Python for web scraping and data extraction.


Homepage

https://www.udemy.com/course/web-scraping-with-python-beautifulsoup/