Webscraper For Mac
WebScraper 4.7.2 - Scan and output website data as CSV or JSON. Download the latest versions of the best Mac apps at safe and trusted MacUpdate.
Your First Web Scraper Once you start web scraping, you start to appreciate all the little things that browsers do for us. The Web, without a layer of HTML formatting, CSS styling, JavaScript execution, and image rendering, can look a little intimidating at first, but in this chapter, as well as the next one, we’ll cover how to format and interpret data without the help of a browser. This chapter will start with the basics of sending a GET request to a web server for a specific page, reading the HTML output from that page, and doing some simple data extraction in order to isolate the content that we are looking for. Urllib or urllib2? If you’ve used the urllib2 library in Python 2.x, you might have noticed that things have changed somewhat between urllib2 and urllib. In Python 3.x, urllib2 was renamed urllib and was split into several submodules: urllib.request, urllib.parse, and urllib.error. Although function names mostly remain the same, you might want to note which functions have moved to submodules when using the new urllib.
Urllib is a standard Python library (meaning you don’t have to install anything extra to run this example) and contains functions for requesting data across the web, handling cookies, and even changing metadata such as headers and your user agent. We will be using urllib extensively throughout the book, so we recommend you read the Python documentation for the library ( urlopen is used to open a remote object across a network and read it. Because it is a fairly generic library (it can read HTML files, image files, or any other file stream with ease), we will be using it quite frequently throughout the book. An Introduction to BeautifulSoup “Beautiful Soup, so rich and green, Waiting in a hot tureen! Who for such dainties would not stoop? Soup of the evening, beautiful Soup!” The BeautifulSoup library was named after a Lewis Carroll poem of the same name in Alice’s Adventures in Wonderland.
In the story, this poem is sung by a character called the Mock Turtle (itself a pun on the popular Victorian dish Mock Turtle Soup made not of turtle but of cow). Like its Wonderland namesake, BeautifulSoup tries to make sense of the nonsensical; it helps format and organize the messy web by fixing bad HTML and presenting us with easily-traversible Python objects representing XML structures. Brother dcp-330c printer drivers for mac. Installing BeautifulSoup Because the BeautifulSoup library is not a default Python library, it must be installed. We will be using the BeautifulSoup 4 library (also known as BS4) throughout this book. The complete instructions for installing BeautifulSoup 4 can be found at; however, the basic method for Linux is: $ sudo apt - get install python - bs4 and for Macs: $ sudo easyinstall pip This installs the Python package manager pip. Then run the following: $ pip install beautifulsoup4 to install the library.
Again, note that if you have both Python 2.x and 3.x installed on your machine, you might need to call python3 explicitly: $ python3 myScript. Py Make sure to also use this when installing packages, or the packages might be installed under Python 2.x, but not Python 3.x: $ sudo python3 setup. Py install If using pip, you can also call pip3 to install the Python 3.x versions of packages: $ pip3 install beautifulsoup4 Installing packages in Windows is nearly identical to the process for the Mac and Linux. Download the most recent BeautifulSoup 4 release from the download URL above, navigate to the directory you unzipped it to, and run: python setup.
Py install And that’s it! BeautifulSoup will now be recognized as a Python library on your machine. You can test this out by opening a Python terminal and importing it: $ python from bs4 import BeautifulSoup The import should complete without errors. In addition, there is an.exe, so you can easily install and manage packages: pip install beautifulsoup4. Keeping Libraries Straight with Virtual Environments If you intend to work on multiple Python projects or you need a way to easily bundle projects with all associated libraries, or you’re worried about potential conflicts between installed libraries, you can install a Python virtual environment to keep everything separated and easy to manage. When you install a Python library without a virtual environment, you are installing it globally. This usually requires that you be an administrator, or run as root, and that Python library exists for every user and every project on the machine.
Fortunately, creating a virtual environment is easy: $ virtualenv scrapingEnv This creates a new environment called scrapingEnv, which you must activate in order to use: $ cd scrapingEnv/ $ source bin/activate After you have activated the environment, you will see that environment’s name in your command line prompt, reminding you that you’re currently working with it. Any libraries you install or scripts you run will be under that virtual environment only. Running BeautifulSoup The most commonly used object in the BeautifulSoup library is, appropriately, the BeautifulSoup object. Let’s take a look at it in action, modifying the example found in the beginning of this chapter: from urllib.request import urlopen from bs4 import BeautifulSoup html = urlopen ( ') bsObj = BeautifulSoup ( html. Read ); print ( bsObj. H1 ) The output is: An Interesting Title As in the example before, we are importing the urlopen library and calling html.read in order to get the HTML content of the page. This HTML content is then transformed into a BeautifulSoup object, with the following structure:.
html →. head → A Useful Page. title → A Useful Page. body → An Int.Lorem ip. h1 → An Interesting Title. div → Lorem Ipsum dolor. Note that the tag that we extracted from the page was nested two layers deep into our BeautifulSoup object structure (html → body → h1).
However, when we actually fetched it from the object, we called the h1 tag directly: bsObj. H1 In fact, any of the following function calls would produce the same output: bsObj. H1 We hope this small taste of BeautifulSoup has given you an idea of the power and simplicity of this library. Virtually any information can be extracted from any HTML (or XML) file, as long as it has some identifying tag surrounding it, or near it.
In, we’ll delve more deeply into some more-complex BeautifulSoup function calls, as well as take a look at regular expressions and how they can be used with BeautifulSoup in order to extract information from websites. Connecting Reliably The web is messy. Data is poorly formatted, websites go down, and closing tags go missing. One of the most frustrating experiences in web scraping is to go to sleep with a scraper running, dreaming of all the data you’ll have in your database the next day—only to find out that the scraper hit an error on some unexpected data format and stopped execution shortly after you stopped looking at the screen. In situations like these, you might be tempted to curse the name of the developer who created the website (and the oddly formatted data), but the person you should really be kicking is yourself, for not anticipating the exception in the first place! Let’s take a look at the first line of our scraper, after the import statements, and figure out how to handle any exceptions this might throw: html = urlopen ( ') There are two main things that can go wrong in this line:.
The page is not found on the server (or there was some error in retrieving it). The server is not found In the first situation, an HTTP error will be returned. This HTTP error may be “404 Page Not Found,” “500 Internal Server Error,” etc. In all of these cases, the urlopen function will throw the generic exception “HTTPError” We can handle this exception in the following way: try: html = urlopen ( ') except HTTPError as e: print ( e ) #return null, break, or do some other 'Plan B' else: #program continues. Note: If you return or break in the #exception catch, you do not need to use the 'else' statement If an HTTP error code is returned, the program now prints the error, and does not execute the rest of the program under the else statement. If the server is not found at all (if, say, was down, or the URL was mistyped), urlopen returns a None object. This object is analogous to null in other programming languages.
We can add a check to see if the returned html is None: if html is None: print ( 'URL is not found' ) else: #program continues Of course, if the page is retrieved successfully from the server, there is still the issue of the content on the page not quite being what we expected. Every time you access a tag in a BeautifulSoup object, it’s smart to add a check to make sure the tag actually exists. If you attempt to access a tag that does not exist, BeautifulSoup will return a None object. The problem is, attempting to access a tag on a None object itself will result in an AttributeError being thrown. The following line (where nonExistentTag is a made-up tag, not the name of a real BeautifulSoup function): print ( bsObj. NonExistentTag ) returns a None object.
Web Scraper For Mac
This object is perfectly reasonable to handle and check for. The trouble comes if you don’t check for it, but instead go on and try to call some other function on the None object, as illustrated in the following: print ( bsObj. SomeTag ) which returns the exception: AttributeError: 'NoneType' object has no attribute 'someTag' So how can we guard against these two situations? The easiest way is to explicitly check for both situations: try: badContent = bsObj. AnotherTag except AttributeError as e: print ( 'Tag was not found' ) else: if badContent None: print ( 'Tag was not found' ) else: print ( badContent ) This checking and handling of every error does seem laborious at first, but it’s easy to add a little reorganization to this code to make it less difficult to write (and, more importantly, much less difficult to read).
This code, for example, is our same scraper written in a slightly different way: from urllib.request import urlopen from urllib.error import HTTPError from bs4 import BeautifulSoup def getTitle ( url ): try: html = urlopen ( url ) except HTTPError as e: return None try: bsObj = BeautifulSoup ( html. Read ) title = bsObj. H1 except AttributeError as e: return None return title title = getTitle ( ') if title None: print ( 'Title could not be found' ) else: print ( title ) In this example, we’re creating a function getTitle, which returns either the title of the page, or a None object if there was some problem with retrieving it. Inside getTitle, we check for an HTTPError, as in the previous example, and also encapsulate two of the BeautifulSoup lines inside one try statement. An AttributeError might be thrown from either of these lines (if the server did not exist, html would be a None object, and html.read would throw an AttributeError).
We could, in fact, encompass as many lines as we wanted inside one try statement, or call another function entirely, which can throw an AttributeError at any point. When writing scrapers, it’s important to think about the overall pattern of your code in order to handle exceptions and make it readable at the same time. You’ll also likely want to heavily reuse code. Having generic functions such as getSiteHTML and getTitle (complete with thorough exception handling) makes it easy to quickly—and reliably—scrape the web. With Safari, you learn the way you learn best.
Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.