Scraping on Matt Thornton

Basic web scraping with Python: Episode 3!

Mon, 13 Apr 2020 11:32:55 +0000

This is the third edition of this post. It was originally an intro to web scraping with Python (in Python 2) using the Requests library. It was then updated to cover some extra topics and also update for Python 3.

The scenario is to download the back catalogue of the excellent MagPi magazine which is published monthly and the PDF is available for free. More info on the background is in the original post.

However, since the original post a fair bit has changed: the MagPi website was updated so the scraping broke, Python has moved on and I found that despite downloading the issues, having them on a Pi meant I never actually read them because I forgot they were there!

Basic web scraping - Part 2: thoughts on debugging and Python3

Mon, 13 May 2019 16:18:14 +0000

Update: 2020-04-13. The main article has been overhauled and republished here! Includes Python3 updates, Dropbox uploading and a code repo.

I’m leaving the below in place for info and posterity and for misc useful tidbits.

A while ago, I posted a basic tutorial on how to do some simple web scraping in Python using requests and beautifulsoup.

It obviously got found in Google because the hits were good. There were however, a couple of comments highlighting issues with it - Brett and Sean saying issues with “TypeError: ‘NoneType’ object is not callable”

Basic web scraping with a Raspberry Pi, Python and Requests

Sun, 06 Jan 2019 23:11:29 +0000

Update: 2020-04-13. Article has been overhauled and republished here! Includes Python3 updates, Dropbox uploading and a code repo.

I’m leaving the below in place for background, info and posterity and for misc useful tidbits.

Update: 2019-05-13 - this has been edited to fix a little bug in the code transcript. There is also a follow-up article on working with Python3 here.

_Update: 2019-12-16 - looks like MagPi have amended their page so the code as below won’t work directly. I’ve left it there for interest but see one of the comments which has amended code. Thanks to Dave for pointing this out!
_