Chris Oliver Chris Oliver

Python downloading and resuming with text progress using urllib

Mar 22, 2010

So recently I have been working on download scripts for Keryx once again. Building a script to do this seemed over my head at first. The plethora of classes and functions was a little daunting at first, but realizing how to use them after reading some examples wasn’t so bad. Urlgrabber seems to do the trick, but I have run into some issues. Why not rewrite it from scratch and learn something along the way?

I had two goals in mind. #1 was providing resume support. This is the most important thing we have been missing in Keryx for a while. We were able to do it with urlgrabber but I did not feel like wading through all the code to learn how it worked and make changes. #2 was file progess. This proved to be a problem with urlgrabber such that some of the output would overlap or be skewed improperly.

In the end, this is what I came up with. It’s essentially a modified version of this code example:

I cleaned up some of the variable names to make things a little clearer and even though it is still quite ugly. Beautification is for another night though.


Continue reading

Nightly Automatic Backups over SSH with Rsync An actually decent Python DBus Tutorial