Hi, I'm Chris

This blog is about programming, linux and other stuff.
Please don’t hesitate to contact me if you have any questions or suggestions for improvements!
contact

Fun with URLError, socket.timeout and python's urlopen...

Think about the following scenario: you want to grab some shit from the web with python’s urlopen and don’t care about timeouts. What do you do? Stupid question, I know. You will put the urlopen call in a try-except-block which will look similar to this piece of code:

from urllib.requests import urlopen, URLError
...
try:
    resp = urlopen('www.devmartin.com')
except URLError:
    #write some logs or just pass
    pass
...

Will this try-except thing catch all timeouts? Fuck no, it won’t! And I had to learn it the hard way. I had some threads running the above code and sometimes I got a timeout error. socket.timeout timed out to be exact. Some time passed until I found this blog post.

Solution: Sometimes you don’t get a URLError but a socket.timeout instead… Therefore, you have to add the socket.timeout to your exception:

from urllib.requests import urlopen, URLError
import socket
...
try:
    resp = urlopen('www.devmartin.com')
except URLError:
    #write some logs or just pass
    pass
except socket.timeout:
    #write more crap to the logs
    pass
...

Have fun.

comments powered by Disqus