Python custom exception best practices

• • • Catch More Specific Exceptions First
    • Don’t Catch Exception
    • Definitely don’t catch BaseException

Catch More Specific Exceptions First

Remember, your except handlers are evaluated in order, so be sure to put more specific exceptions first. For example:

>>> try:
...     my_value = 3.14 / 0
... except ArithmeticError:
...     print("We had a general math error")
... except ZeroDivisionEror:
...     print("We had a divide-by-zero error")
...
We had a general math error

When we tried to divide by zero, we inadvertently raised a ZeroDivisionError. However, because ZeroDivisionError is a subclass of ArithmeticError, and except ArithemticError came first, the information about our specific error was swallowed by the except ArithemticError handler, and we lost more detailed information about our error.

Don’t Catch Exception

It’s bad form to catch the general Exception class. This will catch every type of exception that subclasses the Exception class, which is almost all of them. You may have errors that you don’t care about, and don’t affect the operation of your program, or maybe you’re dealing with a flaky API and want to swallow errors and retry. By catching Exception, you run the risk of hitting an unexpected exception that your program actually can’t recover from, or worse, swallowing an important exception without properly logging it - a huge headache when trying to debug programs that are failing in weird ways.

Definitely don’t catch BaseException

Catching BaseException is a really bad idea, because you’ll swallow every type of Exception, including KeyboardInterrupt, the exception that causes your program to exit when you send a SIGINT (Ctrl-C). Don’t do it.

A little Python knowledge everyday!

If you are doing Python codings, you must have used exceptions, because it is everywhere in the language. For example, when you press Ctrl + c in your terminal, it will raise an KeyboardInterrupt exception. If you divide 0 in your program, you will encounter ZeroDivisionError: division by zero exception.

In Python, exception handling has two parts: “catch” and “throw”. “Catch” refers to the use of try ... except statement and “Throw”refers to the use of raise statement.

Precise Exception Catch

Sometimes people think, “exceptions are a bad thing, and a good program should catch all exceptions and make everything run smoothly”. The code written with the idea will usually contains a large section of ambiguous exception capture logic.

For example:

import requests
import re
def save_page_title(url, filename):
    """Save url title and save to file
        :returns: True for success, otherwise return False
    """
    try:
        resp = requests.get(url)
        obj = re.search(r'<title>(.*)</title>', resp.text)
        if not obj:
            print('save failed: title tag not found in page content')
            return False
        title = obj.grop(1)
        with open(filename, 'w') as fp:
            fp.write(title)
            return True
    except Exception:
        print(f'save failed: unable to save title of {url} to {filename}')
        return False
def main():
    save_page_title('https://google.com', 'url_title.txt')
if __name__ == '__main__':
    main()

The save_page_title functions do several things. It first obtains the content of the web page through the network, then uses the regular matching to get the title, and finally writes the title in the local file.

And here are two steps that are prone to error: network requests and local file operations. So in the code, we wrap these steps in a big block of try ... except statements .

It seems simple and easy to understand, right?

However if you try to run the script above. You will find that the above code cannot be executed successfully. And you will also find that no matter how you modify the value of the URL and the target file, the program will still report the error “save failed: unable to…” . why?

The problem lies in this huge block of try ... except . If you check this code very carefully, you will find that I purposely mistyped obj.group(1) with a missing u .

But due to the ambiguous exception catch that the error that should have been thrown due to the wrong method name AttibuteError was swallowed up. Thus adding unnecessary trouble to our debug process.

The purpose of exception catching is not to catch as many exceptions as possible. It try to catch the most accurate exception. Then such a problem will not happen at all, and accurate capture includes:

• Always catch only those blocks of statements that might throw an exception
• Try to catch only precise exception types, not vague onesException

Therefore, with the new purpose, let’s modify out code to:

import re
import requestsfrom requests.exceptions import RequestException
def save_page_title(url, filename):
    try:
        resp = requests.get(url)
    except RequestException as e:
        print(f'save failed: unable to get page content: {e}')
        return False
    obj = re.search(r'<title>(.*)</title>', resp.text)
    if not obj:
        print('save failed: title tag not found in page content')
        return False
    title = obj.grop(1)



 try:
 with open(filename, 'w') as fp:
 fp.write(title)
 except IOError as e:
 print(f'save failed: unable to write to file {filename}: {e}')
 return False
 else:
 return Truedef main():
 save_page_title('https://google.com', 'url_title.txt')
if __name__ == '__main__':
 main()

This time, it will throw the right exception:

AttributeError: 're.Match' object has no attribute 'grop'. Did you mean: 'group'?

Only Throw Exceptions for Current Level

Let’s take a look at the following code:

def process_image(...):
    try:
        image = Image.open(fp)
    except Exception:
        raise error_codes.INVALID_IMAGE_UPLOADED
    ... ...

In the above code snippet, when developer first time wrote this function, it was under the use cause of processing user uploaded images, therefore when an image can not be opened, it throws an INVALID_IMAGE_UPLOADED error. First glance, nothing wrong with this, right?

What if couple of month later, we need to reuse this process_image function in a different context? Such as process existing images? In this situation, if an image is damaged and cannot open, this process_image still throws INVALID_IMAGE_UPLOADED error, which will cause confusions. What should you do? Write a copy of process_image for the purpose of only throw a different error?

The right way to fix this is to modify the original process_image function so it only throws image processing level exceptions, such as:

class ImageOpenError(Exception):
    passdef process_image(...):
    try:
        image = Image.open(fp)
    except Exception:
        raise ImageOpenError(e)
    ... ...

Then in the upper level, you can catch and re-throw a different but more meaningful error:

def process_existing_images():
    try:
        process_image(fp)
    except ImageOpenError:
        raise BROKEN_IMAGE

In addition to avoiding throwing exceptions higher than the current abstraction level, we should also avoid leaking exceptions lower than the current abstraction level.

If you’ve used the requestsmodule , you may have discovered that the exception it throws when a page fails is not urllib3the original exception of the module it uses under the hood, but an exception that is requests.exceptionswrapped once.

>>> try:
...     requests.get('https://www.invalid-host-foo.com')
... except Exception as e:
...     print(type(e))
...
<class 'requests.exceptions.ConnectionError'>

Don’t Overwhelming Exceptions

Earlier we mentioned that exception capture should be precise and the abstraction level should be consistent. But in the real world, if you strictly follow these processes, you are likely to run into another problem: too much exception handling logic that disrupts the core logic of the code . The specific performance is that the code is filled with a large number of try, except, and raise statements, making the core logic difficult to identify. Like the following code:

def upload_profile_pic(request):
    try:
        profile_file = request.FILES['profile']
    except KeyError:
        raise error_codes.PROFILE_FILE_NOT_PROVIDED
    try:
        resized_profile_file = resize_profile(profile_file)
    except FileTooLargeError as e:
        raise error_codes.PROFILE_FILE_TOO_LARGE
    except ResizeProfileError as e:
        raise error_codes.PROFILE_FILE_INVALID
    try:
        request.user.profile = resized_profile_file
        request.user.save()
    except Exception:
        raise error_codes.INTERNAL_SERVER_ERROR
    return HttpResponse({})

This is a view function that handles user uploads of profiles. Three things are done in this function, and exceptions are caught for each of them. If an exception occurs while doing something, return a user-friendly error to the front end.

Although this processing flow is reasonable, it is obvious that the exception handling logic in the code is a bit “overwhelming”. At first glance, it is all code indentation, and it is difficult to extract the core logic of the code.

So, how can we use context managers to improve our exception handling flow? Let’s go straight to the code.

class raise_api_error:
    """captures specified exception and raise ApiErrorCode instead    :raises: AttributeError if code_name is not valid
    """
    def __init__(self, captures, code_name):
        self.captures = captures
        self.code = getattr(error_codes, code_name)    def __enter__(self):
        return self    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type is None:
            return False        if exc_type == self.captures:
            raise self.code from exc_val
        return False

the above code, we define raise_api_error, which does nothing when entering the context. However, when exiting the context, it will determine whether self.captures captured anything and if so, APIErrorCode will replace it with the exception class.

After using this context manager, the whole function can be made clearer and more concise:

def upload_profile_pic(request):
    with raise_api_error(KeyError, 'PROFILE_FILE_NOT_PROVIDED'):
        profile_file = request.FILES['profile']    with raise_api_error(ResizeProfileError, 'PROFILE_FILE_INVALID'), raise_api_error(FileTooLargeError, 'PROFILE_FILE_TOO_LARGE'):
        resized_profile_file = resize_profile(profile_file)    with raise_api_error(Exception, 'INTERNAL_SERVER_ERROR'):
        request.user.profile = resized_profile_file
        request.user.save()
    return HttpResponse({})

Conclusion

In this article, I share three tips related to exception handling:

• Only catch statements that may throw exceptions to avoid ambiguous catch logic
• Keep the abstract consistency of the module exception class, and wrap the underlying exception class if necessary
• Use Context Managers to simplify repetitive exception handling logic