Exceptions are a crucial part of higher level languages, and although exceptions might be frustrating when they occur, they are your friend. The alternative to an exception is a panic — an error in execution that at best simply makes the program die and at worst can cause a blue screen of death. Exceptions, on the other hand, are tools of communication; they allow the program to tell you what, why, and how something went wrong and then they gracefully terminate your program without destruction. Learning to understand exceptions and throw some of your own is a crucial next step in programming mastery, particularly in Python.

In this short post, we will demonstrate the basics of exception handling in the Python programming language. We will look at how to read exception traceback reports and how to diagnose exceptions. Finally, we'll look at the use of context management to handle standard exception cases. Hopefully, this post will not only make you feel comfortable handling exceptions but also raising them yourself. Eventually, you'll be creating your own exception hierarchies in your code and throwing them at will!

Exceptions

Exceptions are a tool that programmers use to describe errors or faults that are fatal to the program; e.g. the program cannot or should not continue when an exception occurs. Exceptions can occur due to programming errors, user errors, or unexpected conditions like no internet access. Exceptions themselves are objects that contain information about what went wrong. Exceptions are usually defined by their type - which describes broadly the class of exception that occurred, and by a message - a string that says specifically what happened. Here are a few common exception types:

• SyntaxError: raised when the programmer has made a mistake typing Python code correctly.
• AttributeError: attempting to access an attribute on an object that does not exist
• KeyError: attempting to access a key in a dictionary that does not exist
• TypeError: raised when an argument to a function is not the right type (e.g. a str instead of int)
• ValueError: when an argument to a function is the right type but not in the right domain (e.g. an empty string)
• ImportError: raised when an import fails
• IOError: raised when Python cannot access a file correctly on disk

Exceptions are defined in a class hierarchy - e.g. every exception is an object whose class defines its type. The base class is the Exception object. All Exception objects are initialized with a message - a string that describes exactly what went wrong. Constructed objects can then be "raised" or "thrown" with the raise keyword:

raise Exception("Something bad happened!")

The reason the keyword is raise is because Python program execution creates what's called a stack as functions call other functions, which call other functions, etc. When a function (at the bottom of the stack) raises an Exception, it is propagated up through the call stack so that every function gets a chance to handle the exception (more on that later). If the exception reaches the top of the stack, then the program terminates and a traceback is printed to the console. The traceback is meant to help developers identify what went wrong in their code.

Let's take a look at a simple example that creates an artificial call stack:

def first(step, badstep=None):
    # Increment the step
    step += 1

    # Check if this is a bad step
    if badstep == step:
        raise ValueError("Failed after {} steps".format(step))

    # Call sub steps in order
    step = first_task_one(step, badstep)
    step = first_task_two(step, badstep)

    # Return the step that we're on
    return step


def first_task_one(step, badstep=None):
    # Increment the step
    step += 1

    # Check if this is a bad step
    if badstep == step:
        raise ValueError("Failed after {} steps".format(step))

    # Call sub steps in order
    step = first_task_one_subtask_one(step, badstep)

    # Return the step that we're on
    return step


def first_task_one_subtask_one(step, badstep=None):
    # Increment the step
    step += 1

    # Check if this is a bad step
    if badstep == step:
        raise ValueError("Failed after {} steps".format(step))

    # Return the step that we're on
    return step


def first_task_two(step, badstep=None):
    # Increment the step
    step += 1

    # Check if this is a bad step
    if badstep == step:
        raise ValueError("Failed after {} steps".format(step))

    # Return the step that we're on
    return step


def second(step, badstep=None):
    # Increment the step
    step += 1

    # Check if this is a bad step
    if badstep == step:
        raise ValueError("Failed after {} steps".format(step))

    # Call sub steps in order
    step = second_task_one(step, badstep)

    # Return the step that we're on
    return step


def second_task_one(step, badstep=None):
    # Increment the step
    step += 1

    # Check if this is a bad step
    if badstep == step:
        raise ValueError("Failed after {} steps".format(step))

    # Return the step that we're on
    return step


def main(badstep=None, **kwargs):
    """
    This function is the entry point of the program, it does
    work on the arguments by calling each step function, which
    in turn call substep functions.

    Passing in a number for badstep will cause whichever step
    that is to raise an exception.
    """

    step = 0 # count the steps

    # Execute each step one at a time.
    step = first(step, badstep)
    step = second(step, badstep)

    # Return a report
    return "Sucessfully executed {} steps".format(step)


if __name__ == "__main__":
    main()

The above example represents a fairly complex piece of code that has lots of functions that call lots of other functions starting from the main function. The primary question for exception handling is how do we know where our code went wrong? The answer is reported in the traceback which delineates exactly the functions through which the exception was raised. Let's trigger the exception and the traceback:

main(3)

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-5-e46a77400742> in <module>()
----> 1 main(3)


<ipython-input-4-03153844a5cc> in main(badstep, **kwargs)
     12
     13     # Execute each step one at a time.
---> 14     step = first(step, badstep)
     15     step = second(step, badstep)
     16


<ipython-input-4-03153844a5cc> in first(step, badstep)
     28
     29     # Call sub steps in order
---> 30     step = first_task_one(step, badstep)
     31     step = first_task_two(step, badstep)
     32


<ipython-input-4-03153844a5cc> in first_task_one(step, badstep)
     44
     45     # Call sub steps in order
---> 46     step = first_task_one_subtask_one(step, badstep)
     47
     48     # Return the step that we're on


<ipython-input-4-03153844a5cc> in first_task_one_subtask_one(step, badstep)
     56     # Check if this is a bad step
     57     if badstep == step:
---> 58         raise ValueError("Failed after {} steps".format(step))
     59
     60     # Return the step that we're on


ValueError: Failed after 3 steps

To read the traceback, start at the very bottom. As you can see it indicates the type of the exception, followed by a colon, and then the message that was passed to the exception constructor. This information is often enough to figure out what is going wrong. However, if we're unsure where the problem occurred, we can step back through the traceback in a bottom to top fashion.

In the example above, we can see that the exception was raised in the first_task_one_subtask_one function, which was called by first_task_one at line 46, which was called by first at line 30, which was called by main at line 14.

The first part of the traceback indicates the exact line of code and file where the exception was raised, as well as the name of the function in which it was raised. If you called main(3) this indicates that first_task_one_subtask_one is the function where the problem occurred. If you wrote this function, perhaps that is the place to change your code to handle the exception.

However, many times you're using third party libraries or Python standard library modules, meaning the location of the exception raised is not helpful, since you can't change that code. Therefore, you will continue up the call stack until you discover a file/function in the code you wrote. This will provide the surrounding context for why the error was raised, and you can use pdb or even just print statements to debug the variables around that line of code. Alternatively, you can handle the exception, which we'll discuss shortly.

Catching Exceptions

If a programming error caused the exception, the developer can simply change the code to make it correct. However, if the exception was created by bad user input, or by a bad environmental condition (e.g. the wireless is down), then you don't want to crash the program. Instead, you want to provide feedback and allow the user to fix the problem or try again. Therefore, in your code, you can catch exceptions at the place they occur using the following syntax:

try:
    # Code that may raise an exception
except AttributeError as e:
    # Code to handle the exception case
finally:
    # Code that must run even if there was an exception

What we're basically saying is try to run the code in the first block - hopefully, it works. If it raises an AttributeError save that exception in a variable called e (the as e syntax) then we will deal with that exception in the except block. Then finally run the code in the finally block even if an exception occurs. By specifying exactly the type of exception we want to catch (AttributeError in this case), we will not catch all exceptions, only those that are of the type specified, including subclasses. If we want to catch all exceptions, you can use one of the following syntaxes:

try:
    # Code that may raise an exception
except:
    # Except all exceptions

or

try:
    # Code that may raise an exception
except Exception as e:
    # Except all exceptions and capture in variable e

However, it is best practice to capture only the type of exception you expect to happen, because you could accidentally create the situation where you're capturing fatal errors but not handling them appropriately. If you want to catch multiple types of exceptions, you can pass them as a tuple:

try:
    # Code that may raise an exception
except (ValueError, TypeError) as e:
    # Catch only ValueError and TypeError exceptions.

Or you can pass multiple except blocks for different handling:

try:
    # Code that may raise an exception
except ValueError as e:
    # Handle ValueError
except TypeError as e:
    # Handle TypeError

If an exception is raised that is not an instance or a subclass of ValueError or TypeError then it continues to be propagated up the call stack. You can also catch custom exceptions that you define yourself. Here is a complete example of exception handling with real code that randomly generates exceptions:

import random


class RandomError(Exception):
    """
    A custom exception for this code block.
    """
    pass


def randomly_errors(p_error=0.5):
    if random.random() <= p_error:
        raise RandomError(
            "Error raised with {:0.2f} likelihood!".format(p_error)
        )

try:
    randomly_errors(0.5)
    print("No error occurred!")
except RandomError as e:
    print(e)
finally:
    print("This runs no matter what!")

This code snippet demonstrates a couple of things. First, you can define your own program-specific exceptions by defining a class that extends Exception. We have done so and created our own RandomError exception class. Next, we have a function that raises a RandomError with some likelihood which is an argument to the function. Then we have our exception handling block that calls the function and handles it.

Try adapting the code snippet to see how exception handling changes in the following cases:

• Change the likelihood of the error to see what happens
• except Exception instead of RandomError
• except TypeError instead of RandomError
• Call randomly_errors again inside of the except block
• Call randomly_errors again inside of the finally block

Make sure you run the code multiple times since the error does occur randomly!

LBYL vs. EAFP

In Python, exception handling is the standard way of dealing with runtime errors. You may wonder why you must use a try/except block to handle exceptions. Couldn't you simply do a check that the exception won't occur before it does? For example, consider the following code:

if key in mydict:
    val = mydict[key]
    # Do something with val
else:
    # Handle the fact that mydict doesn't have a required key.

This code checks if a key exists in the dictionary before using it, then uses an else block to handle the "exception." This is an alternative to the following code:

try:
    val = mydict[key]
    # Do something with val
except KeyError:
    # Handle the fact that mydict doesn't have a required key.

Both blocks of code are valid. In fact, they have names:

1. Look Before You Leap (LBYL)
2. Easier to Ask Forgiveness than Permission (EAFP)

For a variety of reasons, the second example (EAFP) is more pythonic — that is the preferred Python Syntax, commonly accepted by Python developers. For more on this, please see Alex Martelli's excellent PyCon 2016 talk, Exception and error handling in Python 2 and Python 3.

Context Managers

Python also provides a syntax for embedding common try/except/finally blocks in an easy to read format called context management. To motivate the example, consider the following code snippet:

try:
    fobj = open('path/to/file.txt', 'r')
    data = fobj.read()
except FileNotFoundError as e:
    print(e)
    print("Could not find the necessary file!")
finally:
    fobj.close()

This is a very common piece of code that opens a file and reads data from it. If the file doesn't exist, we simply alert the user that the required file is missing. No matter what, the file is closed. This is critical because if the file is not closed properly, it can be corrupted or not available to other parts of the program. Data loss is not acceptable, so we need to ensure that no matter what the file is closed when we're done with it. So we can do the following:

with open('path/to/file.txt', 'r') as fobj:
    data = fobj.read()

The with as syntax implements context management. On with, a function called the enter function is called to do some work on behalf of the user (in this case open a file), and the return of that function is saved in the fobj variable. When this block is complete, the finally is called by implementing an exit function. (Note that the except part is not implemented in this particular code). In this way, we can ensure that the try/finally for opening and reading files is correctly implemented.

Writing your own context managers is possible, but beyond the scope of this post (though I may write something on it shortly). Suffice it to say, you should always use the with/as syntax for opening files!

Conclusion

This has been a whirlwind tour of exception handling basics in Python. Hopefully, we covered enough to make exceptions feel less terrifying. Remember, exceptions are your friend: they are tools of communication that protect you and your users from critical damage. By understanding how to read the syntax of exceptions we hope that they start to feel more friendly and less frustrating, they are a map to guide you to solutions, not impediments to success!

This is only the beginning of exception management though — handling exceptions that are thrown by the Python interpreter and maybe throwing a few of your own. Exceptions are such a critical tool in programming that most large programs have their own exception hierarchies and throw and handle their own exceptions. If you use larger libraries like Scikit-Learn or Numpy, you've probably already noticed this. I hope you'll soon agree that exceptions are critical to ensuring that things go smoothly in programs written in Python!

Helpful Links

• Exception and error handling in Python 2 and Python 3
• PEP 348: Exception Reorganization

Acknowledgment

Thank you so much to Nicole Donnelly for reviewing and editing this post!

District Data Labs provides data science consulting and corporate training services. We work with companies and teams of all sizes, helping them make their operations more data-driven and enhancing the analytical abilities of their employees. Interested in working with us? Let us know!