Python Generators

This is a detailed tutorial on Python Generators. Learn about yield statements, generator expressions, and the send(), throw(), and close() methods.

What are Python Generators?

These are special kind of functions that instead of returning one value, at last, keeps the ability to return multiple values as time passes and function processing continues. This concept is also known as lazy returning as the function keeps on returning the values as it further goes through the function code.

Let’s say there’s a function that is reading a very large text file. The file contains so many lines that even while ready, it is not possible to load the entire number of file lines at once in your computer’s memory. So, what if you can start the reading process of the file and keep on getting a limited number of lines, every few time units? For example, while the reading process continues, it keeps on printing every 1000 lines to the console every 2 seconds. This way, the function keeps on returning something after every 2 seconds until the reading process is completed for the entire file. Hence, such kind of function is a generator function that generates and returns a number of values with time.

There are multiple ways to create generator objects. The first and most popular way is to use the statement yield instead of the return statement in Python Functions.

Example 1. Simple Generator Function using yield statement

Given below is an example of the Generator function. It is also known as a generator object. The generator threeDates() returns three dates, namely, today, yesterday, and tomorrow in the DD/MM/YYYY format.

from datetime import date, timedelta

def threeDates():
    today = date.today()
    yield "Today: " + today.strftime("%d/%m/%Y")
    
    yesterday = today - timedelta(1)
    yield "Yesterday: " + yesterday.strftime("%d/%m/%Y")
    
    tomorrow = today + timedelta(1)
    yield "tomorrow: " + tomorrow.strftime("%d/%m/%Y")
    
print(threeDates())
print(type(threeDates()))

for date in threeDates():
    print(date)

Output.

<generator object threeDates at 0x7f83f7f44a98>
<class 'generator'>
Today: 30/06/2020
Yesterday: 29/06/2020
tomorrow: 01/07/2020

The code is clear. Here we’ve used the datetime module to fetch the current date and then by using the timedelta object we’ve also calculated the dates for yesterday and tomorrow.

We’ve used the yield statement three times to return these three different dates.

Now when you will call the function, it will not return the three dates, rather it will say, it’s a generator object at some memory location. You can also see this in the output of the above code and also we’ve used the type() function to verify the data type of the generator function object.

A generator object is an iterable object. Therefore, to actually get or yield the values you must iterate over the generator object. In the above code, we’ve used the Python For loop for the purpose of yielding and printing the values from the generator function. You can also use the next() function to fetch one yield value at a time and it is also illustrated in the following example.

Example 2. Infinite Yield Values and the usage of next() Function

For the sake of better understanding, we’ve illustrated another logic here. A Generator function may return an infinite number of values. In the following example, we’ve written a function that yields the multiples of a particular number that can be passed as the argument to the generator function.

def multiplesOf(number):
    count = 0
    while True:
        yield number * count
        count += 1
        
generator1 = multiplesOf(5)
#First Five Multiples of 5, starting from 0
print(next(generator1))
print(next(generator1))
print(next(generator1))
print(next(generator1))
print(next(generator1))

generator2 = multiplesOf(15)
#First 10 multiples of 15
for i in range(0, 10):
    print(next(generator2))

Output.

0
5
10
15
20
0
15
30
45
60
75
90
105
120
135

Note. Do not apply the next() function directly on the generator function, otherwise, you will get the first yield value always. You have to first create an instance of the generator object and then have to refer it to some variable, then you can pass the variable to the next() function to get the values in a sequence.

You can always use the next() function instead of the for loop to fetch the yield values whenever you are not sure about the exact number of values that the given generator function is yielding.

In case, there are no more yields left and you’re still applying the next() function to fetch the next yield value, it will raise the StopIteration Error.

As generators are capable of working with infinite sequences without much memory allocation, you can also use them as an alternative to traditional loop (Infinite For Loop or Infinite While Loop) logic.

Python Generators vs Normal Python Functions

Although you might have already pointed out the differences between the Python Generators and the normal functions, here we have described the differences in detail.

• Generators make use of the yield statements, usually a number of times. On the other hand, the normal functions may use one or more return statements but only once the return statement is executed that come first in execution control.
• Normal functions return the value specified in the return statement at once after completing the execution of the function code that comes before the return statement in order. On the other hand, the generator functions return the generator object and it does not start the execution right away.
• Generator objects are iterable and the functions like next() can be used to fetch the yield values in the sequence. This is because the methods like __iter__() and __next__() are implicitly implemented by the generators.
• Generators remember the state of the function’s local variables while the functions the supposed to forget everything after the return statement is executed.
• In generators, when the iteration is complete or there are no more values to yield, StopIteration error is raised automatically.

Python Generator Expressions

You can also create python generators using simple expressions. It is one of the easiest ways of creating generators without actually defining a function. This concept is quite similar to the concept of lambda functions.

The syntax to create generators using expressions is almost the same as the syntax of Python List comprehension. The only major difference that I observe here is that, in the case of the list comprehension, we make use of the square brackets while in the case of generator expressions, round parenthesis is used.

The technical difference between the list comprehension and the generator expressions is that when you define a list comprehension, the entire list items are generated at once after completing all the iterations in one go. On the other hand, in generator expressions, a single item is created in one go. This is also one of the reasons generator functions are considered more efficient as they retrieve or process the value as per the requirements or when demanded. Therefore, these are also memory efficient than the comprehension way outs.

The following code illustrates how you can a generator object using generator expressions and also shows the syntactical difference from the List Comprehension method.

#List Comprehension Method
#List of squares of the numbers from 0 to 10
aList = [x*x for x in range(0, 11)]

#Generator Expressions Method
#Generator of squares of the numbers from 0 to 10
aGenerator = (x*x for x in range(0, 11))

print(aList)
print(type(aList))

print(aGenerator)
print(type(aGenerator))

Output.

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
<class 'list'>
<generator object <genexpr> at 0x7f49545cdaf0>
<class 'generator'>

As you can see in the output also, the full list is generated with list comprehension while the generator expression has only created the generator object and the yield values will only be calculated or processed when required.

Advantages of Using Python Generators

Python Generators are quite useful for different kinds of purposes. Here I have listed all those things that you should know about Generators that put them up in the list of your choices when you want to achieve some particular goals using Python.

Memory Efficiency

Let’s say you want a normal function in python to first generate a sequence and then return it. The sequence contains a lot of items, let’s say, 10,000 items. There’s a logic written inside the function that generates these items. Your function first has to do all the processing to generate these items and then have to store the sequence so generated in the memory to further return it. Hence, it consumes a lot of memory at once.

Now, we write the same logic but now inside a generated function. Let’s say we have used the yield keyword to return one item at a time in the sequence. This way, only the generator object will be created and no memory will be required to store the 10,000 list items. Any item whenever will be required will be generated by using the generator object.

Hence, generator functions are memory efficient.

Infinite Sequence Possibilities

You simply can’t generate infinite sequences and return them with normal functions. Even if you will try, the function will go into an infinite loop stage and will never return. But there’s no such issue with the Generator Functions. You can specify the logic to create the infinite sequence in the generator function and can yield one value at a time.

We’ve already discussed this concept in Example 2 where we have created a generated function that can generate an infinite number of multiples of a particular number. Another example given below generates a sequence of odd numbers, starting from 1 up to infinity.

def oddNumbers():
    count = 1
    while True:
        yield count
        count += 1
        
print(oddNumbers())
gen = oddNumbers()
#Printing First 5 odd numbers
print(next(gen))
print(next(gen))
print(next(gen))
print(next(gen))
print(next(gen))

Output.

<generator object oddNumbers at 0x7fa70eee45c8>
1
2
3
4
5

Pipelining Generators

In case you want to do multiple operations and that too on multiple series or sequences. Although it seems complex, using generators, the process can be pipelined. This can be better explained with the help of an example.

Let’s say we have a sequence of odd numbers from 1 to 1099. We want to find out the sum of the cubes of the odd numbers from 1 to 1099. The code makes use of the generators for the purpose is written below.

def oddNumbers(last):
    count = 1
    while True:
        yield count
        count += 1
        if count > last:
            break

def cube(numbers):
    for number in numbers:
        yield number**3

print(sum(cube(oddNumbers(1099))))

Output.

365359802500

Reduced Code Length & Easy Implementation

In the end, the Python Generators are easy to implement and reduce the overall length of the code. It automatically implements the class methods like __init__(), __iter__() and __next__() that are required to generate the sequences with normal classes.

I have defined one class and one generator function in the following code. Both generate a sequence containing cubes of numbers from 1 to 100.

class CubeClass:
    def __init__(this, max=0):
        this.count = 0
        this.max = max

    def __iter__(this):
        return this

    def __next__(this):
        if this.count >= this.max:
            raise StopIteration

        result = 3 ** this.count
        this.count += 1
        return result
        
def cubeGenerator(max=0):
    count = 0
    while count < max:
        yield 3 ** count
        count += 1
        
print(CubeClass(10))
print(cubeGenerator(10))

for cube in CubeClass(10):
    print(cube)
    
for cube in cubeGenerator(10):
    print(cube)

As you can observe CubeClass code is quite lengthy and complex while the code of cubeGenerator is simple and can be written in just a few lines.

Output.

<__main__.CubeClass object at 0x7fd0b5ab6438>
<generator object cubeGenerator at 0x7fd0bc1d4ca8>
1
3
9
27
81
243
729
2187
6561
19683
1
3
9
27
81
243
729
2187
6561
19683

Generator Object Methods

There are three advanced methods that you can use with Python Generator objects. These are listed below.

• send()
• throw()
• close()

Let’s understand each of these methods with the help of illustrative examples.

send() Method

The generator.send() method is used to send a given value to the generator that can be used within the generator function body for the calculation of the next upcoming sequence item.

Have a look at the following example to understand the usage of the Python Generator send() method.

#Generates cube of the number sent using send() method
def cube():
    while True:
        number = yield
        yield number ** 3
        
gen = cube()
next(gen)
gen.send(3)

Output.

27

The above example is just to illustrate how you can make use of the send() method to send the yield values to the generator function logic. It is not a suitable practical coding example.

throw() Method

This is another useful method that you can use to run the exceptionally handled generator function logic. In other words. it can be used to raise a given exception manually within the generator function logic. Therefore, you can throw a particular exception in case you wish to execute the code written within the except block. Understand its usage with the help of the following example.

def getNumbers():
    count = 0
    try:
        while True:
            yield count
            count += 1
    except:
        while ValueError:
            while True:
                yield count ** 3
                count += 1
             
gen = getNumbers()   
#Getting simple numbers
print(next(gen))
print(next(gen))
print(next(gen))

#Getting cubes
gen.throw(ValueError)
print(next(gen))
print(next(gen))
print(next(gen))

In the generator function written in the above code, we’ve written the logic to return simple numbers in the try block and the logic to return cubes of the sequence numbers in the except block. Initially, we used the generator object and the next() function to print the first three sequence numbers and then by throwing the ValueError, we started the sequence of the except block logic, i.e. yield the cubes of the sequence numbers.

Output.

0
1
2
27
64
125

This method is quite useful when you want multiple operations to be performed on a running sequence with different types of Exceptions.

close() Method

The Python Generator close() method is used to stop the further iterations of a given sequence. The following example explains the usage of this method.

#Generates infinite numbers sequence, starting from 1
def numbers():
    count = 1
    while True:
        yield count
        count += 1

gen = numbers()
#Starting Sequence
print(next(gen))
print(next(gen))
print(next(gen))

#Stopping Further Sequence
gen.close()

#Trying to fetch the next seqeunce numbers
#Will raise StopIteration Error
print(next(gen))

As you can see in the output, the last line, line no. 19 raises the StopIteration error because we have already stopped the further iteration using the close() method applied on the generator object.

Output.

1
2
3
Traceback (most recent call last):
  File "<stdin>", line 19, in <module>
    print(next(gen))
StopIteration

I hope you found this guide useful. If so, do share it with others who are willing to learn Python and other programming languages. If you have any questions related to this article, feel free to ask us in the comments section.