Lecture 29: Priority Queues and Heapsort

Introducing the priority queue structure; using a priority queue to conveniently implement a sorting algorithm

Motivation

In real life, we often encounter scenarios that involve queuing, such as standing in line at a grocery store or at the movies, or waiting for a haircut. In these “first come, first served” situations, the first person to arrive is the first person served. We can represent this kind of situation with a queue, where we add items to the back of the queue, and remove them from the front.

However, there are other scenarios where the first person to arrive is not the first person served: when boarding an airplane, even if you are the first on line, the first-class passengers will board before you. Or consider patients in an emergency room: if someone arrives early with a comparatively minor injury, and another patient arrives with a more serious problem, the second patient will be treated first. Here, the scenario is more of a “first come, most-important served” situation. How might we represent this data?

29.1 Priority Queues

In the rest of this lecture, we are going to make the following simplification: rather than focusing on adding an item paired with a priority, we are going to instead just consider adding a given priority. (We can add the item data as well if we just construct a custom IComparator object that ranks items based on priorities.) Further, we will choose to represent priorities simply as integers, though any type that can be sorted by a comparator will suffice.

Exercise

After completing this lecture, try adapting the data types here to store arbitrary data with priorities, instead of just priorities.

29.2 Using lists to implement priority queues: too slow

We might imagine that since a priority queue is “like a queue, but with priorities”, we should be able to implement it using a simple list or Deque<Integer>. We can indeed manage this: we just need to maintain our list in decreasing sorted order. Then the maximum value (i.e., the highest-priority item) will be the first item of the list, and so is readily available in constant (i.e. \(O(1)\)) time. But adding an item to a sorted list is expensive: in the worst case, we constantly add a low-priority item, and so for a priority queue already containing \(n\) elements, we would have to walk the entire list to find the right place to insert the item, which takes linear (i.e. \(O(n)\)) time.

Can we do better?

29.3 Choosing a better data structure for the task

One compelling possibility is to use a binary tree: accessing the root of the tree takes constant time, and accessing any item in the tree takes \(O(\mathit{depth\ of\ tree})\). But using an arbitrary binary tree is not enough: technically, the following tree is a binary tree:

Every level of the tree (except the first) contains two items, so this kind of tree of height \(h\) contains only \(2h\) items. Said another way, accessing item \(n\) takes \(n/2\) steps, or \(O(n)\) or linear time.

Of course, this is a pathologically bad binary tree — it’s not taking much advantage of the tree shape. Suppose we considered a binary tree that contains as many nodes as possible for its height:

In fact, let’s elevate this goal to a definition: a full binary tree is a binary tree where every level (except possibly the last) is completely full, and the last level of the tree is filled from left to right. Accordingly:

We can now successfully access any item in the tree in logarithmic time, and the root of the tree in constant time. Now if only we can ensure that the root of the tree always contains the maximum value, we’d be very nearly done! How can we do that?

29.4 Heaps and invariants

In particular, the left tree is a valid heap, even though the values in the lowest level are not sorted — because we’re not trying to sort our data! The right tree is not a valid heap, because 5 has a child 6, which is not in the correct heap order.

We now have two different invariants: a structural one (fullness) that ensures our trees will be conveniently short, and a logical one (heap-ordering) that ensures the values are ordered conveniently for our purposes. How can we exploit this?

29.5 Designing a priority queue

Let’s now figure out how to define the operations add and removeMax. (In the next section, we’ll see how to represent this design and implement it conveniently.) We’ll start by asserting that our priority queue will be represented by a heap, that is, a full binary tree that obeys the heap-ordering property. Here is an example of a valid heap:

29.5.1 Adding items to a heap

Let’s try to add 70 to this heap. Where should we add it? We have to ensure that we preserve both invariants, but no matter where we put 70, it looks like we’ll violate at least one of them: if we add it next to the 15, then it violates the heap-ordering property (because it is greater than its parent), and if we add it anywhere else, we violate the fullness property. What to do?

We now have a procedure that seems to work to insert an item into a heap. Does it always work, and does it always terminate? We know that it preserves the fullness property, because by construction, we always insert the item into the next available location. And the whole point of swapping the item with its parent is to restore the heap-ordering property. Further, every time we swap the inserted item with its parent, we get closer and closer to the root, so eventually we must reach the root and terminate. At that point, we have preserved the fullness property and restored the heap-ordering property, so this does work in all cases.

Next, how efficient is it? Assuming we can easily (as in, constant-time) find the next available spot to insert the item, the fullness property ensures that our tree has height \(O(\log_2 n)\), so the upheap operation takes at most \(O(\log_2 n)\) time. Therefore, our insertion operation happens in logarithmic time — much better than linear. (Of course, there’s a big assumption here, that we can easily find the spot at which to insert...and to see that we can do this efficiently, we’ll need one clever implementation choice, which we’ll address in the next section.)

29.5.2 Removing the maximum item from a heap

The maximum item of the heap is easily available: it’s sitting right at the root. If we just blindly “remove” the root, we have a hole in our tree, which breaks the tree in half. How can we fix this? It’s tempting to try to “undo” the upheap operation, but it’s much harder to figure out how to (efficiently) get from the root to the end of the tree than it is to figure out how to get from the end of the tree to the root. (Without examining the entirety of the tree, there’s no way to know whether to swap the root downwards to the right or to the left.)

Again we have to ask, does this procedure always work, and how efficient is it? Since we replace the root with the last item, we preserve the fullness property by construction. And the whole point of the downheap operation is to restore the heap-ordering property. Since every time we swap the item downward, we come closer to the leaves of the tree, we can only swap a finite number of times before terminating.

Again assuming we can find the last item efficiently, moving it to the root takes constant time, and the downheap operation runs in time proportional to the height of the tree, which, thanks to the fullness property, is logarithmic in the size of tree. Perfect.

29.6 Implementing a priority queue

Now we come to the clever implementation trick that makes heaps so efficient to implement. Let’s index each of the nodes in our tree, starting at the root and moving left to right in each level. Here, the indices of each node are shown colored and in brackets:

Let’s represent the tree above as an ArrayList<Integer>. The root will be at index 0, and the children of each node will be at their appropriate indices. Now we can see why the fullness property insists that the last layer of the tree be filled from left to right: a full tree containing \(n\) nodes will simply be stored in indices \(0\) through \(n-1\) of an ArrayList, with no gaps! And even better, we now can see that finding the next available location in the tree is trivial: it’s simply the next index in the ArrayList.

So our initial heap example,

Index:   0    1    2    3    4    5    6    7    8    9
Data: [ 80 | 60 | 50 | 30 | 50 | 40 | 20 | 10 | 20 | 15 ]

29.7 Heapsort

How efficient is heapsort? Building the heap inserts \(n\) items, and each insertion costs \(O(\log_2 n)\) work, so building takes \(O(n\log_2 n)\). Removing all the items removes \(n\) items, and each removal costs \(O(\log_2 n)\) work, so all the removals together take \(O(n\log_2 n)\) work also...so the entire algorithm takes \(O(n\log_2 n)\) work. Notice that unlike quicksort, there is no bad worst-case behavior, and unlike merge-sort, there’s no need for any additional storage. Heapsort is an efficient, in-place sorting algorithm.

Exercise

Implement all the algorithms sketched above in Java.