droid
Composite Keys and Index-Only Scans
Indexes and the Optimizer · Stage 6 of 11

Composite Keys and Index-Only Scans

Index several columns at once, and answer a query from the index alone

Build indexes over several columns, use one for any WHERE that bounds a prefix of its columns, enforce UNIQUE across all of them, and skip the table entirely when the index holds every column a SELECT reads.

Not run yet · 10 tests
Concepts in this stage
  • A composite key sorts by its first column, then its second: it helps a prefix of its columns and never the second alone
  • An equality prefix followed by one range is still one contiguous walk of the leaves
  • Every entry carries the table key, so the key is in every index for free
  • An index covers a query when it holds every column the query reads; then the second descent disappears
  • Covering changes how a plan reads, never which index it reads
Where this stage sits in the engine
Talking to the user
REPL
Sessions
Printer
Server
Wire protocol
Understanding SQL
Lexer
Parser
Validator
Answering a query
Planner
Executor
Keeping changes safe
TransactionsWrite-ahead log
Undo log
Lock manager
Read views
Storing rows
Row formatSlotted page
B+Tree
Reaching the disk
PagerDatabase file
this stagebuilt alreadystill to comeA statement enters at the top; rows come back up.

Why this stage exists

Two ideas share this stage because they are two sides of one fact: an index entry can hold more than one value.

The first is the composite key. Queries often pin more than one column: where city = 'x' and age = 30. An index on city finds every row in the city and filters the ages; an index on (city, age) goes straight to the rows that match both.

The second is the covering index. Every lookup so far paid for two descents: one in the index, one in the table to fetch the row. If the entry already holds every column the query needs, the second descent has nothing to fetch.

A phone book sorted by last name, then first name

Entries of an index on (city, age) sort by city first, and by age only among rows with the same city:

Index on (city, age): (x,20,30) (x,21,21) (x,22,12) ... (y,20,10) (y,21,1) ...
header
cells = 6
cell 0
key = 30
new
cell 1
key = 21
new
cell 2
key = 12
new
cell 3
key = 10
cell 4
key = 1
cell 5
key = 22
free space
Each cell is 12 bytes. A cell that moves one position moves by 12 bytes.
Cells labelled by their table key. All of city x sits together, in age order. Age 21 appears once in x and once in y, far apart.

That ordering decides which WHERE clauses the index can serve, exactly as a phone book sorted by last name, then first name, can find "Smith" or "Smith, Anna" but not every "Anna":

WHEREUsable?Why
city = 'x' and age = 30yes, both columnsone value of each: one spot
city = 'x'yes, the first columnall of city x is one run of entries
city = 'x' and age > 20yes, = then a rangeinside city x, ages are in order: still one run
city > 'm'yes, a range on the firstthe cities are in order
age = 30noage 30 is scattered across every city

The rule: an index helps through a prefix of its columns, = on as many leading columns as the WHERE pins, then at most one range on the next. Everything after a range, and every column the prefix does not reach, goes into the Filter.

The second descent, and when it disappears

Every entry ends with the row's table key. So an index on (city, age) actually holds city, age and id, and a query that reads only those has nothing to fetch from the table:

select city, name from t where city = 'x'
Project (cols=[city, name])
↑ 10 rows
ran
IndexScan (table=t, index=ca, key=('x'))
10 entries, 10 table lookups
select city, id from t where city = 'x'
Project (cols=[city, id])
↑ 10 rows
ran
IndexOnlyScan (table=t, index=ca, key=('x'))
10 entries, 0 table lookups
Same index, same entries. name is not in the index, so the first query fetches each row; the second never touches the table.

The [EXEC] line shows it: rows_examined=10 index_lookups=0. And the pager shows it louder: on a table of wide rows, the covered query reads a handful of index pages, while the same WHERE with one more column reads the table page by page.

count(*) is the favourite trick: it reads no column, so any index whose columns cover the WHERE answers it without the table.

Covering decides how to read the chosen index, never which index to choose. That keeps the plan predictable now; weighing a covering index against a better-matching one is a cost question for the cost-model stage.

The algorithm in words

1

Build a composite entry

The entry is the row's values in each indexed column, in the order the index lists them, then the table key.

  1. 1.Check each listed column: it exists (00402), and it is not listed twice (00408).
  2. 2.For each row, concatenate the values of the listed columns, in index order, then the table key. Their bytes together must not pass 255 (01604).
  3. 3.Compare entries column by column, by the rules of each column's type, NULL first, then by table key.
  4. 4.A unique composite index conflicts only when every column is equal and none is NULL.
2

Match a WHERE to an index

Walk the index's columns from the first, and stop at the first one the WHERE does not pin.

  1. 1.For each index, count how many of its leading columns have an = index bound, in index order, stopping at the first that has none.
  2. 2.If the next column has a range, take its first lower and first upper bound as well.
  3. 3.Prefer the table key, then the index with the most = columns, then one with a range after them, then the one whose first = comes first in the WHERE, then the oldest index.
  4. 4.With no = on any index's first column, fall back to the range rule of the previous stage, on an index whose first column has a range.
3

Read only the index when it covers the query

Decide once the scan is chosen.

  1. 1.List every column the SELECT reads: select list, WHERE, ORDER BY, GROUP BY, HAVING, aggregate arguments. * reads them all.
  2. 2.If each is a column of the chosen index or the table's key, the scan is an IndexOnlyScan.
  3. 3.An IndexOnlyScan builds each row from the entry's values and table key, and never descends the table: index_lookups stays 0.
  4. 4.UPDATE and DELETE always read the row; their scans are never index-only.

Output contract for this stage

These are the exact lines the tests check. The full definition is on the output contract page, section 58.

InputOutput
create index ca on t (city, age);Create index statement executed successfully
indexes[INDEX] ca on t (city, age) root=<page>
explain select * from t where age = 30 and city = 'x';IndexScan (table=t, index=ca, key=('x', 30))
explain select * from t where city = 'x';IndexScan (table=t, index=ca, key=('x'))
explain select * from t where city = 'x' and age > 20;IndexScan (table=t, index=ca, key=('x'), range: age > 20)
explain select * from t where age = 30;Filter (age = 30) over SeqScan (table=t)
explain select city, id from t where city = 'x';Project (cols=[city, id]) over IndexOnlyScan (table=t, index=ca, key=('x'))
a covered SELECT[EXEC] rows_examined=<n> index_lookups=0
an index column the table lacks; one listed twice[ERROR:00402]; [ERROR:00408] <message>

A one-column index prints as before: key='x', range: name > 'b'.

What you should see when done

shell
$droid --db people.db
Welcome to droid!
>create table t (id int, city varchar(10), age int, name varchar(20));
Create table statement executed successfully
>create index ca on t (city, age);
Create index statement executed successfully
>insert into t (id, city, age, name) values (1, 'x', 30, 'ada');
Insert statement executed successfully (1 rows)
>insert into t (id, city, age, name) values (2, 'x', 25, 'bob');
Insert statement executed successfully (1 rows)
>insert into t (id, city, age, name) values (3, 'y', 30, 'cy');
Insert statement executed successfully (1 rows)
>select id, age from t where city = 'x';
id | age
2 | 25
1 | 30
(2 rows)
[EXEC] rows_examined=2 index_lookups=0
>select name from t where city = 'x' and age = 30;
name
ada
(1 rows)
[EXEC] rows_examined=1 index_lookups=1
>.exit
exiting.. good bye!

Common bugs in this stage

  • Using the index for its second column alone. The entries are not in that order; the rows come back wrong or missing.
  • A range on the first column followed by = on the second used as one seek. After a range, the second column is no longer in order; it belongs in the Filter.
  • Printing the prefix in WHERE order. key=(...) follows the index's column order, whatever order the WHERE was written in.
  • Forgetting the table key when deciding coverage. select id, city is covered by an index on city.
  • An index-only UPDATE. It must read the row to write it.
  • A unique composite index refusing ('a', NULL) twice. A NULL in any column means no conflict.

Language notes

The page above is the same for every language. These are the only language-specific hints.

  • C: compare composite entries with a loop over the index's column types, decoding one value at a time; stop at the first difference.
  • C++: a std::vector<Value> per entry with a lexicographic compare reads clearly; encode it once for the page.
  • Rust: a Vec<Value> with Ord implemented per value compares lexicographically for free.
  • Zig: keep the column types in the index handle and walk them in the compare function.

Checklist

One line per thing the tests check. The tags name the test cases.

  • An equality on both columns of a composite index finds the row through it
    both-columns
  • An equality on the first column alone uses the index, rows in (city, age) order
    prefix-only
  • An equality on the second column alone cannot use the index
    second-column-alone
  • An equality on the first column and a range on the second walk only the rows between
    prefix-plus-range
  • When the index holds every column the query reads, the table is not read
    index-only-scan
  • Every entry holds the table key, so a query of indexed columns and the key is covered
    pk-is-covered
  • An index-only scan reads fewer pages than the same scan that looks rows up
    index-only-fewer-reads
  • A unique index on two columns refuses a pair it already holds, and only that
    unique-composite
  • An index column the table lacks, or listed twice
    composite-errors
  • explain: key=(...) lists the prefix in index order; IndexOnlyScan when covered
    composite-plans