MySQL deep-dive · Part 7 of 9. Next: Schema changes without downtime.
In Part 6 you saw that isolation is
half illusion: plain SELECTs read from an MVCC snapshot, a consistent
point-in-time view that never blocks and is never blocked. That covers reads.
But the moment a transaction intends to change a row — or even declares that
it’s about to — MVCC snapshots aren’t enough, and InnoDB reaches for the other
half of the machinery: locks. This chapter is about that half, and about
the failure mode it produces under load — deadlocks — which are not a bug to be
eliminated but a condition to be handled.
Hold onto one sentence for the whole article: InnoDB locks index records, not rows, and not tables. Almost every surprising thing about MySQL locking falls out of that.
Shared and exclusive locks
There are two kinds of row-level lock, and they differ only in how they tolerate company.
A shared (S) lock says “I’m reading this row and I don’t want it to change
under me.” Many transactions can hold an S lock on the same row at once —
they’re all just reading. You take one explicitly with SELECT ... FOR SHARE.
An exclusive (X) lock says “this row is mine to modify.” Only one
transaction can hold it, and while it does, no one else can take an S or an X
lock on that row — they wait. Every UPDATE and DELETE takes an X lock on the
rows it touches, and you can request one explicitly with SELECT ... FOR UPDATE.
The compatibility rule is the whole story: S is compatible with S; X is compatible with nothing.
Three granularities: record, gap, and next-key
“Row-level lock” is a convenient lie. InnoDB actually has three flavours of lock, and understanding the difference is what separates people who can debug a deadlock from people who guess.
A record lock locks a single index record — one existing row, found through an index. This is what you picture when you think “lock the row.”
A gap lock locks the open interval between two index records. It locks
nothing that exists — there’s no row there — it locks the possibility of a
row. Its only job is to stop another transaction from INSERTing a value into
that range. If ids 10 and 20 exist and you gap-lock between them, nobody can
insert id 15 until you’re done.
A next-key lock is the combination: a record lock on a row plus the gap
immediately before it. This is InnoDB’s default behaviour for a locking read
under REPEATABLE READ. When you run a ranged SELECT ... FOR UPDATE, InnoDB
doesn’t just lock the matching rows — it next-key-locks them, sealing the gaps
around them too.
Why next-key locks are how RR prevents phantoms
Recall from Part 6 that REPEATABLE READ promises the same query returns the same rows twice within a transaction —
no phantoms, no new rows appearing between reads. For plain snapshot reads,
MVCC delivers that for free. But a locking read (FOR UPDATE) reads the
latest committed data, not the snapshot — so MVCC can’t help it. Something has
to physically prevent a concurrent insert into the range you just read.
That something is the gap lock. When transaction A runs:
SELECT * FROM accounts WHERE balance > 1000 FOR UPDATE;
InnoDB next-key-locks every matching record and the gaps between them. Now if
transaction B tries INSERT INTO accounts (balance) VALUES (5000), it lands in
a locked gap and waits. A can re-run its query and get the identical set.
The phantom is prevented not by magic in the snapshot, but by gap locks blocking
the range from filling in.
The gotcha that eats concurrency: locks attach to index records
Here is the sentence from the top, cashed out. Because locks attach to index records, InnoDB can only lock a row by finding it through an index. So the locking behaviour of a write depends entirely on how the query is executed — the same lesson about scans from Part 4, now with teeth.
Consider:
UPDATE accounts SET frozen = 1 WHERE owner_email = 'a@b.com';
If owner_email has an index, InnoDB seeks straight to the matching records and
locks just those. If it does not, InnoDB must do a full scan — and it takes
an X lock on every index record it examines, not just the ones that match
the WHERE and get changed. To find your one row, it walks and locks the entire
table’s worth of records. For the duration of that transaction, essentially
nobody else can write to accounts.
Deadlocks: a cycle, a detector, a victim
A deadlock is not “the database got stuck.” It’s a precise structure: a cycle of waits.
- Transaction A holds a lock on row 1 and wants row 2.
- Transaction B holds a lock on row 2 and wants row 1.
Neither can proceed, because each is waiting for a lock the other holds and will only release on commit. Left alone they’d wait forever. So InnoDB doesn’t leave it alone: it runs a deadlock detector that watches the wait-for graph and, the instant a cycle forms, breaks it. It picks a victim — usually the transaction that has done the least work, because it’s the cheapest to undo — rolls that transaction back entirely, and lets the other proceed. The victim’s client gets:
ERROR 1213 (40001): Deadlock found when trying to restart transaction
That rollback is InnoDB working correctly. It detected an unresolvable situation and resolved it the only way possible: by sacrificing one party so the other can make progress.
Worked example: the classic transfer deadlock
Money transfers are the textbook case because the business logic naturally locks two rows in an order that depends on the direction of the transfer.
Transaction A transfers from account 1 to account 2. Transaction B transfers from account 2 to account 1. They run at the same time. Watch the interleaving:
-- Transaction A (transfer 1 -> 2) -- Transaction B (transfer 2 -> 1)
BEGIN; BEGIN;
UPDATE accounts SET balance = balance - 100
WHERE id = 1; -- A holds X lock on id=1
UPDATE accounts SET balance = balance - 50
WHERE id = 2; -- B holds X lock on id=2
UPDATE accounts SET balance = balance + 100
WHERE id = 2; -- A WAITS for id=2 (B has it)
UPDATE accounts SET balance = balance + 50
WHERE id = 1; -- B WAITS for id=1 (A has it)
-- cycle: A waits on B, B waits on A
Now A holds id=1 and needs id=2; B holds id=2 and needs id=1. That’s the cycle. InnoDB’s detector spots it immediately, picks a victim — say B — rolls B back, and returns 1213 to B’s client. A commits fine. B’s transfer simply didn’t happen.
Notice what caused it: the two transactions acquired their locks in opposite orders. A went 1→2, B went 2→1. That’s the entire disease.
The fix: lock in a deterministic order
The cure is to make every transaction acquire its locks in the same order, regardless of business direction. Sort by something stable — the primary key is perfect. Grab both rows up front, in ascending id order, with a single locking read:
BEGIN;
-- Lock BOTH rows first, in a fixed order that never depends on direction:
SELECT * FROM accounts WHERE id IN (1, 2) ORDER BY id FOR UPDATE;
-- Now do the arithmetic; the locks are already held, in a consistent order.
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;
Because both transactions now lock id=1 before id=2, one of them wins the race
for id=1 and the other queues behind it — a plain wait, not a cycle. The loser
blocks briefly, then proceeds once the winner commits. No cross-wait can form,
so no deadlock. The ORDER BY id is the load-bearing part: it turns “lock in
whatever order the transfer implies” into “always lock ascending.”
Consistent ordering is the single most effective structural defence. But it is not the only defence, and it is not a guarantee.
You cannot eliminate deadlocks — so retry
The most expensive misconception in this whole area:
“A deadlock means my code is broken. If I get the locking right, I’ll never see one.”
No. Under real concurrency, deadlocks are normal and expected. Consistent lock ordering handles the two-row transfer case cleanly, but a system of any size has locking paths you didn’t anticipate: a gap lock from one query colliding with an insert from another, two different code paths touching overlapping rows through different indexes, a background job crossing a user request. You can drive the frequency toward zero. You cannot prove it is zero. And InnoDB rolling back a victim is the system functioning, not failing.
Which means there is exactly one mandatory countermeasure: retry logic. A 1213 is not an exception to log and surface to the user — it’s a signal to replay the transaction from the top. The victim was rolled back cleanly, leaving no partial state, so re-running it is safe.
def transfer_with_retry(conn, from_id, to_id, amount, attempts=3):
for attempt in range(attempts):
try:
with conn.begin(): # BEGIN ... COMMIT
lo, hi = sorted((from_id, to_id)) # consistent lock order
conn.execute(
"SELECT * FROM accounts WHERE id IN (%s, %s) "
"ORDER BY id FOR UPDATE", (lo, hi))
conn.execute("UPDATE accounts SET balance = balance - %s "
"WHERE id = %s", (amount, from_id))
conn.execute("UPDATE accounts SET balance = balance + %s "
"WHERE id = %s", (amount, to_id))
return # committed; done
except DeadlockError: # MySQL error 1213
if attempt == attempts - 1:
raise # give up after N tries
continue # replay the whole txn
Two things make this correct. First, the whole transaction is inside the
retry loop — you replay from BEGIN, not from the middle, because the rollback
threw away everything. Second, it’s bounded: a handful of attempts, ideally with
a little backoff, so a persistent conflict fails loudly instead of spinning.
The defences, stacked
There’s no single knob. You layer cheap structural habits and keep the one non-negotiable at the bottom:
- Consistent lock ordering. Always acquire locks in a deterministic order (ascending primary key), independent of business direction. Kills the classic cross-wait cycle.
- Keep transactions short. Locks are held until commit. A transaction that does network I/O or user think-time while holding X locks is a deadlock and 1205 factory. Acquire late, commit fast.
- Index your
WHEREcolumns for writes. Lock the rows you mean, not every row you scan. This alone removes a huge class of accidental wide locks. - Consider
READ COMMITTEDfor high-contention write paths to shed gap locks — at the cost of phantoms. - Always retry on 1213. Non-negotiable. Everything above reduces frequency; only retry makes the residual deadlocks a non-event.
If you can explain why an unindexed
UPDATElocks the whole table, and why the transfer deadlock disappears the moment both transactions lockidascending — but retry logic is still mandatory anyway — you understand InnoDB locking better than most people shipping to it.