Report
How Do You Know You Caught Everything?
Changing a database schema in code you didn't write, and why the tool you're probably using can't tell you what it missed.
The schema you inherited
You have a database schema, and it lives inside a real application. Models, serializers, route handlers, migrations, a reporting query somebody wrote years ago, a background job nobody has opened since.
The schema needs to change. A field should be nullable. A column has the wrong name. Two tables want to become three. This is ordinary work, most of what maintaining an application consists of, and completely unlike designing a schema from scratch. Greenfield, you decide. Brownfield, you negotiate with everything that already depends on a decision somebody made earlier.
The week after the rename
You rename a column. The migration is four lines. The obvious breaks happen immediately — the model, the serializer, the two routes that reference it — and you fix those in twenty minutes.
Then you spend a week finding the rest: the query built from a string, the report that still runs but now means something different, the downstream consumer reading a field whose shape moved underneath it.
Power Platform, which was my tool of choice in 2024, doesn't have this problem. The logical name is fixed at creation and immutable; the display name — the thing anyone actually reads — is free to change. Renaming is a label edit, and nothing downstream can break because nothing downstream points at the label.
Protobuf does the same thing with field numbers. Avro doesn't, and matches on name, which is why the same rename is invisible in one and fatal in the other.
Most stacks conflate the two. Your column name is the identifier and is the label, so a rename is simultaneously cosmetic and structural — and you find out which by shipping it.
Two properties make this worse than ordinary refactoring.
Unseen failures. Code that reads a renamed column raises an error. Code that reads a column whose meaning changed keeps working and returns the wrong answer. Dashboards keep rendering. Reports keep running. Nobody files a bug, because from the outside nothing broke.
No finish line. At some point you stop finding things, and stopping feels like finishing. It's not the same. There's no moment where a tool tells you the set is empty, so you ship, and then you wait.
You know how to change a schema. But how do you know you caught everything?
What people actually do
1. Don't change it
The most widely practiced approach is avoidance, and it's rational. The
rename isn't worth a week. So you add a nullable column beside the old one. You
leave the misleading name and write a comment. You add _v2 and
dual-write.
Each of these is cheap on the day and permanent by accident. The cost is deferred as technical debt: a schema that records the history of changes people were afraid to make, a table where three columns mean almost the same thing, and a new engineer who can't tell which one is authoritative.
2. Point an AI at it
This is the current approach, and it's better than what came before. Point a capable model at the repository, let it search, let it propose edits. It's fast, it reads code well, and it will find references that a careful engineer skims past.
But it can't tell you what it missed.
Text search has no notion of binding. user_id in
a query, user_id as a local variable in an unrelated function,
user_id in a docstring, and user_id as the column are
the same string. Deciding which occurrences are the schema element requires
knowing what names refer to, which is a question about scope and imports, not
about characters.
So the model reasons about it, case by case, and it's often right. The problem is what happens at the boundary. A process that finds forty-seven of fifty references reports success, because the three it didn't find were never in the candidate set. There's nothing to be uncertain about. Confidence goes up; certainty doesn't move.
A tool that can't enumerate its own blind spots can't support a coverage claim, and coverage is the only thing you actually wanted.
3. Parse the language
The rigorous answer is to stop searching text and start parsing the language.
Python ships ast
in the standard library: free, fast, native C, no dependencies. It gives you real
structure: this is a class definition, this is an attribute assignment, this name
is in this scope. That's exactly what text search lacks.
For finding things, ast is entirely adequate.
It can't give your code back.
The Abstract Syntax Tree is abstract in a specific and costly way. It discards
comments completely. It reduces whitespace to INDENT and
DEDENT tokens. It drops the parentheses you added for readability.
ast.unparse(), available since Python 3.9, regenerates source from
the tree, with indentation set to exactly four spaces, whatever you were using.
As the survey puts it: like a JPEG, the AST is lossy.
That confines ast to analysis. It can tell you where to change
things; you still make the change yourself.
The tradeoff all three make
The three approaches trade against each other. Avoidance keeps your code
readable and never changes it. AI-with-grep changes your code and can't prove
coverage. ast proves coverage and can't change your code without
wrecking it.
The reason isn't aesthetic.
Run a formatting-lossy tool across your models. It performs the rename correctly. It also reformats every file it touched, strips the comment explaining why a field was nullable, and collapses alignment somebody set up deliberately.
Now look at the pull request. Four hundred changed lines, four of which are the rename.
Nobody merges that. Not because it's wrong (it may be exactly right) but because nobody can check whether it's right. The signal is buried in noise the tool generated. The reviewer either approves blind or redoes the change by hand, and both outcomes are worse than not having run the tool.
A change you can't review is a change you have to trust, which is what you were trying to stop doing.
Concrete instead of abstract
LibCST (Meta/Instagram, MIT licensed) parses Python to a Concrete Syntax Tree rather than an abstract one. Concrete means lossless: comments, whitespace, parentheses, all retained. Parse, modify, regenerate, and the output is byte-identical apart from your change. Its own design note describes the compromise precisely: looks like an AST, preserves like a CST.
That single property resolves the trade. You get the structural rigor of parsing and a diff a human can read.
The adoption figures are more informative than the star count, which is a modest ~1,800. Roughly 3.1 million weekly downloads, 409 dependent packages, classified by the survey as a key ecosystem project. It's the core of Instagram's linting and automated refactoring across an enormous Python codebase, and it runs in production at Instawork and SeatGeek. Runtime is Python 3.9+, but it parses 3.0 through 3.14, which is what you need when pointing a tool at code you didn't write and don't control.
The visitor and transformer patterns are more ceremony than walking a tree
with ast.walk(), and the learning curve is real. The survey scores
LibCST 9.5 against ast's 6.5, with
the whole gap coming from formatting preservation and modification APIs.
The survey covers three others. Rope is the only serious contender and can't parse Python past 3.10; RedBaron supports nothing newer than Python 3.7; Bowler was archived in 2025 and its own documentation now points at LibCST.
What a readable diff buys you
The diff is small enough to read. Only the lines that changed, changed. Review is possible, which means approval means something.
Coverage is a list. Parsing gives you an enumerable set of references. You can count them, read them, and check them off.
The blind spots are enumerable too. This is the part that separates a tool from a claim. Parsing tells you what it couldn't resolve, so you can see the edges of what you don't know.
Comments and intent survive. The note explaining why a field is nullable is still there after the tool runs. That documentation is often the only record of a decision.
The tool can hand code back. Not a report of what to change — the changed files, ready to review.
It reads modern Python. 3.0 through 3.14, which is the actual range you encounter in codebases you inherit.
What it still can't see
Static analysis has a boundary.
Parse a codebase and references sort into three groups.
- Some resolve — you know exactly which schema element they mean, and a tool can rewrite them.
- Some are only locatable — you can find the line, but deciding what it means requires a human.
- And some are opaque: attribute access built at runtime, queries assembled from strings, anything reached through reflection. No amount of parsing recovers those, because the information isn't in the source text.
What a graph of resolved references gives you isn't "nothing broke." It's: every reference I can see is accounted for, and here is the list of the ones I can't. That's a claim that survives contact with a real codebase.
What I built with it
SEA — Schema Evolution Automation — is built on LibCST for the reasons above. I am not the first to land on it: Pydantic's own bump-pydantic made the same choice to carry codebases through the v1-to-v2 migration, and was archived once that migration finished. Same reasoning, applied to a one-time migration instead of a recurring one.
It classifies a schema change against a catalogue of designated change types, generates the code, and splices it into models, schemas, routes, and migrations, returning files you can read a diff of. On a 149.5k-line payments platform, the scan completes in 3:06 with an 81-item worklist, sorted into exactly the three categories above.
The generated application is yours and runs without SEA.
If you would like to see it work on a real open-source codebase, the deep dives run monthly: ninety minutes, materials provided, everybody's hands on the keyboard.
Source
The library comparison draws on Survey of Software 1.104.1 — Python Code Parsing & AST Libraries, which evaluates six libraries across formatting preservation, modification APIs, maintenance status, documentation, and ease of use. The survey is neutral and general; the argument on this page is applied and opinionated.