Study Notes by Niladri & AI

ch22-large-scale-changes

17 min read

Chapter 22: Large-Scale Changes

seg large-scale-changes lsc codebase refactoring tooling rosie

Status: Notes complete

Overview

Chapter 22 addresses a class of engineering problem unique to large organizations: changes that must touch thousands or millions of files across a codebase that no single person or team fully owns. Google calls these Large-Scale Changes (LSCs). They include compiler upgrades, API migrations, security patches across a codebase, deprecating old frameworks, and updating programming idioms at scale.

The chapter’s central thesis is that LSCs are fundamentally different from ordinary code changes in kind, not just in degree. The normal code review and submission process was designed for changes made by a single team in a small number of files with well-understood context. LSCs break every assumption in that model: they span dozens of teams, touch files the author has never read, require a review process that cannot expect per-change human scrutiny, and produce conflicts with virtually every in-flight change in a large codebase.

The chapter traces the infrastructure, policies, culture, and tooling Google has built to make LSCs feasible — particularly a system called Rosie — and argues that this infrastructure is not optional at Google’s scale. Without it, large categories of necessary changes simply would not happen, and the codebase would accumulate a form of technical debt that is qualitatively different from ordinary debt: not just expensive, but impossible to pay down.

Core Concepts

Large-Scale Change (LSC): A code change that is logically a single unit but that, due to the scale of the codebase, must be implemented as many individual submitted changes. The change may touch thousands of files across hundreds of teams. Examples: replacing a deprecated API, updating all callers of a function whose signature changed, migrating from one authentication library to another, or enforcing a new code style across the codebase.

Atomic change: A change that is submitted as a single commit — all of it lands or none of it does. The simplest and safest kind of change, but technically impossible for most LSCs because no version control system can atomically commit changes to millions of files that are being actively edited by thousands of engineers.

Haunted Graveyard: A section of code that everyone knows needs to change but that nobody ever touches because it is too risky, too complex, or too entangled. LSC infrastructure must be capable of touching haunted graveyards, or the policy of “anyone can submit an LSC” is incomplete.

Rosie: Google’s internal large-scale change tooling. Rosie takes a large change, splits it into shards, identifies reviewers for each shard, manages the submission process, handles conflicts and rebases, and tracks the progress of the overall migration.

ClangMR / JavaMR: Compiler-based refactoring tools for C++ and Java respectively that operate on the AST (Abstract Syntax Tree) rather than raw text. They produce semantically correct transformations (unlike sed/regex-based approaches that can be confused by comments, string literals, or complex syntax) and are the standard tools for creating LSC changes at Google.

What Is a Large-Scale Change?

An LSC is not defined by its total line count but by its structural characteristics:

  1. Cross-team scope: The change touches code owned by teams other than the team making the change
  2. Atomicity infeasibility: It is not practical to submit as a single atomic commit
  3. Review ownership ambiguity: The normal assumption that the committing engineer is the domain expert breaks down — the LSC author may not understand the context of every file they modify
  4. Conflict rate: The change will conflict with other in-flight changes at a rate that requires systematic management rather than ad-hoc rebasing

Examples of LSCs at Google:

  • Upgrading the C++ standard from C++14 to C++17 across the entire codebase
  • Replacing usages of a deprecated authentication token with a new API
  • Migrating from one logging framework to another
  • Enforcing a new set of compiler warnings across all C++ code
  • Adding required parameters to a function called at millions of call sites
  • Replacing a vulnerable cryptographic function with a patched version

What LSCs are not: Incremental feature development across many files (that is still owned by a single team), large refactors within a single codebase owned by one team, or changes that happen to touch many files but can reasonably be reviewed by their owners.

Who Deals with LSCs?

At Google, LSCs are managed by a small number of specialist teams and individual contributors with deep expertise in the LSC tooling and process:

  • Platform teams: Infrastructure, language, and compiler teams frequently initiate LSCs because they own shared foundations whose changes cascade to all dependents
  • Security teams: Security patches that require updating call sites across the codebase
  • Code health teams: Dedicated teams whose mission is maintaining codebase health, including deprecation of old patterns
  • Individual contributors: Any engineer at Google can theoretically initiate an LSC after getting authorization from the appropriate steering committee

The key structural feature: the initiator of an LSC is rarely the owner of the code being changed. This distinguishes LSCs from normal changes and requires a different social contract around code review.

Barriers to Atomic Changes

The authors identify several reasons why the “obvious” solution — make the change as a single atomic commit — is infeasible:

Technical Barriers

Repository scale: Git and most VCS systems do not efficiently handle commits touching millions of files. At Google’s scale, a single commit modifying 10 million lines of code would be technically impossible to create, review, or roll back.

Merge conflicts: A change touching 100,000 files will conflict with essentially every other in-flight change in the codebase. Resolving conflicts in a single atomic change becomes a Sisyphean task: by the time you resolve the last conflict, the first resolution has become stale.

Build system constraints: The build system must be able to verify that the entire change is correct before submission. At sufficient scale, even a no-op build verification across the entire codebase takes hours.

Human Scale Barriers

Review capacity: No team can effectively review a change across thousands of files they do not own. Standard code review requires the reviewer to have sufficient context to evaluate correctness, style, and implications — context that is only possible for the files a team owns.

Heterogeneity: Different parts of the codebase have different code styles, local conventions, and constraints. A change that is correct in one part of the codebase may need to be expressed differently in another. No single person can hold all this context.

Testing surface: The change must be tested, but the test suite for a million-file change is effectively the entire company’s test suite. Running it as a unit is infeasible.

Social Barriers

Ownership: Code review at Google requires approval from the code owners of each file. An atomic change touching files owned by 500 different teams would require 500 simultaneous approvals — coordination that is practically impossible.

Risk: Owners of code they did not ask to have changed are naturally conservative. Getting 500 teams to approve a risky change simultaneously is harder than getting each team to approve their small shard separately.

No Haunted Graveyards

One of the chapter’s most important principles is encapsulated in the phrase: “No Haunted Graveyards.”

A haunted graveyard is code that everyone knows is wrong, deprecated, or dangerous, but that nobody changes because:

  • It is too complex to understand safely
  • It is too tightly coupled to too many other things
  • Changes to it have historically caused mysterious failures
  • The original authors have left and nobody else understands it

Haunted graveyards are the enemy of LSC programs. If certain code is off-limits to LSCs, then:

  • Security patches cannot be applied uniformly
  • Deprecated APIs cannot be fully removed
  • New invariants cannot be enforced across the entire codebase
  • The codebase becomes partitioned into “safe” code that can be maintained and “haunted” code that cannot

The LSC program’s effectiveness depends on the ability to touch all code that needs to change. This requires:

  • Comprehensive test coverage of haunted code, so changes can be verified
  • Ownership clarity, so there is always someone who can review and approve changes to any file
  • A culture that does not treat any code as too sacred or too risky to change through proper process

The principle also has a positive corollary: the existence of haunted graveyards is evidence of a broken process, not just of complex code. If code cannot be touched, that is a systemic failure — of testing, of documentation, or of ownership — that must be remedied.

LSC Infrastructure

The chapter identifies five categories of infrastructure that must be in place before LSCs are feasible at scale:

1. Policies and Culture

LSCs require explicit organizational support:

  • A clear policy for initiating LSCs: Who can initiate? What authorization is required? What review process applies?
  • A culture of accepting LSC-initiated changes: Code owners must be willing to approve changes they did not author and may not fully understand, trusting the LSC process
  • Escalation paths: When a code owner refuses a change or is unresponsive, there must be a process for escalation

At Google, there is a central Large-Scale Change Steering Committee that authorizes LSCs and provides governance. This gives LSC authors the organizational backing to ask for approvals they otherwise could not compel.

2. Codebase Insight

LSCs require tools to:

  • Find all instances of the pattern being changed across the entire codebase (code search, semantic analysis)
  • Understand dependencies between the changed code and its callers
  • Identify owners of each file that needs to change
  • Estimate scope before committing to the LSC

Without comprehensive code search and dependency analysis, LSC authors cannot know what they need to change, and changes will be incomplete — leaving a patchwork of old and new patterns.

3. Change Management Tooling

Creating an LSC requires:

  • Automated change generation: Tools that can generate correct code transformations across millions of files (ClangMR, JavaMR, sed, or custom scripts)
  • Shard management: Splitting the overall change into chunks that can each be independently reviewed and submitted
  • Conflict handling: Tools that can automatically rebase shards when they conflict with other submissions
  • Progress tracking: Dashboards showing what fraction of the overall migration is complete

This is what Rosie provides.

4. Testing

Testing LSCs is different from testing ordinary changes:

  • Each shard must pass tests independently — a shard that breaks tests must be identified and fixed without blocking other shards
  • Test coverage must be sufficient that a broken shard can be detected; if code has no tests, there is no way to verify that the LSC change is correct
  • The testing infrastructure must scale — if the total LSC touches 50,000 files, the testing infrastructure must be able to verify 50,000 independently-submitted changes in a reasonable time

5. Language and Tooling Support

Mechanical code transformations require:

  • AST-based refactoring tools (not just regex/sed) that understand the semantics of the code and can make changes that are syntactically and semantically correct
  • A build system that scales to verify changes across the entire codebase
  • Code review tooling that supports the LSC workflow (showing reviewers only the relevant context, tracking approval across thousands of shards)

The LSC Process

Google’s LSC process has four steps:

Step 1: Authorization

Before any code is changed:

  • The LSC author writes a migration proposal describing the change, its scope (estimated number of files/teams affected), its motivation, and its risks
  • The proposal is reviewed by the Large-Scale Change Steering Committee
  • Authorization grants the author the right to submit changes to files owned by other teams, with the expectation that owners will approve appropriately scoped changes

Authorization is critical because it provides the organizational backing for the social aspects of the LSC. When an owner asks “why is this change being made to my code?”, the authorization document is the answer.

Step 2: Change Creation

The actual code changes are generated:

  • Automated tools (ClangMR, JavaMR, sed scripts, custom Python scripts) generate the transformations
  • The changes are verified by running tests on a representative sample before full-scale deployment
  • Edge cases are identified: code that the automated tool cannot handle safely is flagged for manual treatment
  • The change is split into shards — typically by directory, by team ownership, or by some other natural boundary

Step 3: Sharding and Submitting

Rosie manages the submission process:

  • Each shard is a separate CL (Change List — Google’s term for a commit/PR) touching a subset of the overall changed files
  • Rosie identifies the appropriate reviewers for each shard (based on code ownership)
  • Rosie sends shards out for review, tracks approvals, and submits approved shards
  • When a shard conflicts with another in-flight change, Rosie automatically rebases it and re-runs tests
  • Unapproved or failing shards are flagged for manual attention

The shard submission process may take days or weeks for large migrations. Progress is tracked on a migration dashboard.

Step 4: Cleanup

After the main migration is complete:

  • Verification: Confirming that all instances of the old pattern have been updated (code search confirms no remaining instances)
  • Removing infrastructure: Deleting the old API, removing compatibility shims, cleaning up the migration tooling
  • Documentation: Recording what was done, why, and any lessons learned for future migrations

Cleanup is as important as the migration itself. A migration that leaves compatibility shims in place “just in case” is not complete — it leaves the old complexity alongside the new, creating confusion.

Rosie: Google’s LSC Tooling

Rosie is the central tool in Google’s LSC infrastructure. Its core functions:

LSC Change (full)
       |
       v
   [ Rosie ]
       |
       |--- Shard 1 (files owned by Team A) ---> Review ---> Submit
       |--- Shard 2 (files owned by Team B) ---> Review ---> Submit
       |--- Shard 3 (files owned by Team C) ---> Review ---> Submit
       |--- ...
       |--- Shard N (files owned by Team Z) ---> Review ---> Submit
       |
       v
   Migration Complete

What Rosie provides:

  • Automatic sharding of the LSC into individually reviewable CLs
  • Reviewer assignment based on code ownership metadata
  • Automatic rebase when shards conflict with submitted changes
  • Test integration — each shard runs the test suite for its affected code
  • Dashboard and progress tracking — the LSC author can see at a glance which shards are approved, failing, conflicting, or awaiting review
  • Escalation support — when a reviewer is unresponsive, Rosie can trigger escalation to team leads

Rosie’s design principle: Each shard must be independently correct and independently approvable. A reviewer who approves their shard should not need to understand the overall migration — they need only verify that the change to their code is correct.

Why LSCs Need a Different Review Process

Normal code review assumes:

  • The author is the expert on the change
  • The reviewer has enough context to evaluate correctness independently
  • The change scope is small enough that a single reviewer can hold it in mind

LSCs violate all three assumptions:

  • The author may not understand the code being changed — they are applying a mechanical transformation, not an expert modification
  • The reviewer (the code owner) understands the code but not the global migration — they are evaluating whether the transformation is safe in their context
  • The scope is too large for any single reviewer to evaluate

The LSC review model at Google therefore shifts:

  • Reviewers are asked to evaluate: “Is this transformation correct and safe for my code?” — not “Is this a good idea overall?”
  • The “good idea” question was answered at the authorization stage
  • Reviewers are given explicit context about the overall migration (the authorization document)
  • Reviewers are expected to approve well-formed shards promptly, trusting the LSC process for the global correctness question

This is a significant change to the social contract of code review. It requires trust in the LSC process and the steering committee, which is why authorization governance is important.

TL;DRs

(Faithful reproduction from the book’s end-of-chapter TL;DR section)

  • An LSC (Large-Scale Change) is a change that is logically a unit but that is too large to be implemented as a single atomic commit.
  • Making LSCs feasible requires infrastructure and policy, not just tooling. Culture, governance, and testing are as important as the code transformation tools.
  • The No Haunted Graveyard policy is essential: LSC infrastructure must be capable of touching all code that needs to change. If some code is off-limits, the LSC program is incomplete.
  • LSC changes should be reviewed differently from normal CLs: reviewers evaluate local correctness in their code, not the global wisdom of the migration.
  • Rosie automates the mechanical parts of LSC management: sharding, reviewer assignment, conflict handling, and progress tracking.
  • AST-based refactoring tools (ClangMR, JavaMR) are essential for producing semantically correct code transformations at scale.
  • Cleanup — removing the old API and compatibility shims — is as important as the migration itself. An incomplete cleanup leaves technical debt.
  • LSCs are a key mechanism for keeping a large codebase healthy over time: without them, deprecated APIs never get fully removed and security patches cannot be applied uniformly.
  • The authorization step provides organizational backing for the social work of getting other teams to accept changes to their code.
  • Any organization with a large, shared codebase and a desire to maintain it over time will need something like the LSC infrastructure Google has built.

Key Takeaways

  1. LSCs are qualitatively different from large ordinary changes — they cross team boundaries, cannot be submitted atomically, require a different review model, and need dedicated infrastructure that ordinary commits do not.
  2. No Haunted Graveyards is a prerequisite for codebase health — if certain code cannot be touched by LSCs, security patches cannot be applied uniformly, deprecated APIs cannot be fully removed, and the codebase becomes partitioned into maintainable and unmaintainable regions.
  3. The LSC process has four steps: authorization (governance and organizational backing), change creation (automated code transformation), sharding and submitting (Rosie manages this), and cleanup (removing old patterns and infrastructure).
  4. Rosie automates the mechanical complexity of splitting a large change into reviewable shards, assigning reviewers, handling conflicts, and tracking progress — turning an organizationally infeasible change into a manageable process.
  5. AST-based tools (ClangMR, JavaMR) are essential for semantically correct transformations — regex and sed cannot reliably transform code because they do not understand syntax, scoping, or type information.
  6. Authorization provides the organizational foundation for the social contract of LSC review: it answers “why is someone changing my code?” and gives owners reason to trust the process.
  7. LSC reviewers evaluate local correctness, not global wisdom — the global question was answered at authorization; owners are asked only whether the specific change to their code is safe, reducing review burden.
  8. Test coverage determines whether an LSC is possible — code without tests cannot be safely transformed at scale; haunted graveyards often persist because they lack the test coverage needed to verify transformations.
  9. Cleanup is mandatory, not optional — leaving compatibility shims and old APIs in place after a migration increases complexity without reducing the burden that motivated the migration; a migration is not complete until the old infrastructure is removed.
  10. LSC infrastructure is not overhead — it is what makes large codebases maintainable over time — without it, technical debt accumulates in the form of deprecated APIs and security vulnerabilities that can never be fully remediated.

Last Updated: 2026-06-02

Graph View

Backlinks