When your company rolls out a new tool such as GitHub Copilot, you will want to measure the impact of the tool on your engineering systems and assess the tool's contribution to your company's goals.
GitHub's Engineering System Success Playbook (ESSP) provides strategies and metrics for driving meaningful and measurable improvements. The playbook recommends a three-step process for solving engineering challenges:
- Identify the current barriers to success.
- Evaluate what needs to be done to achieve your goals.
- Implement your changes, monitor results, and adjust.
Define your goals
Choose an engineering outcome before deciding which Copilot metrics to track. Usage and adoption signals show only whether people are using the tool. Outcome measures from your engineering systems and developer feedback help you assess whether the changes you want are actually occurring.
Start with one or two goals and communicate them to your teams. Record a baseline before changing your rollout. Use the existing guidance for each goal to plan training and select outcome measures.
Usage signals do not identify whether a particular interaction produced a test, fixed a vulnerability, or prevented an incident. Use them to understand adoption alongside your outcome measures, not as a substitute for those measures. For example, more generated code does not establish better test coverage, and a higher acceptance rate does not establish code quality or developer satisfaction.
Increase test coverage
If gaps in testing are delaying releases or reducing confidence, track line and branch coverage from your continuous integration reports. Also track time spent writing tests and developer confidence. Compare these outcomes with chat and agent usage trends among participating teams. See Increasing test coverage in your company with GitHub Copilot.
Accelerate pull requests
If review delays are slowing delivery, track pull request lead time and review cycles, while checking for defects after deployment. Use adoption cohorts, pull requests merged per user, and time to merge in the impact dashboard to explore how adoption relates to delivery. See Accelerating pull requests in your company with GitHub Copilot.
Reduce security debt
If unresolved vulnerabilities are accumulating, track the security alert backlog and time to remediate in security overview. Consider agent and code review usage trends among teams working on remediation alongside these outcomes. See Reducing security debt in your company with GitHub Copilot.
Improve code quality
If maintainability or reliability problems are increasing, review organization-level code health and open findings over time in Code Quality. Compare these with adoption and feature usage trends among the teams working on those problems. See Exploring GitHub Code Quality results in your organization.
Keep the set of analyzed repositories consistent. Open finding counts can change when analysis is enabled on more repositories or findings are dismissed, not only when code is improved.
Reduce production incidents
If refactoring and maintenance work aims to improve stability, track production incidents related to the refactored code using your incident records. Review those results alongside usage trends among teams using Copilot for that work. See Using GitHub Copilot to reduce technical debt.
Improve developer experience
If your goal is to reduce workflow friction, gather feedback on usefulness, trust, and satisfaction through surveys or retrospectives. Use sustained usage, feature adoption, and acceptance rate trends to identify questions to explore with developers, rather than inferring satisfaction from activity alone. See Measuring the success of a GitHub Copilot trial.
Compare results responsibly
Use the same comparison approach each time you review progress:
-
Align the scope and period. Compare usage and outcomes for the same teams and reporting dates. Record the repositories and tools included, and keep metric definitions consistent with your baseline.
The impact dashboard compares users in different adoption cohorts, not the same users before and after adoption. Differences between cohorts can reflect their work or experience as well as their use of Copilot. See Interpreting usage and adoption metrics for GitHub Copilot.
-
Check what else changed. Record changes in staffing, project complexity, release schedules, training, or engineering practices. These can affect outcomes independently of Copilot.
-
Check quality alongside speed. If pull requests merge faster, also review defects, incidents, and developer feedback. Increased output alone does not establish an improvement in engineering outcomes.
-
Investigate before attributing results. Review trends over multiple periods and ask teams how they used Copilot. Describe observed changes as associations unless you have additional evidence that supports a causal conclusion.
Use the combined findings to choose a next action. For example, if test coverage is unchanged despite growing usage, ask teams whether they are using Copilot to write tests. Address any training or workflow barriers before expanding that approach.