Performance Regressions Policy

Basic Policy

Performance is a critical goal for Firefox releases, as well as Mozilla’s other products and services. We cannot allow performance regressions to go unnoticed or unresolved during our development cycles. When a regression is identified, a bug will be filed and the patch author will be asked to provide information via the needinfo flag in bugzilla. We expect a response and reasonable dialog within 3 business days. If a resolution is not found we will back out the offending patch(es) and the patch author can reland when they have more time to investigate.

Sheriffing Criteria

In order to make efficient use of our performance sheriff resources, the following criteria must be met before results will be monitored for regressions:

• Test jobs run on integration or mozilla-beta.
• Test jobs run as tier 2 or above.
• Tests can be run and debugged locally.
• Tests are documented in the Firefox Source Docs or Mozilla Wiki.

In the future this list will be extended to include thresholds for noise, engineer traction, and fix ratio. In the meantime please consider these aspects when working on performance tests.

Bug Requirements

To be fair, some initial work up front should be done prior to filing a bug and requesting a needinfo from the original author. Here is what to expect:

• On integration branches (higher volume), a performance sheriff will have verified the root cause within 5 business days of the patch landing.
• A patch or set of patches from a bug must be identified as the root cause. This can take place through retriggers on the tree or in the case of many patches landing at once this would take place through a push to try backing out the suspected patch(es).
• Links in the bug to document the regression (and any related regressions/improvements).
• If we are confident this is the root cause and it meets a 2% regression threshold, then the needinfo request will mention that this policy will be enforced.

Acceptable Outcomes

What we expect from a developer when asking them to look at a regression:

• A promise to attempt a fix at the bug is agreed upon, the bug is assigned to someone and an estimated timeline is documented in the bug.
• The bug will contain enough details and evidence to support accepting this regression, we will mark it as wontfix.
• It is agreed that this should be backed out and a sheriff will verify the backout removes the regressions.

Other Scenarios

There are many reasons why a regression will not fall into the above workflow. Here a few examples of other scenarios and how we will handle them:

• A bug related to the alert is not filed within 5 business days of the patch landing. This removes the urgency and required action.
• We only caught a regression at uplift time. There is a chance this isn’t easily determined, this will be documented and identified patch authors will use their judgement to fix the bug.
• Regression is unrelated to code (e.g. infrastructure changes) - this should be documented in the bug and closed as wontfix.
• When we uplift to Beta, all regressions filed before the uplift that show up on the upstream branch will have a needinfo flag set and require action to be taken.
• A backout or fix doesn’t fully resolve the regression. We will do our best to understand the difference and fix or document the remainder.