Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide three-state output: "changed", "not changed", "unsure" #192

Open
cfallin opened this issue Jul 28, 2022 · 0 comments
Open

Provide three-state output: "changed", "not changed", "unsure" #192

cfallin opened this issue Jul 28, 2022 · 0 comments

Comments

@cfallin
Copy link
Member

cfallin commented Jul 28, 2022

Right now, Sightglass uses a single threshold based on a confidence interval computed by Behrens-Fisher to determine whether a sampled statistic shifted between configurations.

The result of this is that we get either "changed" (i.e., benchmark got 5% faster) or "not changed". However, the latter answer can also appear if we simply don't have enough data points to prove statistical significance, or if the system is too noisy.

This "false negative" is somewhat dangerous: we could make a change, see that it is performance-neutral according to Sightglass, and accept it, but actually we just didn't turn the knobs up high enough.

Ideally, Sightglass should provide a third output of "unsure" if the measurements aren't precise enough to prove either "changed" or "not changed" to the desired confidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant