Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

median implementation is slow #26

Open
belm0 opened this issue Nov 19, 2021 · 1 comment
Open

median implementation is slow #26

belm0 opened this issue Nov 19, 2021 · 1 comment

Comments

@belm0
Copy link

belm0 commented Nov 19, 2021

TLDR; don't use skip lists

I've found that a sorted list implemented with standard list + bisect module will be faster for tracking median than rolling's implementation up to about N < 50,000. For example, for N of 10,000 it's 4x faster. Resources explaining why list + bisect are unbeatable at these sizes: http://www.grantjenks.com/docs/sortedcontainers/implementation.html, http://www.grantjenks.com/docs/sortedcontainers/performance-scale.html

Of course, rolling would want to perform well for N > 50,000 too. So use sortedcontainers.SortedList. Even though it doesn't beat standard list + bisect until about N of 20,000, it's still faster than rolling's implementation for all sizes. For example, for N of 100,000 using SortedList is 2x faster.

@ajcr
Copy link
Owner

ajcr commented Nov 24, 2021

Many thanks for this interesting observation and links, @belm0.

List + bisect sounds like a good first alternative to the skiplist and shouldn't be too difficult to implement (?).

Perhaps if there's demand for increased performance at the window sizes you mention, SortedList can be used. I quite like the fact that rolling has no third-party dependencies yet, but there's no requirement to keep it this way if there's a good reason to use one.

ajcr added a commit that referenced this issue Feb 5, 2023
Add "sortedlist" option for faster rolling median (#26)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants