Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supports pipeline event based scheduling. #54259

Open
1 task done
stdpain opened this issue Dec 24, 2024 · 0 comments · Fixed by #54260 · May be fixed by #54338
Open
1 task done

Supports pipeline event based scheduling. #54259

stdpain opened this issue Dec 24, 2024 · 0 comments · Fixed by #54260 · May be fixed by #54338

Comments

@stdpain
Copy link
Contributor

stdpain commented Dec 24, 2024

Feature request

Currently our pipeline uses poller mode for scheduling. This leads to some of the following problems.

  1. An unintended blocking implementation may result in all queries not being scheduled in a timely manner.
  2. poller may not be scheduled in time.
  3. The pipeline poller consumes an entire CPU core, resulting in a waste of resources.

Describe the solution you'd like

The main purpose of this PR (#49972) is to remove pollers through the observer notify mechanism, and we will continue this work and make better improvements.

It can save one CPU core in the scenarios such as point query/TPCDS.

image

In the new implementation, we don't need a poller thread at all, or even a block driver queue. And there is no need to introduce additional locks during scheduling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants