Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature scalar regexp match benchmark #13789

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open

Feature scalar regexp match benchmark #13789

wants to merge 28 commits into from

Conversation

zhuliquan
Copy link
Contributor

Which issue does this PR close?

Closes #.

Rationale for this change

What changes are included in this PR?

Adding benchmark for scalar regex matching, I pick 6 cases for benchmark:
1、email
2、ip
3、phnone number
4、html tag
5、url
6、format date string ('yyyy-MM-dd')

Are these changes tested?

Are there any user-facing changes?

zhuliquan and others added 27 commits December 11, 2024 23:22
@zhuliquan
bench: scalar regex match benchmark
ecd4793
@crepererum @zhuliquan
refactor: migrate LinearSearch to HashTable (apache#13658)
c3e0951 
For apache#13433.
@comphead @zhuliquan
Minor: Comment temporary function for documentation migration (apache…
1e507ad 
…#13669)

* Minor: Comment temporary function for documentation migration

* Minor: Comment temporary function for documentation migration
@comphead @alamb @zhuliquan
Minor: Rephrase MSRV policy to be more explanatory (apache#13668)
61fd077 
* Minor: Rephrase MSRV policy to be more explanatory

Co-authored-by: Andrew Lamb <[email protected]>

* MSRV policy update

---------

Co-authored-by: Andrew Lamb <[email protected]>
@korowa @zhuliquan
fix: repartitioned reads of CSV with custom line terminator (apache#1…
67260a0 
…3677)
@findepi @zhuliquan
chore: macros crate cleanup (apache#13685)
3618cfe 
* Remove unused dependencies from macros crate

* rename macro lib to user_doc
@jiashenC @zhuliquan
Refactor regexplike signature (apache#13394)
d3e0860 
* update

* update

* update

* clean up errors

* fix flags types

* fix failed example
@alamb @zhuliquan
Performance: enable array allocation reuse (ScalarFunctionArgs gets…
a960c6d 
… owned `ColumnReference`) (apache#13637)

* Improve documentation

* Pass owned args to ScalarFunctionArgs

* Update advanced_udf with example of reusing arrays

* clarify rationale for cloning

* clarify comments

* fix expected output
@jonahgao @zhuliquan
Temporary fix for CI (apache#13689)
d39852d
@jonahgao @comphead @zhuliquan
refactor: use LazyLock in the user_doc macro (apache#13684)
98372cc 
* refactor: use `LazyLock` in the `user_doc` macro

* Fix cargo doc

* Update datafusion/macros/src/lib.rs

* Fix doc comment

---------

Co-authored-by: Oleks V <[email protected]>
@Alexhuszagh @zhuliquan
Unlock lexical-write-integer version. (apache#13693)
e8226f5 
Issue was patched as of lexical release 1.0.5.

Reverts apache#13689
Closes apache#13686
@akurmustafa @zhuliquan
Minor: Use div_ceil
bd91271
@haohuaijin @zhuliquan
Fix hash join with sort push down (apache#13560)
45926ab 
* fix: join with sort push down

* chore:
insert some value

* apply suggestion

* recover handle_costom_pushdown change

* apply suggestion

* add more test

* add partition
@richox @zhuliquan
Improve substr() performance by avoiding using owned string (apache#1…
16d2ab1 
…3688)

Co-authored-by: zhangli20 <[email protected]>
@andygrove @zhuliquan
reinstate down_cast_any_ref (apache#13705)
d8c9cfb
@tlm365 @zhuliquan
Optimize performance of character_length function (apache#13696)
f8c0efe 
* Optimize performance of  function

Signed-off-by: Tai Le Manh <[email protected]>

* Add pre-check array is null

* Fix clippy warnings

---------

Signed-off-by: Tai Le Manh <[email protected]>
@dependabot @zhuliquan
Update prost-build requirement from =0.13.3 to =0.13.4 (apache#13698)
2d8bd42 
Updates the requirements on [prost-build](https://github.com/tokio-rs/prost) to permit the latest version.
- [Release notes](https://github.com/tokio-rs/prost/releases)
- [Changelog](https://github.com/tokio-rs/prost/blob/master/CHANGELOG.md)
- [Commits](tokio-rs/prost@v0.13.3...v0.13.4)

---
updated-dependencies:
- dependency-name: prost-build
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@comphead @zhuliquan
Minor: Output elapsed time for sql logic test (apache#13718)
5dc6e42 
* Minor: Output elapsed time for sql logic test
@jonahgao @zhuliquan
refactor: simplify the make_udf_function macro (apache#13712)
4fb9d2a
@Weijun-H @zhuliquan
refactor: replace Vec with IndexMap for expression mappings in `P…
fa0440b 
…rojectionMapping` and `EquivalenceGroup` (apache#13675)

* refactor: replace Vec with IndexMap for expression mappings in ProjectionMapping and EquivalenceGroup

* chore

* chore: Fix CI

* chore: comment

* chore: simplify
@Eason0729 @zhuliquan
Handle alias when parsing sql(parse_sql_expr) (apache#12939)
d3c459e 
* fix: Fix parse_sql_expr not handling alias

* cargo fmt

* fix parse_sql_expr example(remove alias)

* add testing

* add SUM udaf to TestContextProvider and modify test_sql_to_expr_with_alias for function

* revert change on example `parse_sql_expr`
@alamb @zhuliquan
Improve documentation for TableProvider (apache#13724)
ddfc9e5
@findepi @zhuliquan
Reveal implementing type and return type in simple UDF implementations (
b494157 
apache#13730)

Debug trait is useful for understanding what something is and how it's
configured, especially if the implementation is behind dyn trait.
@alamb @zhuliquan
minor: Extract tests for EXTRACT AND date_part to their own file (a…
3b5daa2 
…pache#13731)
@goldmedal @zhuliquan
Support unparsing UNNEST plan to UNNEST table factor SQL (apache#…
50ce883 
…13660)

* add `unnest_as_table_factor` and `UnnestRelationBuilder`

* unparse unnest as table factor

* fix typo

* add tests for the default configs

* add a static const for unnest_placeholder

* fix tests

* fix tests
@zhuliquan
Merge branch 'apache:main' into feature-scalar_regexp_match_benchmark
13b581a
@zhuliquan
Merge branch 'apache:main' into feature-scalar_regexp_match_benchmark
065eb47
@github-actions github-actions bot added the core Core DataFusion crate label Dec 15, 2024
@zhuliquan zhuliquan mentioned this pull request Dec 15, 2024
Open
@Omega359
Copy link
Contributor

I ran your benchmark and it looks good. I am unsure though the benefit of this benchmark over the existing 'regx' benchmark which targets the udf versions of the Postgresql symbols - most of the logic is in the regexp::regexp_is_match function for both.

@zhuliquan
Copy link
Contributor Author

I ran your benchmark and it looks good. I am unsure though the benefit of this benchmark over the existing 'regx' benchmark which targets the udf versions of the Postgresql symbols - most of the logic is in the regexp::regexp_is_match function for both.

Emm, regx benchmark is aim at regex functions, this benchmark is aim at PR #12270

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
core Core DataFusion crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.