Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add auto analyze service tserver and master flags #25097

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions docs/content/preview/reference/configuration/yb-master.md
Original file line number Diff line number Diff line change
Expand Up @@ -987,6 +987,18 @@ expensive when the number of yb-tservers, or the number of databases goes up.

{{< /note >}}

## Auto Analyze Service flags

Auto Analyze service flags are {{<tags/feature/tp>}}.
yifanguan marked this conversation as resolved.
Show resolved Hide resolved

yifanguan marked this conversation as resolved.
Show resolved Hide resolved
See also [Auto Analyze Service TServer flags](../yb-tserver/#auto-analyze-service-flags).

##### ysql_enable_auto_analyze_service

Enable the Auto Analyze service which automatically runs ANALYZEs to update table statistics for tables which have changed more than a configurable threshold.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Enable the Auto Analyze service which automatically runs ANALYZEs to update table statistics for tables which have changed more than a configurable threshold.
Enable the Auto Analyze service, which automatically runs ANALYZE to update table statistics for tables that have changed more than a configurable threshold.


Default: false

## Advanced flags

##### --allowed_preview_flags_csv
Expand Down
67 changes: 67 additions & 0 deletions docs/content/preview/reference/configuration/yb-tserver.md
Original file line number Diff line number Diff line change
Expand Up @@ -1624,6 +1624,73 @@ After a DDL statement that includes updating DocDB system catalog completes, YB-
When the flag `ysql_ddl_transaction_wait_for_ddl_verification` is enabled, YSQL waits for any YB-Master background operations to finish before returning control to the user.
{{< /note >}}

## Auto Analyze Service flags

Auto Analyze service flags are {{<tags/feature/tp>}}.
yifanguan marked this conversation as resolved.
Show resolved Hide resolved

yifanguan marked this conversation as resolved.
Show resolved Hide resolved
{{< note title="Note" >}}

To fully enable Auto Analyze service, you need to enable `ysql_enable_auto_analyze_service` on all YB-Masters and YB-TServers, as well as `ysql_enable_table_mutation_counter` on all YB-TServers.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To fully enable Auto Analyze service, you need to enable `ysql_enable_auto_analyze_service` on all YB-Masters and YB-TServers, as well as `ysql_enable_table_mutation_counter` on all YB-TServers.
To fully enable the Auto Analyze service, you need to enable `ysql_enable_auto_analyze_service` on all YB-Masters and YB-TServers, and `ysql_enable_table_mutation_counter` on all YB-TServers.


{{< /note >}}

See also [Auto Analyze Service Master flags](../yb-master#auto-analyze-service-flags).

##### ysql_enable_auto_analyze_service

Enable the Auto Analyze service which automatically runs ANALYZEs to update table statistics for tables which have changed more than a configurable threshold.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Enable the Auto Analyze service which automatically runs ANALYZEs to update table statistics for tables which have changed more than a configurable threshold.
Enable the Auto Analyze service, which automatically runs ANALYZE to update table statistics for tables that have changed more than a configurable threshold.


Default: false

##### ysql_enable_table_mutation_counter
Enable counting of mutations on a per-table basis. These mutations are used to automatically run ANALYZE as soon as the mutations of a table cross
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Enable counting of mutations on a per-table basis. These mutations are used to automatically run ANALYZE as soon as the mutations of a table cross
Enable mutation counting on a per-table basis. Mutations are used by the Auto Analyze service to run ANALYZE when the number of mutations of a table exceeds

a certain threshold (decided based on [ysql_auto_analyze_threshold](#ysql-auto-analyze-threshold) and [ysql_auto_analyze_scale_factor](#ysql-auto-analyze-scale-factor)).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
a certain threshold (decided based on [ysql_auto_analyze_threshold](#ysql-auto-analyze-threshold) and [ysql_auto_analyze_scale_factor](#ysql-auto-analyze-scale-factor)).
the threshold determined by the [ysql_auto_analyze_threshold](#ysql-auto-analyze-threshold) and [ysql_auto_analyze_scale_factor](#ysql-auto-analyze-scale-factor) settings.


Default: false

##### ysql_auto_analyze_threshold

The minimum number of mutations needed to run ANALYZE on a table.

Default: 50

##### ysql_auto_analyze_scale_factor

The fraction defining when sufficient mutations have been accumulated to run ANALYZE for a table.
ANALYZE runs when the mutation count becomes larger than `ysql_auto_analyze_scale_factor * <table_size> + ysql_auto_analyze_threshold`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ANALYZE runs when the mutation count becomes larger than `ysql_auto_analyze_scale_factor * <table_size> + ysql_auto_analyze_threshold`.
ANALYZE runs when the mutation count exceeds `ysql_auto_analyze_scale_factor * <table_size> + ysql_auto_analyze_threshold`.


Default: 0.1

##### ysql_auto_analyze_batch_size

The max number of tables the auto analyze service tries to analyze in a single ANALYZE statement.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The max number of tables the auto analyze service tries to analyze in a single ANALYZE statement.
The maximum number of tables the Auto Analyze service tries to analyze in a single ANALYZE statement.


Default: 10

##### ysql_cluster_level_mutation_persist_interval_ms

Interval at which the reported node level table mutation counts are persisted to the underlying auto-analyze mutations table.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the "auto-analyze mutations table"


Default: 10000

##### ysql_cluster_level_mutation_persist_rpc_timeout_ms

Timeout for the rpcs used to persist mutation counts in the auto-analyze mutations table.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Timeout for the rpcs used to persist mutation counts in the auto-analyze mutations table.
Timeout for the RPCs used to persist mutation counts in the auto-analyze mutations table.


Default: 10000

##### ysql_node_level_mutation_reporting_interval_ms

Interval at which the node level table mutation counts are sent to the auto analyze service which tracks table mutation counts at the cluster level.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Interval at which the node level table mutation counts are sent to the auto analyze service which tracks table mutation counts at the cluster level.
Interval, in milliseconds, at which the node-level table mutation counts are sent to the Auto Analyze service, which tracks table mutation counts at the cluster level.


Default: 5000

##### ysql_node_level_mutation_reporting_timeout_ms

Timeout for the node level mutation reporting rpc to auto-analyze service.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Timeout for the node level mutation reporting rpc to auto-analyze service.
Timeout, in milliseconds, for the node-level mutation reporting RPC to the Auto Analyze service.


Default: 5000

## Advanced flags

##### backfill_index_client_rpc_timeout_ms
Expand Down