[DRAFT] Allow customization of conversion to relative paths #324
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ashvina
commented
Feb 5, 2024
| } | ||
|
|
||
| @VisibleForTesting | ||
| protected void initializeFilters(String tableDataPath) { |
There was a problem hiding this comment.
TODO: Add test to validate customization
| String newPath = oldPath.substring(basePath.length() + 1); | ||
|
|
||
| // create a new data file | ||
| return OneDataFile.builder() |
There was a problem hiding this comment.
TODO: is there an easier way to clone an object using lombok such that some fields can be updated?
ashvina
changed the title
ashvina
force-pushed
the
323-conversion-to-relative-paths
branch
from
1a7cccb to
8d6dc0c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses #323
The DeltaClient target converts a dataFile's absolute path to a relative path by default. This behavior is usually desirable, as relative paths are more compatible with query engines. However, the behavior is also fixed and cannot be changed. This can be problematic for users who want to keep absolute paths or handle path schemes differently. To address this issue, this change introduces the ability to inject custom code and allow for customization of the path conversion process.
This change introduces a FilterManager that transforms the input list of data files using a set of FilterChains. The user can configure the FilterChains (add list of filters) to apply different transformations to the data files. For example, one FilterChain can convert absolute paths to relative paths, and another can emit telemetry data for the data files. The user can also add new implementations of FilterChains or remove all of them.
The FilterManager can be used to inject the FilterChains at various points in the conversion process. These filters will be added incrementally.
A simpler way to initialize the filters with user configuration will be possible after the PerTableConfig related changes are done.
This change added new tests.