chunking functions now accept new docs as parameters instread of tokens #113
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rangehow
requested changes
Dec 26, 2024
Comment on lines
+11
to
+29
| from ._utils import clean_str | ||
| from ._utils import compute_mdhash_id | ||
| from ._utils import decode_tokens_by_tiktoken | ||
| from ._utils import encode_string_by_tiktoken | ||
| from ._utils import is_float_regex | ||
| from ._utils import list_of_list_to_csv | ||
| from ._utils import logger | ||
| from ._utils import pack_user_ass_to_openai_messages | ||
| from ._utils import split_string_by_multi_markers | ||
| from ._utils import truncate_list_by_token_size | ||
| from .base import BaseGraphStorage | ||
| from .base import BaseKVStorage | ||
| from .base import BaseVectorStorage | ||
| from .base import CommunitySchema | ||
| from .base import QueryParam | ||
| from .base import SingleCommunitySchema | ||
| from .base import TextChunkSchema | ||
| from .prompt import GRAPH_FIELD_SEP | ||
| from .prompt import PROMPTS |
There was a problem hiding this comment.
Please make this part of the code cleaner.
|
From my personal perspective, the scope of this PR is much broader than the custom length mentioned in #111. This PR moves most of the logic from the user-invisible |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fix #111