Skip to content

Issues: huggingface/tokenizers

ByteLevelBPETokenizer output seems weird
#203 opened Mar 24, 2020 by seyyaw
Open 2
Training a model from in-memory data
#198 by loicbarrault was closed Nov 28, 2020
Closed 1
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

59 Open 961 Closed
59 Open 961 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Memory leak is observed when using the AutoTokenizer and AutoModel with Python 3.10.*
#1706 opened Dec 27, 2024 by KhoiTrant68
2 of 4 tasks
DecodeStream raise error
#1705 opened Dec 26, 2024 by irexyc
batch_encode_plus doesn't work correctly
#1704 opened Dec 18, 2024 by tempdeltavalue
1
Cannot find package 'tokenizers-linux-x64-musl' - Alpine support
#1703 opened Dec 14, 2024 by PylotLight
if split_special_tokens==True，fast_tokenizer is slower than slow_tokenizer
#1700 opened Dec 12, 2024 by gongel
Tokenizer Training Errors: pyo3_runtime.PanicException: called Result::unwrap() on an Err value: TryFromIntError(())
#1698 opened Dec 10, 2024 by Chimaco37
1
Request for pre-tokenizer that creates words based length alone.
#1697 opened Dec 10, 2024 by filbeofITK
How to determine the splicing logic in post_processor based on the sentence to be tokenized?
#1696 opened Dec 5, 2024 by gongel
Bug: is_pretokenized is not used when calling tokenizer.encode(...)
#1695 opened Nov 29, 2024 by jannessm
Rust Issue on Unix in 0.21.0 Version
#1694 opened Nov 28, 2024 by insculptor
0.20.4 Partial release?
#1690 opened Nov 26, 2024 by tazarov
6
Llama-3.2 offset-mapping needs fixing
#1688 opened Nov 26, 2024 by kyrawilson
1
Question: Shrinking Tokenizer Vocabulary for Reduced Memory Consumption with Pre-Trained Model (LLaMA) Fine-Tuning
#1686 opened Nov 23, 2024 by Amerehei
Prebuilding tokenizers for Windows arm
#1684 opened Nov 20, 2024 by hariji814
1
wikitext-103-raw-v1.zip is not available on the amazonaws anymore
#1683 opened Nov 18, 2024 by gec1-dev
Mismatch between slow and fast tokenizer
#1682 opened Nov 15, 2024 by KaiLv69
2
out of memory when training a BBPE tokenizer on a large corpus
#1681 opened Nov 14, 2024 by yucc-leon
2
Option to disable cache for FromPretrained and FromFile Feature Request
#1680 opened Nov 12, 2024 by daulet
2
Reduce vocab size for BPE tokenizer Feature Request
#1668 opened Oct 29, 2024 by fzyzcjy
8
docs-check.yml uses node12 which is deprecated
#1658 opened Oct 17, 2024 by hamirmahal
1
1
Allow users to select/write encoding strategies Feature Request
#1655 opened Oct 16, 2024 by pietrolesci
2
Serializing k-mer style pre-tokenizer
#1654 opened Oct 15, 2024 by millanp95
Inconsistent behaviour of PreTrainedTokenizerFasts on diacritics marked texts bug Something isn't working
#1663 opened Oct 11, 2024 by sven-nm
2 of 4 tasks
1
Disable pretty-print when saving tokenizer.json files Feature Request
#1656 opened Oct 7, 2024 by xenova
1
How to build a custom tokenizer on top of a exsiting Llama 3.2 tokenizer? training
#1644 opened Oct 5, 2024 by yakhyo
7
ProTip! Find all open issues with in progress development work with linked:pr.