nixos-unstable · x86_64-linux · 151fa4e8ddfd · generated 2026-10-07T01:01:54Z

python314Packages.tokenizers 0.23.2 single

Fast State-of-the-Art Tokenizers optimized for Research and Production

Name
python3.14-tokenizers-0.23.2
Package set
python314Packages
Homepage
https://github.com/huggingface/tokenizers
License
Apache-2.0
Source
pkgs/development/python-modules/tokenizers/default.nix:147
Derivation
/nix/store/h8k3280lps99y74vgjny38mlxfgc833h-python3.14-tokenizers-0.23.2.drv

Maintainers 1

HandleNameContactVia
GaetanLepage Gaetan Lepage GitHub · gaetan@glepage.com listed on package

Dependencies 28

PackageAsMaintainersUsed by (transitive)Status
python3 3.14.7 propagatedBuildInputs
mweinelt
 72,537 team-only
python314Packages.huggingface-hub 1.30.0 propagatedBuildInputs
GaetanLepage, osbm
 211 maintained
openssl 3.5.8 buildInputs
andir, balsoft, pyrox0, thillux
 76,863 maintained
cargo 1.98.1 nativeBuildInputs
Mic92, winterqt
 28,718 team-only
ensureNewerSourcesForZipFilesHook 26.05pre-git nativeBuildInputs
74,596 unmaintained setup hook
pkg-config 0.29.2 nativeBuildInputs
78,016 unmaintained
python313Packages.pythonOutputDistHook 26.05pre-git nativeBuildInputs
74,564 unmaintained setup hook
python313Packages.pythonRemoveBinBytecodeHook 26.05pre-git nativeBuildInputs
74,584 unmaintained setup hook
python314Packages.datasets 4.5.0 nativeBuildInputs
osbm
 188 single
python314Packages.numpy 2.5.2 nativeBuildInputs
doronbehar
 11,439 single
python314Packages.pypaBuildHook 26.05pre-git nativeBuildInputs
71,473 unmaintained setup hook
python314Packages.pypaInstallHook 26.05pre-git nativeBuildInputs
71,694 unmaintained setup hook
python314Packages.pytest-asyncio 1.4.0 nativeBuildInputs
dotlambda
 68,242 single
python314Packages.pytestCheckHook 26.05pre-git nativeBuildInputs
70,075 unmaintained setup hook
python314Packages.pythonCatchConflictsHook 26.05pre-git nativeBuildInputs
71,690 unmaintained setup hook
python314Packages.pythonImportsCheckHook 26.05pre-git nativeBuildInputs
71,715 unmaintained setup hook
python314Packages.pythonMetadataCheckHook 26.05pre-git nativeBuildInputs
dotlambda
 71,460 single setup hook
python314Packages.pythonNamespacesHook 26.05pre-git nativeBuildInputs
71,713 unmaintained setup hook
python314Packages.pythonRemoveTestsDirHook 26.05pre-git nativeBuildInputs
71,713 unmaintained setup hook
python314Packages.pythonRuntimeDepsCheckHook 26.05pre-git nativeBuildInputs
71,662 unmaintained setup hook
python314Packages.requests 2.34.2 nativeBuildInputs
fabaff
 43,259 single
python314Packages.setuptools-rust 1.13.0 nativeBuildInputs
7,694 unmaintained
python314Packages.tiktoken 0.12.0 nativeBuildInputs
happysalada
 209 single
python314Packages.wrapPython 26.05pre-git nativeBuildInputs
71,719 unmaintained setup hook
rustc 1.98.1 nativeBuildInputs
Mic92, winterqt
 30,080 team-only
writableTmpDirAsHomeHook 26.05pre-git nativeBuildInputs
66,113 unmaintained setup hook
bashNonInteractive 5.3p15 other input
andir, balsoft, infinisil, pyrox0
 102,441 maintained
stdenv 26.05pre-git other input
90,330 team-only

Used by 34 directly, 181 transitively

PackageAsMaintainersUsed by (transitive)Status
python314Packages.transformers 5.17.0 propagatedBuildInputs
GaetanLepage, happysalada, pashashocky
 114 maintained
python314Packages.litellm 1.102.1 propagatedBuildInputs
happysalada
 50 single
python314Packages.faster-whisper 1.2.1 propagatedBuildInputs
mweinelt
 18 single
python314Packages.cohere 7.0.4 propagatedBuildInputs
natsukium
 16 single
python314Packages.sentence-transformers 6.1.0 propagatedBuildInputs
14 unmaintained
python314Packages.chromadb 1.5.9 propagatedBuildInputs
fabaff, sarahec
 6 maintained
python314Packages.pyopenjtalk-plus 0.4.1-post9 nativeBuildInputs
jcaesar
 5 single
python314Packages.curated-transformers 2.0.1 propagatedBuildInputs
danieldk
 3 single
aider-chat 0.86.1 propagatedBuildInputs
happysalada, yzx9
 1 maintained
python314Packages.exllamav3 1.5.4 propagatedBuildInputs
BatteredBunny
 1 single broken
python314Packages.mflux 0.15.4 propagatedBuildInputs
GaetanLepage
 1 single
python314Packages.pytorch-tokenizers 1.5.0 propagatedBuildInputs
GaetanLepage
 1 single
python314Packages.unclecode-litellm 1.81.13 propagatedBuildInputs
fabaff
 1 single
aider-chat-full 0.86.1 propagatedBuildInputs
happysalada, yzx9
 0 maintained
aider-chat-with-bedrock 0.86.1 propagatedBuildInputs
happysalada, yzx9
 0 maintained
aider-chat-with-browser 0.86.1 propagatedBuildInputs
happysalada, yzx9
 0 maintained
aider-chat-with-help 0.86.1 propagatedBuildInputs
happysalada, yzx9
 0 maintained
aider-chat-with-playwright 0.86.1 propagatedBuildInputs
happysalada, yzx9
 0 maintained
crewai 1.15.18 propagatedBuildInputs
liberodark
 0 single
datatrove 0.2.0 propagatedBuildInputs
luftmensch-luftmensch
 0 single
immich-machine-learning 3.2.4 propagatedBuildInputs
diogotcorreia, dotlambda, jvanbruegge, Scrumplex, titaniumtown
 0 maintained
litellm 1.102.1 propagatedBuildInputs
happysalada
 0 single
paperless-ngx 3.2.1 other input
erikarvstedt, leona-ya, SuperSandro2000
 0 maintained
pocket-tts 3.3.0 propagatedBuildInputs
GaetanLepage
 0 single
python314Packages.crawl4ai 0.9.1 nativeBuildInputs
fabaff
 0 single
python314Packages.crewai 1.15.18 propagatedBuildInputs
liberodark
 0 single
python314Packages.fastembed 0.8.0 propagatedBuildInputs
happysalada
 0 single
python314Packages.guidance 0.3.1 nativeBuildInputs
natsukium
 0 single
python314Packages.langchain-huggingface 1.2.2 propagatedBuildInputs
natsukium, sarahec
 0 maintained
python314Packages.langchain-mistralai 1.1.6 propagatedBuildInputs
sarahec
 0 single
python314Packages.pocket-tts 3.3.0 propagatedBuildInputs
GaetanLepage
 0 single
python314Packages.torchtitan 0.3.0 propagatedBuildInputs
GaetanLepage
 0 single
python314Packages.transformers_4 4.57.6 propagatedBuildInputs
happysalada, pashashocky
 0 maintained broken
tabbyapi 0-unstable-2026-09-28 propagatedBuildInputs
BatteredBunny
 0 single