staging · x86_64-linux · 8e1af4ab17b2 · generated 2026-11-10T01:02:05Z

python314Packages.tokenizers 0.23.3 single

Fast State-of-the-Art Tokenizers optimized for Research and Production

Name
python3.14-tokenizers-0.23.3
Package set
python314Packages
Homepage
https://github.com/huggingface/tokenizers
License
Apache-2.0
Source
pkgs/development/python-modules/tokenizers/default.nix:148
Derivation
/nix/store/8xf20gd652pzz0qsm1idz8b0bpjrif90-python3.14-tokenizers-0.23.3.drv
Repology
not reported as outdated · look up (as of 2026-11-10T01:00:17Z)

Maintainers 1

HandleNameContactVia
GaetanLepage Gaetan Lepage GitHub · gaetan@glepage.com listed on package

Dependencies 29

PackageAsMaintainersUsed by (transitive)Status
python3 3.14.10 propagatedBuildInputs 71,771 team-only
python314Packages.huggingface-hub 2.4.0 propagatedBuildInputs 215 maintained
oniguruma 6.9.10 buildInputs 4,279 single
openssl 3.5.9 buildInputs 76,574 maintained
cargo 1.99.1 nativeBuildInputs 29,289 team-only
ensureNewerSourcesForZipFilesHook 26.05pre-git nativeBuildInputs
74,250 unmaintained setup hook
pkg-config 0.29.2 nativeBuildInputs
77,688 unmaintained
python313Packages.pythonOutputDistHook 26.05pre-git nativeBuildInputs
74,218 unmaintained setup hook
python313Packages.pythonRemoveBinBytecodeHook 26.05pre-git nativeBuildInputs
74,238 unmaintained setup hook
python314Packages.datasets 5.1.0 nativeBuildInputs 189 single outdated
python314Packages.numpy 2.5.4 nativeBuildInputs 11,879 single
python314Packages.pypaBuildHook 26.05pre-git nativeBuildInputs
70,722 unmaintained setup hook
python314Packages.pypaInstallHook 26.05pre-git nativeBuildInputs
70,948 unmaintained setup hook
python314Packages.pytest-asyncio 1.4.1 nativeBuildInputs 66,695 single
python314Packages.pytestCheckHook 26.05pre-git nativeBuildInputs
69,251 unmaintained setup hook
python314Packages.pythonCatchConflictsHook 26.05pre-git nativeBuildInputs
70,944 unmaintained setup hook
python314Packages.pythonImportsCheckHook 26.05pre-git nativeBuildInputs
70,969 unmaintained setup hook
python314Packages.pythonMetadataCheckHook 26.05pre-git nativeBuildInputs 70,710 single setup hook
python314Packages.pythonNamespacesHook 26.05pre-git nativeBuildInputs
70,967 unmaintained setup hook
python314Packages.pythonRemoveTestsDirHook 26.05pre-git nativeBuildInputs
70,967 unmaintained setup hook
python314Packages.pythonRuntimeDepsCheckHook 26.05pre-git nativeBuildInputs
70,916 unmaintained setup hook
python314Packages.requests 2.34.2 nativeBuildInputs 41,215 single
python314Packages.setuptools-rust 1.13.0 nativeBuildInputs
7,947 unmaintained
python314Packages.tiktoken 0.12.0 nativeBuildInputs 210 single outdated
python314Packages.wrapPython 26.05pre-git nativeBuildInputs
70,973 unmaintained setup hook
rustc 1.99.1 nativeBuildInputs 32,538 team-only
writableTmpDirAsHomeHook 26.05pre-git nativeBuildInputs
64,758 unmaintained setup hook
bashNonInteractive 5.3p20 other input 102,212 maintained
stdenv 26.05pre-git other input
90,021 team-only

Used by 34 directly, 182 transitively

PackageAsMaintainersUsed by (transitive)Status
python314Packages.transformers 5.22.1 propagatedBuildInputs 114 maintained outdated
python314Packages.litellm 1.108.0 propagatedBuildInputs 51 maintained
python314Packages.faster-whisper 1.2.1 propagatedBuildInputs 19 single
python314Packages.cohere 7.0.4 propagatedBuildInputs 18 single outdated
python314Packages.sentence-transformers 6.2.0 propagatedBuildInputs
14 unmaintained
python314Packages.chromadb 1.5.9 propagatedBuildInputs 6 maintained
python314Packages.pyopenjtalk-plus 0.4.1-post9 nativeBuildInputs 5 single outdated
python314Packages.curated-transformers 2.0.1 propagatedBuildInputs 3 single
aider-chat 0.86.1 propagatedBuildInputs 1 maintained outdated
python314Packages.exllamav3 1.6.6 propagatedBuildInputs 1 single broken
python314Packages.mflux 0.15.4 propagatedBuildInputs 1 single outdated
python314Packages.pytorch-tokenizers 1.5.0 propagatedBuildInputs 1 single outdated
python314Packages.unclecode-litellm 1.81.13 propagatedBuildInputs 1 single
aider-chat-full 0.86.1 propagatedBuildInputs 0 maintained
aider-chat-with-bedrock 0.86.1 propagatedBuildInputs 0 maintained
aider-chat-with-browser 0.86.1 propagatedBuildInputs 0 maintained
aider-chat-with-help 0.86.1 propagatedBuildInputs 0 maintained
aider-chat-with-playwright 0.86.1 propagatedBuildInputs 0 maintained
crewai 1.15.18 propagatedBuildInputs 0 single
datatrove 0.2.0 propagatedBuildInputs 0 single
immich-machine-learning 3.3.6 propagatedBuildInputs 0 maintained
litellm 1.108.0 propagatedBuildInputs 0 maintained outdated
paperless-ngx 3.3.4 other input 0 maintained outdated
pocket-tts 4.0.1 propagatedBuildInputs 0 single
python314Packages.crawl4ai 0.9.1 nativeBuildInputs 0 single outdated
python314Packages.crewai 1.15.18 propagatedBuildInputs 0 single outdated
python314Packages.fastembed 0.8.0 propagatedBuildInputs 0 single outdated
python314Packages.guidance 0.3.1 nativeBuildInputs 0 single
python314Packages.langchain-huggingface 1.2.3 propagatedBuildInputs 0 maintained
python314Packages.langchain-mistralai 1.1.6 propagatedBuildInputs 0 single
python314Packages.pocket-tts 4.0.1 propagatedBuildInputs 0 single
python314Packages.torchtitan 0.3.0 propagatedBuildInputs 0 single outdated
python314Packages.transformers_4 4.57.6 propagatedBuildInputs 0 maintained broken
tabbyapi 0-unstable-2026-11-01 propagatedBuildInputs 0 single