staging · x86_64-linux · 8e1af4ab17b2 · generated 2026-11-10T01:02:05Z

python313Packages.tokenizers 0.23.3 single

Fast State-of-the-Art Tokenizers optimized for Research and Production

Name
python3.13-tokenizers-0.23.3
Package set
python313Packages
Homepage
https://github.com/huggingface/tokenizers
License
Apache-2.0
Source
pkgs/development/python-modules/tokenizers/default.nix:148
Derivation
/nix/store/kspfc23v8mhybrk1a7a0d36qss8hg15r-python3.13-tokenizers-0.23.3.drv
Repology
not reported as outdated · look up (as of 2026-11-10T01:00:17Z)

Maintainers 1

HandleNameContactVia
GaetanLepage Gaetan Lepage GitHub · gaetan@glepage.com listed on package

Dependencies 29

PackageAsMaintainersUsed by (transitive)Status
python313 3.13.18 propagatedBuildInputs 14,168 team-only
python313Packages.huggingface-hub 2.4.0 propagatedBuildInputs 175 maintained
oniguruma 6.9.10 buildInputs 4,279 single
openssl 3.5.9 buildInputs 76,574 maintained
cargo 1.99.1 nativeBuildInputs 29,289 team-only
ensureNewerSourcesForZipFilesHook 26.05pre-git nativeBuildInputs
74,250 unmaintained setup hook
pkg-config 0.29.2 nativeBuildInputs
77,688 unmaintained
python313Packages.datasets 5.1.0 nativeBuildInputs 149 single outdated
python313Packages.numpy 2.5.4 nativeBuildInputs 5,321 single
python313Packages.pypaBuildHook 26.05pre-git nativeBuildInputs
11,689 unmaintained setup hook
python313Packages.pypaInstallHook 26.05pre-git nativeBuildInputs
11,717 unmaintained setup hook
python313Packages.pytest-asyncio 1.4.1 nativeBuildInputs 8,226 single
python313Packages.pytestCheckHook 26.05pre-git nativeBuildInputs
10,197 unmaintained setup hook
python313Packages.pythonCatchConflictsHook 26.05pre-git nativeBuildInputs
11,701 unmaintained setup hook
python313Packages.pythonImportsCheckHook 26.05pre-git nativeBuildInputs
11,734 unmaintained setup hook
python313Packages.pythonMetadataCheckHook 26.05pre-git nativeBuildInputs 11,689 single setup hook
python313Packages.pythonNamespacesHook 26.05pre-git nativeBuildInputs
11,726 unmaintained setup hook
python313Packages.pythonOutputDistHook 26.05pre-git nativeBuildInputs
74,218 unmaintained setup hook
python313Packages.pythonRemoveBinBytecodeHook 26.05pre-git nativeBuildInputs
74,238 unmaintained setup hook
python313Packages.pythonRemoveTestsDirHook 26.05pre-git nativeBuildInputs
11,726 unmaintained setup hook
python313Packages.pythonRuntimeDepsCheckHook 26.05pre-git nativeBuildInputs
11,703 unmaintained setup hook
python313Packages.requests 2.34.2 nativeBuildInputs 6,780 single
python313Packages.setuptools-rust 1.13.0 nativeBuildInputs
5,174 unmaintained
python313Packages.tiktoken 0.12.0 nativeBuildInputs 162 single outdated
python313Packages.wrapPython 26.05pre-git nativeBuildInputs
11,727 unmaintained setup hook
rustc 1.99.1 nativeBuildInputs 32,538 team-only
writableTmpDirAsHomeHook 26.05pre-git nativeBuildInputs
64,758 unmaintained setup hook
bashNonInteractive 5.3p20 other input 102,212 maintained
stdenv 26.05pre-git other input
90,021 team-only

Used by 23 directly, 138 transitively

PackageAsMaintainersUsed by (transitive)Status
python313Packages.transformers 5.22.1 propagatedBuildInputs 90 maintained outdated
python313Packages.litellm 1.108.0 propagatedBuildInputs 38 maintained
python313Packages.cohere 7.0.4 propagatedBuildInputs 8 single outdated
python313Packages.sentence-transformers 6.2.0 propagatedBuildInputs
8 unmaintained
python313Packages.faster-whisper 1.2.1 propagatedBuildInputs 6 single
python313Packages.chromadb 1.5.9 propagatedBuildInputs 3 maintained
python313Packages.vllm 0.33.0 propagatedBuildInputs 3 maintained
python313Packages.curated-transformers 2.0.1 propagatedBuildInputs 2 single
python313Packages.pytorch-tokenizers 1.5.0 propagatedBuildInputs 1 single outdated
python313Packages.unclecode-litellm 1.81.13 propagatedBuildInputs 1 single
python313Packages.crawl4ai 0.9.1 nativeBuildInputs 0 single outdated
python313Packages.crewai 1.15.18 propagatedBuildInputs 0 single outdated
python313Packages.exllamav3 1.6.6 propagatedBuildInputs 0 single broken
python313Packages.fastembed 0.8.0 propagatedBuildInputs 0 single outdated
python313Packages.guidance 0.3.1 nativeBuildInputs 0 single
python313Packages.langchain-huggingface 1.2.3 propagatedBuildInputs 0 maintained
python313Packages.langchain-mistralai 1.1.6 propagatedBuildInputs 0 single
python313Packages.mflux 0.15.4 propagatedBuildInputs 0 single outdated
python313Packages.pocket-tts 4.0.1 propagatedBuildInputs 0 single
python313Packages.pyopenjtalk-plus 0.4.1-post9 nativeBuildInputs 0 single outdated
python313Packages.torchtitan 0.3.0 propagatedBuildInputs 0 single outdated
python313Packages.transformers_4 4.57.6 propagatedBuildInputs 0 maintained broken
vllm 0.33.0 propagatedBuildInputs 0 maintained