staging · x86_64-linux · 8e1af4ab17b2 · generated 2026-11-10T01:02:05Z

python314Packages.pytorch-tokenizers 1.5.0 single outdated

C++ implementations for various tokenizers (sentencepiece, tiktoken, etc.)

Name
python3.14-pytorch-tokenizers-1.5.0
Package set
python314Packages
Homepage
https://github.com/meta-pytorch/tokenizers
License
BSD-3-Clause
Source
pkgs/development/python-modules/pytorch-tokenizers/default.nix:102
Derivation
/nix/store/bhag99x3lkr94fam2i7rb30amv7nqwpp-python3.14-pytorch-tokenizers-1.5.0.drv
Repology
outdated, newest version is 1.6.0 · python:pytorch-tokenizers (as of 2026-11-10T01:00:17Z)

Maintainers 1

HandleNameContactVia
GaetanLepage Gaetan Lepage GitHub · gaetan@glepage.com listed on package

Dependencies 23

PackageAsMaintainersUsed by (transitive)Status
python3 3.14.10 propagatedBuildInputs 71,771 team-only
python314Packages.sentencepiece 0.2.1 propagatedBuildInputs 59 single outdated
python314Packages.tiktoken 0.12.0 propagatedBuildInputs 210 single outdated
python314Packages.tokenizers 0.23.3 propagatedBuildInputs 182 single
ensureNewerSourcesForZipFilesHook 26.05pre-git nativeBuildInputs
74,250 unmaintained setup hook
python313Packages.pythonOutputDistHook 26.05pre-git nativeBuildInputs
74,218 unmaintained setup hook
python313Packages.pythonRemoveBinBytecodeHook 26.05pre-git nativeBuildInputs
74,238 unmaintained setup hook
python314Packages.cmake 4.4.4 nativeBuildInputs
13,771 unmaintained
python314Packages.pybind11 3.1.0 nativeBuildInputs 9,673 single
python314Packages.pypaBuildHook 26.05pre-git nativeBuildInputs
70,722 unmaintained setup hook
python314Packages.pypaInstallHook 26.05pre-git nativeBuildInputs
70,948 unmaintained setup hook
python314Packages.pytestCheckHook 26.05pre-git nativeBuildInputs
69,251 unmaintained setup hook
python314Packages.pythonCatchConflictsHook 26.05pre-git nativeBuildInputs
70,944 unmaintained setup hook
python314Packages.pythonImportsCheckHook 26.05pre-git nativeBuildInputs
70,969 unmaintained setup hook
python314Packages.pythonMetadataCheckHook 26.05pre-git nativeBuildInputs 70,710 single setup hook
python314Packages.pythonNamespacesHook 26.05pre-git nativeBuildInputs
70,967 unmaintained setup hook
python314Packages.pythonRemoveTestsDirHook 26.05pre-git nativeBuildInputs
70,967 unmaintained setup hook
python314Packages.pythonRuntimeDepsCheckHook 26.05pre-git nativeBuildInputs
70,916 unmaintained setup hook
python314Packages.setuptools 83.0.0 nativeBuildInputs 70,580 team-only outdated
python314Packages.transformers 5.22.1 nativeBuildInputs 114 maintained outdated
python314Packages.wrapPython 26.05pre-git nativeBuildInputs
70,973 unmaintained setup hook
bashNonInteractive 5.3p20 other input 102,212 maintained
stdenv 26.05pre-git other input
90,021 team-only

Used by 1 directly, 1 transitively

PackageAsMaintainersUsed by (transitive)Status
python314Packages.executorch 1.5.1 propagatedBuildInputs 0 single outdated