#i8dZddlZddlZddlmZej eZdZdZ ddl Z e jdZdZ ejdd Zd Zej"d Zd ed efdZd ed efdZd efdZy#e ef$rejdYGwxYw)aToken estimation utilities. Uses tiktoken when available, falls back to a CJK-aware heuristic. For the heuristic: - ASCII/Latin text: ~4 chars per token - CJK characters: ~1.5 chars per token (tiktoken cl100k_base) Part of claw-compactor. License: MIT. N)OptionalFzgpt-4Tz.tiktoken available, using cl100k_base encodingz/tiktoken unavailable, using CJK-aware heuristicg?z6[\u4e00-\u9fff\u3400-\u4dbf\u3000-\u303f\uff00-\uffef]textreturnc|syttj|}t||z }|tz }|tz }t dt ||zS)z{Estimate tokens using CJK-aware heuristic. CJK characters are counted at ~1.5 chars/token, everything else at ~4. r)len_CJK_REfindallCJK_CHARS_PER_TOKENCHARS_PER_TOKENmaxint)r cjk_chars other_chars cjk_tokens other_tokenss 6/home/crogers2287/claw-compactor/scripts/lib/tokens.py_heuristic_tokensr$sW GOOD)*Id)i'K00J0L q#j</0 11c| td|sytr$tttj |St |S)zEstimate the number of tokens in *text*. Uses tiktoken (cl100k_base) when available, otherwise a CJK-aware heuristic. Returns 0 for empty strings. Raises TypeError if *text* is None. z-estimate_tokens() requires a string, got Noner) TypeError_tiktoken_available_encoderr encoder)rs restimate_tokensr2sB |GHH x38??4()) T ""rctS)z5Return True if tiktoken is being used for estimation.)rrrusing_tiktokenrBs r)__doc__reloggingtypingr getLogger__name__loggerrrtiktokenencoding_for_modeldebug ImportError Exceptionr r compiler strrrrboolrrrrr/s     8 $ D*x**73H LLAB "**N O 2C 2C 2 ## ## # Q YD LLBCDs)A>>BB