#i& hdZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z ejeZej"dZej"dZdZdZd Zej"d Zej"d Zd ed e efdZd"dededed efdZde ed e eeffdZde ed e eeffdZeefde ededed e eeffdZde eefd e eeffdZ dZ!dede eefd efdZ"dede eefd efdZ#de eefded dfdZ$ded e eeffd Z% d#d e ee&ffd!Z'y)$aDictionary-based compression using auto-learned codebooks. Scans workspace memory files, learns high-frequency n-grams, builds a codebook mapping long phrases to short `$XX` codes, and applies/reverses substitutions for lossless compression. Part of claw-compactor. License: MIT. N)Counter)Path)DictListOptionalSetTuplez \$[A-Z]{2,3}z(\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\bz(/[A-Za-z0-9_.~-]+){3,}nreturnc g}tdD]U}tdD]E}|jdtd|zztd|zzt||k\sA|ccSWtdD]v}tdD]f}tdD]V}|jdtd|zztd|zztd|zzt||k\sP|cccShx|S)zAGenerate *n* unique short codes: $AA..$ZZ, then $AAA.. if needed.$A)rangeappendchrlen)r codesijks :/home/crogers2287/claw-compactor/scripts/lib/dictionary.py_generate_codesr%sE 2Yr A LLs26{*Sa[8 95zQ   2Y!r !A2Y ! S3rAv;.R!VExtract word n-grams from *text*, filtering by minimum length. )rsplitrrjoinMIN_PHRASE_LEN)rrr counterwordsr rgrams r_tokenize_ngramsr*8syG  JJLE 5%!) $#s5zA~)* #A88E!AEN+D4yN* "  ## Nrtextsc"t}|D]Q}tj|D]7}|jd}dj |dddz}||xxdz cc<9S|j Dcic]\}}|dk\s ||c}}Scc}}w)z?Find frequently occurring IP prefixes (3-octet) across *texts*..Nr r")r_IP_REfindallr$r%items)r+r'rippartsprefixcounts r_extract_ip_prefixesr6FsyG!..& !BHHSMEXXeBQi(3.F FOq O !! 07}} Mmfe%1*FEM MM Ms 4 B B cg}|D];}tj|D]!}|j|j#=t |dkriSt }|D]N}|j d}tdt |D]#}dj|d|}||xxdz cc<%P|jD cic]\}} | dk\s || c} }Scc} }w)zNFind frequently occurring path prefixes (directory components) across *texts*.r./r Nr") _PATH_REfinditerrgrouprrr$rr%r1) r+ all_pathsrmr'pathr3depthr4r5s r_extract_path_prefixesr@QsI(""4( (A   QWWY ' (( 9~ yG! 31c%j) !EXXeFUm,F FOq O !!07}} Mmfe%1*FEM MM Ms  CCmin_freq max_entriescb|siSt}|D]}|jt|t|}|j D]7\}}t |t k\st|j|d|||<9t|}|j D]7\} }t | t k\st|j| d||| <9|j D cgc] \} }||k\rt | t k\r| |f"} } }| jddttt | |} i} t}t| | D]G\\} }}d}|D]}| |vs|| vs d}n|r!| | |<|j| t | |k\sF| S| Scc}} w)zBuild a codebook from a list of text documents. Scans for high-frequency n-grams, IPs, and paths. Returns a dict mapping short codes ($XX) to the phrases they replace. rc*|dt|dzS)Nr"rrxs rz build_codebook..s!A$QqT"2rT)keyreverseF)rupdater*r6r1rr&maxgetr@sortrminsetzipadd)r+rArBcombinedrip_freqsr2r5 path_freqsr>phrase candidatesrcodebook used_phrases_countcodeskipexistings rbuild_codebookr^gs  yH0(./0$E*H^^%; E r7n $x||B2E:HRL;(.J!'')? e t9 & dA!6>HTN?&^^- FE H V!> J OO2DOA CJ= >E!H UL"%j%"8 $$ H!X%7    x=K '  O  O5s-%F+rXc|siStt|}|jdr|S|jDcic]\}}|| c}}Scc}}w)zNormalize codebook to {code: phrase} format. Accepts either {code: phrase} or {phrase: code} format. Detects format by checking if keys start with '$'. r)nextiter startswithr1)rX first_keyrVr[s r_normalize_codebookrdsR  T(^$IC 2:1ABf BBBs AzDLRc|r|s|St|}|jdt}t|j dD]-\}}|jdt}|j||}/|S)zApply codebook substitutions to *text*. Lossless. Accepts codebook in either {code: phrase} or {phrase: code} format. Pre-existing '$' characters are escaped so they survive roundtrip. rc t|d S)Nr"rErFs rrHzcompress_text..#ad)rrI)rdreplace_DOLLAR_ESCAPEsortedr1)rrX normalizedresultr[rVescaped_phrases r compress_textross x $X.J \\#~ .Fz//17KL6 f^<56 Mrc|r|s|St|}|}t|jdD]\}}|j||}|jtd}|S)z{Reverse codebook substitutions. Lossless. Accepts codebook in either {code: phrase} or {phrase: code} format. c t|d S)NrrErFs rrHz!decompress_text..rgrrhr)rdrkr1rirj)rrXrlrmr[rVs rdecompress_textrrsh x $X.J Fz//17KL. ff-.^^NC 0F Mrr>ct|}|jjddd|d}|jt j |ddd y ) zSave codebook to a JSON file.T)parentsexist_okr")versionentriesr.F)indent ensure_asciiutf-8encodingN)rparentmkdir write_textjsondumps)rXr>datas r save_codebookrsI :DKKdT2X .DOODJJtAEBWOUrct|}|jstd|tj|j d}t |trd|vrtd||dS)zLoad codebook from a JSON file.zCodebook not found: rzr{rwzInvalid codebook format: ) rexistsFileNotFoundErrorrloads read_text isinstancedict ValueError)r>rs r load_codebookrsn :D ;;="6tf =>> ::dnngn6 7D dD !Yd%:4TF;<<  ?rc* ||}| t|}t }nnt|trVt|trF|}dj|j }|}t || t|}t }nddddddS|r ||z |z dznd}t |}t fd|D} td|jD} ||z | z } |r| |z dznd} ||t|dt| dt|| d S) uCalculate compression statistics. Can be called as: compression_stats(texts_dict, codebook) — where texts_dict maps filenames to content compression_stats(original_str, compressed_str, codebook)  rg)original_charscompressed_charsgross_reduction_pctcodebook_entries codes_useddc3,K|] }|vsd yw)r"N).0r[ compresseds r z$compression_stats.. sD41CQDs c3VK|]!\}}t|t|zdz#yw)r.NrE)rrvs rrz$compression_stats..s%ODAqCFSVOa/Os')r.)rrrnet_reduction_pctrr) rrrr%valuesrordsumr1round)texts_or_originalcodebook_or_compressedrXoriginalorig_lencomp_lenall_text reductionrlrcodebook_overhead net_saved net_reductionrs @rcompression_statsrs:$+ x=z? %t ,rs  33   8 $ 2::o &rzz/*    ? @ 2::0 1stCy& 3 s s 7 NS Nd38nNN$s)NS#XN0#8 9888 #s(^ 8vC$sCx.CT#s(^C"tCH~#$#c3hC"VDcNV$V4Vc3h>B- #v+-r