ei@^dZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z n #e $rdZ YnwxYwddl mZddlmZddlmZddlmZejeZdZdZeeegdfZeeeeefgdfZeegdfZeeegdfZ egeeeeffZ!dd ed eeeeffd Z"d eed efdZ#deed efdZ$deeeefd eefdZ%ded efdZ&dZ'dZ(dZ)GddZ*dS)aShared file sync manager for remote execution backends. Tracks local file changes via mtime+size, detects deletions, and syncs to remote environments transactionally. Used by SSH, Modal, and Daytona. Docker and Singularity use bind mounts (live host FS view) and don't need this. N)Path)Callable)get_hermes_home)_file_mtime_keyg@HERMES_FORCE_FILE_SYNC /root/.hermescontainer_basereturnctddlm}m}m}g}|D]<}|dd|d}||d|f=||D]%}||d|df&||D]%}||d|df&|S)a[Enumerate all files that should be synced to a remote environment. Combines credentials, skills, and cache into a single flat list of (host_path, remote_path) pairs. Credential paths are remapped from the hardcoded /root/.hermes to *container_base* because the remote user's home may differ (e.g. /home/daytona, /home/user). r)get_credential_file_mountsiter_cache_filesiter_skills_filescontainer_pathr host_path)r )tools.credential_filesr r rreplaceappend)r r r rfilesentryremotes >/home/div/.hermes/hermes-agent/tools/environments/file_sync.pyiter_sync_filesr+s# $&E++--33'(00 ^Q    eK(&12222"".AAADD eK(%0@*ABCCCC!!@@@DD eK(%0@*ABCCCC L remote_pathscFddd|DzS)zK|]}tj|VdSNshlexquote).0ps r z$quoted_rm_command..Js*DD!u{1~~DDDDDDrjoin)rs rquoted_rm_commandr(Hs( chhDD|DDDDD DDrdirscFddd|DzS)z>Build a shell ``mkdir -p`` command for a batch of directories.z mkdir -p rc3>K|]}tj|VdSrr )r#ds rr%z'quoted_mkdir_command..Os*!?!?Q%+a..!?!?!?!?!?!?rr&)r)s rquoted_mkdir_commandr-Ms( !?!?$!?!?!??? ??rrc4td|DS)zCExtract sorted unique parent directories from (host, remote) pairs.cVh|]&\}}tt|j'S)strrparentr#_rs r z%unique_parent_dirs..Ts-CCC 63tF||*++CCCr)sorted)rs runique_parent_dirsr7Rs CCUCCC D DDrpathctj}t|d5tfddD]}|| dddn #1swxYwY|S)z$Return hex SHA-256 digest of a file.rbc.dS)Ni)read)fsrz_sha256_file..[s!&&--rrN)hashlibsha256openiterupdate hexdigest)r8hchunkr=s @r _sha256_filerGWsA dD  Q////55  E HHUOOOO  ;;==s,AA"%A")lceZdZdZeddfdedededede dzde dzf d Z d d d e d dfdZ ddedzd dfdZded dfdZded dfdZddZ ddedeeeefdzd edzfdZ ddedeeeefdzd edzfdZdS)FileSyncManageruTracks local file changes and syncs to a remote environment. Backends instantiate this with transport callbacks (upload, delete) and a file-source callable. The manager handles mtime-based change detection, deletion tracking, rate limiting, and transactional state. Not used by bind-mount backends (Docker, Singularity) — those get live host FS views and don't need file sync. N get_files_fn upload_fn delete_fn sync_intervalbulk_upload_fnbulk_download_fnc||_||_||_||_||_i|_i|_d|_||_dS)Ng) _get_files_fn _upload_fn_bulk_upload_fn_bulk_download_fn _delete_fn _synced_files_pushed_hashes_last_sync_time_sync_interval)selfrNrOrPrQrRrSs r__init__zFileSyncManager.__init__psP*#-!1#;=.0&)+rF)forcer`r c|sLtjts(t j}||jz |jkrdS|}d|Dg}t|j }|D]R\}}t|}||j ||kr6| ||f|||<Sfd|j D} |s| st j|_dSt|j } t|j } |r(tdt!|| r(tdt!|  |rE|j>||tdt!|n:|D]7\}}|||td||8| r0|| td| |D]\}}t)||j |<| D]3} || d|j | d4||_ t j|_dS#t,$rL} | |_ | |_ t j|_td | Yd} ~ dSd} ~ wwxYw) aTRun a sync cycle: upload changed files, delete removed files. Rate-limited to once per ``sync_interval`` unless *force* is True or ``HERMES_FORCE_FILE_SYNC=1`` is set. Transactional: state only committed if ALL operations succeed. On failure, state rolls back so the next cycle retries everything. Nch|]\}}|Sr0r0r3s rr5z'FileSyncManager.sync..sFFF91fFFFrcg|]}|v| Sr0r0)r#r$current_remote_pathss r z(FileSyncManager.sync..s$TTT1a?S6S6SQ6S6S6Srzfile_sync: uploading %d file(s)z+file_sync: deleting %d stale remote file(s)z#file_sync: bulk-uploaded %d file(s)zfile_sync: uploaded %s -> %szfile_sync: deleted %sz-file_sync: sync failed, rolled back state: %s)osenvironget_FORCE_SYNC_ENVtime monotonicr\r]rUdictrZrrr[loggerdebuglenrWrVrYrGpop Exceptionwarning)r^r`now current_files to_upload new_filesr remote_pathfile_key to_delete prev_files prev_hashesr$excrds @rsynczFileSyncManager.syncsV RZ^^O<< .""CT))D,???**,, FF FFF,. +,, &3 . . "I{&y11H!%%k22h>>   i5 6 6 6%-Ik " "UTTT 2TTT   #'>#3#3D F$,-- 4.//  L LL:C NN K K K  X LLFI W W W Q YT1=$$Y/// BC NNSSSS.7YY*I{OOI{;;;LL!?KXXXX A *** 4i@@@+4 K K& ;3? 3J3J#K00 1 1 a&&&#''40000!*D #'>#3#3D  Q Q Q!+D "-D #'>#3#3D NNJC P P P P P P P P P  Qs D'J22 L|jdS|js#|jstddS|p t dz }|jddd}ttD]} | |dS#t$r[}|}|tdz krAt|}t d|dz||tj|Yd}~{d}~wwxYwt dt|dS) aPull remote changes back to the host filesystem. Downloads the remote ``.hermes/`` directory as a tar archive, unpacks it, and applies only files that differ from what was originally pushed (based on SHA-256 content hashes). Protected against SIGINT (defers the signal until complete) and serialized across concurrent gateway sandboxes via file lock. Nu+sync_back: no prior push state — skippingz .sync.lockT)parentsexist_okrz2sync_back: attempt %d failed (%s), retrying in %dsz%sync_back: all %d attempts failed: %s)rXr[rZrmrnrr2mkdirrange_SYNC_BACK_MAX_RETRIES_sync_back_oncerq_SYNC_BACK_BACKOFFrrrjsleep)r^r~ lock_pathlast_excattemptr|delays r sync_backzFileSyncManager.sync_backsJ  ! ) F " 4+=  LLF G G G F 5O$5$5E td;;;%)344 & &G &$$Y/// & & &3a777.w7ENNL! S%Ju%%% & >@VX`aaaaas<B C9AC44C9rctjtju}gd}|rBtjtj}fd}tjtj| |||rU|Utjtj|r6tjtj tjdSdSdSdS#|rT|Stjtj|r3tjtj tjwwwwxYw)z>Single sync-back attempt with SIGINT protection and file lock.Ncj||ftddS)Nz/sync_back: SIGINT deferred until sync completes)rrmrn)signumframedeferred_sigints r _defer_sigintz6FileSyncManager._sync_back_once.._defer_sigints4&&777 NOOOOOr) threadingcurrent_thread main_threadsignal getsignalSIGINT_sync_back_lockedrfkillgetpid)r^ron_main_threadoriginal_handlerrrs @rrzFileSyncManager._sync_back_oncesT #133y7L7N7NN(*  8%/ >>  P P P P P M&- 7 7 7 8  " "9 - - - 8"2"> fm-=>>>"8GBIKK77777 8 8">">88 8"2"> fm-=>>>"8GBIKK7777 8">8s 1C##AD=ct|dSt|d} tj|tj|tj|tj|dS#tj|tj|wxYw)z;Sync-back under file lock (serializes concurrent gateways).Nw)fcntl_sync_back_implrAflockLOCK_EXLOCK_UNclose)r^rlock_fds rrz!FileSyncManager._sync_back_lockeds =  " " " Fy#&&  K / / /  " " " K / / / MMOOOOO K / / / MMOOOOs 3B5C c |jtd t|}n#t$rg}YnwxYwt jd5}|t|j tj |j}n#t$rd}YnwxYw|tkr/td|t ddddSt jd5}t#j|j5}||d dddn #1swxYwYd}tj|D]r\}}} | D]g} tj || } tj | |} d | z} |j| }|t3| }||krznd}|| |}|4|| |}|td | tj |r2|0t3|}||krtd | tjtj |dtAj!| ||dz }it|rt"d|ntddddn #1swxYwYddddS#1swxYwYdS)z1Download, diff, and apply remote changes to host.Nz/_sync_back_impl called without bulk_download_fnz.tar)suffixruBsync_back: remote tar is %d bytes (cap %d) — skipping extractionzhermes-sync-back-)prefixdata)filter/z(sync_back: skipping %s (no host mapping)uwsync_back: conflict on %s — host modified since push, remote also changed. Applying remote version (last-write-wins).T)rrz%sync_back: applied %d changed file(s)z%sync_back: no remote changes detected)#rX RuntimeErrorlistrUrqtempfileNamedTemporaryFilernamerfr8getsizeOSError_SYNC_BACK_MAX_BYTESrmrrTemporaryDirectorytarfilerA extractallwalkr'relpathr[rhrG_resolve_host_path_infer_host_pathrnexistsmakedirsdirnameshutilcopy2info)r^ file_mappingtftar_sizestagingtarapplieddirpath _dirnames filenamesfname staged_filerelrw pushed_hash remote_hashr host_hashs rrzFileSyncManager._sync_back_impl!s  ! )PQQ Q  2 2 4 455LL   LLL  ( 7 7 7A J2  " "4== 1 1 1 7??2733    ...X2A JA JA JA JA JA JA JA J ,4GHHH1 JG\"'**;cNN76N:::;;;;;;;;;;;;;;;57WW5E5E'%'%1GY !*&%&%&(gll7E&B&B  gook7CC&)Ci &*&9&=&=k&J&J '2*6{*C*CK*k99 ( :+/K%)$;$;K$V$V $,(,(=(=k<(X(XI(0 & $N$/!"!"!"!)7>>)44"9P(4Y(?(?I(K77 &%H%0 !"!"!" BGOOI$>$>NNNN [)<<<1 M&%PJKK GQQQQLL!HIIIc1 J1 J1 J1 J1 J1 J1 J1 J1 J1 J1 J1 J1 J1 J1 J!A JA JA JA JA JA JA JA JA JA JA JA JA JA JA JA JA JA Js!: A A !(L? $B/.L?/ B>;L?=B>>0L?;L?L'*E L'E L'E GL' L?'L+ +L?.L+ /L??MMrwrc6||ng}|D]\}}||kr|cSdS)zAFind the host path for a known remote path from the file mapping.Nr0)r^rwrmappinghostrs rrz"FileSyncManager._resolve_host_pathosC#/":,,#  LD&$$ %trc||ng}|D]}\}}tt|j}||dzr?tt|j}|t |d}||zcS~dS)uInfer a host path for a new remote file by matching path prefixes. Uses the existing file mapping to find a remote->host directory pair, then applies the same prefix substitution to the new file. For example, if the mapping has ``/root/.hermes/skills/a.md`` → ``~/.hermes/skills/a.md``, a new remote file at ``/root/.hermes/skills/b.md`` maps to ``~/.hermes/skills/b.md``. Nr)r1rr2 startswithro) r^rwrrrr remote_dirhost_dirrs rrz FileSyncManager._infer_host_pathxs#/":,,# ) )LD&T&\\011J%%j3&677 )tDzz011$S__%5%56&(((( )trr)r N)__name__ __module__ __qualname____doc___SYNC_INTERVAL_SECONDS GetFilesFnUploadFnDeleteFnfloat BulkUploadFnBulkDownloadFnr_boolr}rrrrrr1rtuplerrr0rrrMrMes 6.226,, ,, ,  , %t+ ,)4/,,,,&%*IQIQIQTIQdIQIQIQIQ^&b&bTD[&bD&b&b&b&bP88$88882 4 D    LJLJLJLJ^IMc)-eCHo)>)EQTW[Q[GKC'+E#s(O'rs  LLLL EEE,,,,,,333333  8 $ $* S#J$ %eCHo./56 4&$,' T#YK% & b$uS#X// 0 Cd5c?>S:EDIE#EEEE @tCy@S@@@@ Ed5c?3ES EEEE ss-dddddddddds 1;;