eiS dZddlZddlZddlZddlZddlZddlmZmZm Z m Z ddl Z ddl m Z ddlmZmZmZddlmZddlmZmZmZddlmZmZdd lmZdd lmZej e!Z"d e#d e$fd Z%d e&fdZ'd e#fdZ(de#d e$fdZ)da*da+d e e,ee#e#fe,e#e e#e e#fffdZ-d e#fdZ.d e$fdZ/d e$fdZ0ddZ1d e#fdZ2d e3e#fdZ4dZ5da6da7dZ8dZ9ej:ddZ;de#de&d e&fd Zd&e d e fd'Z?d(e d eee#e ffd)Z@d!e d eee#e ffd*ZAd+e d ee#e ffd,ZBd-ZCd.e d e$fd/ZDdd0e e#d e,e e e e#ee#e fffd1ZEd e e#fd2ZFed3d45ZGd#d#deCfd6e#d7e#d8e#d0e e#d9eHd e e#f d:ZI dd6e#d=e#d0e e#d>eHd?e$d@e#d e e#fdAZJd6e#d=e#d0e e#dBeHdCeHd e e#f dDZKdEe#d e#fdFZLdaMdGZNddIe#dJeHd e&fdKZOdLee#d eee#e ffdMZPddIe#dJeHd e&fdOZQdLee#d eee#e ffdPZRddIe#dJeHd e#fdQZSddRdeCfdLee#dSe#dTe$d0e e#d9eHd e#f dUZTddVdRdeCfd7e#dWe#dXe#dTe$d0e e#d9eHd e#fdYZUd e$fdZZVd e$fd[ZWd e$fd\ZXe!d]kr eYd^eYd_eWZZe/Z[e$ej:d`d#\Z]e$ej:dad#\Z^eXZ_eFZ`eZre(ZaeYdbeaeadckr eYddneadekr eYdfneadgkr eYdhne^rBeYdiej:da\bdjnTe]r eYdknFe[reYdle.n-eYdmn!eYdneYdoe2e_s"eYdpeYdqeYdrneYdse`eZs ecdteYdue_reYdve`eYdweCdxeGjdr0eYdyeGjeeYdzeGjfd{eGjed|n eYd}eYd~eYdeYdeYd#eYdeYdeYd#eYdeYdeYdeYdeYde_reYdeYdeYdeYd#eYdeYdeYdeYdeYdeYdeYdeYd#eYdeYdeYdeYdeYdeYdeYdeYdeYdeYdeYdeYdddlgmhZhmiZiddddIdddidIgddZjddddLdddiddNdidLgddZkehjlddejdeWe4ddehjlddekdeWe4dRdd dS)a% Standalone Web Tools Module This module provides generic web tools that work with multiple backend providers. Backend is selected during ``hermes tools`` setup (web.backend in config.yaml). When available, Hermes can route Firecrawl calls through a Nous-hosted tool-gateway for Nous Subscribers only. Available tools: - web_search_tool: Search the web for information - web_extract_tool: Extract content from specific web pages - web_crawl_tool: Crawl websites with specific instructions Backend compatibility: - Exa: https://exa.ai (search, extract) - Firecrawl: https://docs.firecrawl.dev/introduction (search, extract, crawl; direct or derived firecrawl-gateway. for Nous Subscribers) - Parallel: https://docs.parallel.ai (search, extract) - Tavily: https://tavily.com (search, extract, crawl) LLM Processing: - Uses OpenRouter API with Gemini 3 Flash Preview for intelligent content extraction - Extracts key excerpts and creates markdown summaries to reduce token usage Debug Mode: - Set WEB_TOOLS_DEBUG=true to enable detailed logging - Creates web_tools_debug_UUID.json in ./logs directory - Captures all tool calls, results, and compression metrics Usage: from web_tools import web_search_tool, web_extract_tool, web_crawl_tool # Search the web results = web_search_tool("Python machine learning libraries", limit=3) # Extract content from URLs content = web_extract_tool(["https://example.com"], format="markdown") # Crawl a website crawl_data = web_crawl_tool("example.com", "Find contact information") N)ListDictAnyOptional) Firecrawl)async_call_llmextract_content_or_reasoningget_async_text_auxiliary_client) DebugSession)build_vendor_gateway_urlread_nous_access_tokenresolve_managed_tool_gateway)managed_nous_tools_enabledprefers_gateway) is_safe_url)check_website_accessnamereturncptj|}t|o|SN)osgetenvboolstrip)rvals 1/home/div/.hermes/hermes-agent/tools/web_tools.py_has_envrGs* )D//C # $ $$c~ ddlm}|diS#ttf$ricYSwxYw)z5Load the ``web:`` section from ~/.hermes/config.yaml.r load_configweb)hermes_cli.configr!get ImportError Exceptionr s r_load_web_configr'Ks^111111{}}  +++  # s #&<<cztdpd}|dvr|Sdt dpt dp t fdt dfd t d fd t d ff}|D] \}}|r|cS dS) zDetermine which web backend to use. Reads ``web.backend`` from config.yaml (set by ``hermes tools``). Falls back to whichever API key is present for users who configured keys manually without running setup. backend)parallel firecrawltavilyexar,FIRECRAWL_API_KEYFIRECRAWL_API_URLr+PARALLEL_API_KEYr-TAVILY_API_KEYr. EXA_API_KEY)r'r$lowerrr_is_tool_gateway_ready) configuredbackend_candidatesr) availables r _get_backendr9Ss#$$((339r@@BBHHJJJ??? h233px@S7T7TpXnXpXpq X0112 8,--. ''(  1  NNN  ;rr)c|dkrtdS|dkrtdS|dkrtS|dkrtdSdS) z:Return True when the selected backend is currently usable.r.r3r+r1r,r-r2F)rcheck_firecrawl_api_key)r)s r_is_backend_availabler<nsk% &&&**++++&((((())) 5rctjdd}tjddd}|s|sdSi}|r||d<|r||d<|d|pd|pdffS) zHReturn explicit direct Firecrawl kwargs + cache key, or None when unset.r/r*r0/Napi_keyapi_urldirect)rrrrstrip)r?r@kwargss r_get_direct_firecrawl_configrDsi+R006688Gi+R006688??DDG 7tF$#y$#y Hgow$? ??rc tdS)z(Return configured Firecrawl gateway URL.r,)r rr_get_firecrawl_gateway_urlrGs #K 0 00rc2tdtduS)zGReturn True when gateway URL and a Nous Subscriber token are available.r, token_readerN)r_read_nous_access_tokenrFrrr5r5s ' BY Z Z Zbf ffrc"tduS)zBReturn True when direct Firecrawl config is explicitly configured.N)rDrFrr_has_direct_firecrawl_configrMs ' ) ) 55rcJd}tr|dz }t|)z>Raise a clear error for unsupported web backend configuration.zWeb tools are not configured. Set FIRECRAWL_API_KEY for cloud Firecrawl or set FIRECRAWL_API_URL for a self-hosted Firecrawl instance.u With your Nous subscription you can also use the Tool Gateway — run `hermes tools` and select Nous Subscription as the web provider.)r ValueError)messages r&_raise_web_backend_configuration_errorrQs= s "##  S  W  rc(tsdS dS)zAReturn optional managed-gateway guidance for Firecrawl help text.r*zc, or use the Nous Tool Gateway via your subscription (FIRECRAWL_GATEWAY_URL or TOOL_GATEWAY_DOMAIN))rrFrr_firecrawl_backend_help_suffixrSs$ % ' 'r 9rcXgd}tr|gd|S)zEReturn tool metadata env vars for the currently enabled web backends.)r3r1r2r/r0)FIRECRAWL_GATEWAY_URLTOOL_GATEWAY_DOMAINTOOL_GATEWAY_SCHEMETOOL_GATEWAY_USER_TOKEN)rextend)requiress r_web_requires_envr[sOH"##        Orcft}|tds|\}}n_tdt}|(tdt |j|jd}d|d|jf}tt|krtStd i|a |a tS) zGet or create Firecrawl client. When ``web.use_gateway`` is set in config, the Tool Gateway is preferred even if direct Firecrawl credentials are present. Otherwise direct Firecrawl takes precedence when explicitly configured. Nr"r,rIzTFirecrawl client initialization failed: missing direct config and tool-gateway auth.)r?r@z tool-gatewayr@rF) rDrrrKloggererrorrQnous_user_tokengateway_origin_firecrawl_client_firecrawl_client_configr) direct_configrC client_configmanaged_gateways r_get_firecrawl_clientrfs122M )?)? - 6 0     " LLo p p p 2 4 4 4'6&5    9   + $)A])R)R  !++F++, rcddlm}t1tjd}|st d||atS)zwGet or create the Parallel sync client (lazy initialization). Requires PARALLEL_API_KEY environment variable. r)ParallelNr1VPARALLEL_API_KEY environment variable not set. Get your API key at https://parallel.air?)r+rh_parallel_clientrrrO)rhr?s r_get_parallel_clientrlsf "!!!!!).// : $8G444 rcddlm}t1tjd}|st d||atS)zxGet or create the Parallel async client (lazy initialization). Requires PARALLEL_API_KEY environment variable. r) AsyncParallelNr1rirj)r+rn_async_parallel_clientrrrO)rnr?s r_get_async_parallel_clientrp sf '&&&&&%).// : "/w!?!?!? !!rTAVILY_BASE_URLzhttps://api.tavily.comendpointpayloadcJtjd}|std||d<td|d}t d||tj||d}| | S)zSend a POST request to the Tavily API. Auth is provided via ``api_key`` in the JSON body (no header-based auth). Raises ``ValueError`` if ``TAVILY_API_KEY`` is not set. r2z\TAVILY_API_KEY environment variable not set. Get your API key at https://app.tavily.com/homer?r>zTavily %s request to %s<)jsontimeout) rrrO_TAVILY_BASE_URLlstripr]infohttpxpostraise_for_statusrv)rrrsr?urlresponses r_tavily_requestr s i())G   >   !GI  6 6 4 4 6 6C KK)8S999z#GR888H  ==??rrc g}t|dgD]^\}}||dd|dd|dd|dzd_dd |id S) zNormalize Tavily /search response to the standard web search format. Tavily returns ``{results: [{title, url, content, score, ...}]}``. We map to ``{success, data: {web: [{title, url, description, position}]}}``. resultstitler*r~content)rr~ descriptionpositionTr"successdata) enumerater$append)r web_resultsiresults r _normalize_tavily_search_resultsr4s Kx||Ir::;;  6ZZ,,::eR((!::i44A       e[%9 : ::rr* fallback_urlc g}|dgD]}|d|}|ddp|dd}|||dd||||dddd|d gD]\}||d|ddd|d d d |d|id ]|dgD]F}t|tr|nt|}||dddd d |id G|S)zNormalize Tavily /extract or /crawl response to the standard document format. Maps results to ``{url, title, content, raw_content, metadata}`` and includes any ``failed_results`` / ``failed_urls`` as error entries. rr~ raw_contentr*rr sourceURLrr~rrrmetadatafailed_resultsr^extraction failedr)r~rrrr^r failed_urls)r$r isinstancestr) rr documentsrr~rawfailfail_urlurl_strs r_normalize_tavily_documentsrEs ')I,,y"--   jj --jj++Hvzz)R/H/HZZ,,&)FJJw4K4KLL        -r22  88E<00XXg':;;$dhhul&C&CD       LL33   (377J((S]]($g.        rvaluec l|dSt|ttttt t fr|St|dr% |S#t$rYnwxYwt|dr4 d|j DS#t$rYnwxYw|S)zBConvert SDK objects to plain python data structures when possible.N model_dump__dict__cDi|]\}}|d||S_ startswith).0kvs r z$_to_plain_object..}s0UUUTQ1<>H 1$((92E2EFFL $##(););E)B)BCC  N,^-?-? -J-JKK   xD%ghr&B&BCCC Ir scrape_resultct|}t|tsiS|d}t|tr|S|S)zINormalize Firecrawl scrape payload shape across SDK and gateway variants.r)rrrr$)r result_plainnesteds r_extract_scrape_payloadrsV#M22L lD ) )   f % %F&$ rclientcddlm}tt|ddpd}||jpd}|dkp|dS)z?Return True when the resolved auxiliary backend is Nous Portal.r)urlparsebase_urlr*znousresearch.comz.nousresearch.com) urllib.parserrrhostnamer4endswith)rrrhosts r_is_nous_auxiliary_clientrss%%%%%%76:r228b99H HX   ' -2 4 4 6 6D % % K7J)K)KKrmodelctd\}}tjdd}|p|p|}i}|$t |rddlm}|pddgi}|||fS) zHResolve the current web-extract auxiliary client, model, and extra body. web_extractAUXILIARY_WEB_EXTRACT_MODELr*Nr)get_auxiliary_extra_bodytagszproduct=hermes-agent)r rrrragent.auxiliary_clientr)rr default_modelconfigured_modeleffective_model extra_bodyrs r_resolve_web_extract_auxiliaryrs;MJJFMy!>CCIIKK@/@=O!#J 7??CCCCCC--//UF=S500k chars), uses chunked processing with synthesis. For extremely large content (>2M chars), refuses to process entirely. Args: content (str): The raw content to process url (str): The source URL (for context, optional) title (str): The page title (for context, optional) model (str): The model to use for processing (default: google/gemini-3-flash-preview) min_length (int): Minimum content length to trigger processing (default: 5000) Returns: Optional[str]: Processed markdown content, or None if content too short or processing fails ii 順ri@Bz 2MB limit). Refusing to process.z[Content too large to process: z.1fzdMB. Try using web_crawl with specific extraction instructions, or search for a more focused source.]z:Content too short (%d < %d chars), skipping LLM processingNzTitle: zSource:   r*z5Content large (%d chars). Using chunked processing...z+Processing content with LLM (%d characters)4 [... summary truncated for context management ...]r?z*Content processed: %d -> %d chars (%.1f%%)dzweb_extract LLM summarization failed (%s). Tip: increase auxiliary.web_extract.timeout in config.yaml or switch to a faster auxiliary model.xu' [Content truncated — showing first , of z chars. LLM summarization timed out. To fix: increase auxiliary.web_extract.timeout in config.yaml, or use a faster auxiliary model. Use browser_navigate for the full page.]) lenr]warningdebugrjoinrz_process_large_content_chunked_call_summarizer_llmr&r)rr~rrrMAX_CONTENT_SIZECHUNK_THRESHOLD CHUNK_SIZEMAX_OUTPUT_SIZE content_lensize_mb context_info context_strprocessed_contentprocessed_lengthcompression_ratioe truncateds rprocess_content_with_llmrs 8!OJO@'ll  ) ) )!I-G NNY[b c c cgWgggg g  # # LLUWbdn o o o4  3    1% 1 1 2 2 2  2    03 0 0 1 1 1:FNdii --66B   ( ( KKOQ\ ] ] ]7eZ   A;OOO"6w U"S"SSSSSSS  ~$%%77$56F6F$GKC%C! ##455 BMPQ// 0; > >WZ  KKDkScevy|e| } } }   5 FF4C4L    ,_,- w< %d charszChunk %d/%d failed: %s2)rrr]rzr&rr)r%r&rsummaryrchunksrrs rsummarize_chunkz7_process_large_content_chunked..summarize_chunksL #Oi!mOOVOOOJ0 % G D DiRSmUXY_U`U`befsbtbtvy{BwCwCDDDg% % # # # NN3Y]CKKQTUVQWQWX[Y[X[Q\ ] ] ]d? " " " " " " #sBB C!ACC!C!c.g|]\}}||SrFrF)rrchunkr,s r z2_process_large_content_chunked..s) I I I81e__Q & & I I IrNc|dS)NrrF)xs rz0_process_large_content_chunked..s AaDr)keyz ## Section rrzAll chunk summarizations failedzB[Failed to process large content: all chunk summarizations failed]zGot %d/%d chunk summariesz [... truncated ...]zSynthesizing %d summaries...z --- a'You have been given summaries of different sections of a large document. Synthesize these into ONE cohesive, comprehensive summary that: 1. Removes redundancy between sections 2. Preserves all key facts, figures, and actionable information 3. Is well-organized with clear structure 4. Is under z characters zSECTION SUMMARIES: z, Create a single, unified markdown summary.z9No auxiliary model for synthesis, concatenating summariesrrr zdYou synthesize multiple summaries into one cohesive, comprehensive summary. Be thorough but concise.r rrrrrz3Synthesis LLM returned empty content, retrying onceu>Synthesis failed after retry — concatenating chunk summariesrrz+Synthesis complete: %d -> %d chars (%.2f%%)rzSynthesis failed: %sz. [... truncated due to synthesis failure ...]rF)rrrr]rzrrtuplerrrgathersortedrrrrrr r&)rrrr"r#rr.tasksr summariesr%r*rcombined_summariessynthesis_promptrrrfallbackr r final_summary original_len final_len compressionrr+r,s `` @@rrrs,F 1c'llJ / /!j.() e KK8#f++zRRR##S#U3PXY\P]K]E^########( J I I Iy7H7H I I IENE*******GI$W..AAAGG 7  G   E9q=EEGEE F F F T 6777SS KK+S^^S[[III 9~~1 v;; ( (,_,-0IIF  KK.I???&++I66 .  . .  . . . . .72PQV2W2W/ OZ  _  NNV W W W{{9--H8}}..#$4_$458QQO"$!/UVV,<==     3(2K %'66+666666664X>>  C NNP Q Q Q+::k::::::::H8BBM  NN[ \ \ \{{9--H8}}..#$4_$458QQO }   / /)*:?*:;>vvM7||  && 2>2B2Bi,..  A."NNNAMB??NNNrzExa extract: %d URL(s)T)r@r*rr) r[rSr]rzrrM get_contentsrr@r~rr)r^rSrrrrr~rs r _exa_extractrgs /.....~ONNNNNN KK(#d))444  -- .H G"(b   +#jB ""&)E::        Nrc 6ddlm}|rdddStjdd}|dvrd}t d |||tj |g||t|d  }g}t|j pgD]P\}}|jpg}||jpd |jpd |rd |nd |dzdQdd|idS)z;Search using the Parallel SDK and return results as a dict.rrRrTFrUPARALLEL_SEARCH_MODEagentic)fastzone-shotrkz)Parallel search: '%s' (mode=%s, limit=%d))search_queries objectivemode max_resultsr*rYrrZTr"r)r[rSrrr4rr]rzrlbetar\rrrexcerptsrr~rr) rOrPrSrprrrrrss r_parallel_searchrtse......~:&5999 9+Y 7 7 = = ? ? E E G GD 222 KK;UD%PPP#%%*11w rNN 2HKx/5266  6?(b:#\'R19A388H---rA       e[%9 : ::rc BKddlm}|r d|DStdt |t j|dd{V}g}|jpgD]Y}|j pd}|sd |j pg}|j pd}|j pd}|||||||d d Z|jpgD];}||j pddd|jp|jpd d |j pdid<|S)zExtract content from URLs using the Parallel async SDK. Returns a list of result dicts matching the structure expected by the LLM post-processing pipeline (url, title, content, metadata). rrRcg|]}|ddd SrarFrcs rr/z%_parallel_extract..rerzParallel extract: %d URL(s)T)r^ full_contentNr*rrrrr)r~rrr^r)r[rSr]rzrrprrextractrrwrrsr~rrerrorsr error_type) r^rSrrrrr~rr^s r_parallel_extractr{s /.....~ONNNNNN KK-s4yy999/116>> ?H G"(b   %+ 9kk&/"7R88GjB ""&)E::       &B  9?]Me&6M:M$eio26        Nrc||dddddd} ddlm}|rtddSt}|d krt ||}t |d id g|d <tj|d d}t ||d<t d|t |S|dkrt||}t |d id g|d <tj|d d}t ||d<t d|t |S|dkrtd||td|t!|dddd}t#|}t |d id g|d <tj|d d}t ||d<t d|t |Std||t%||}t)|} t | } td| dd | id}| |d <tj|d d}t ||d<t d|t |S#t*$r} dt-| } td| | |d<t d|t t| cYd} ~ Sd} ~ wwxYw)aE Search the web for information using available search API backend. This function provides a generic interface for web search that can work with multiple backends (Parallel or Firecrawl). Note: This function returns search result metadata only (URLs, titles, descriptions). Use web_extract_tool to get full content from specific URLs. Args: query (str): The search query to look up limit (int): Maximum number of results to return (default: 5) Returns: str: JSON string containing search results with the following structure: { "success": bool, "data": { "web": [ { "title": str, "url": str, "description": str, "position": int }, ... ] } } Raises: Exception: If search fails or API key is not set )rOrPNr) parametersr^ results_countoriginal_response_sizefinal_response_sizerRrTFrr+rr"r~r indent ensure_asciirweb_search_toolr.r-zTavily search: '%s' (limit: %d)r\rm)rOrqinclude_raw_contentinclude_imagesz'Searching the web for: '%s' (limit: %d)zFound %d search resultsTrzError searching web: %sr^)r[rS tool_errorr9rtrr$rvdumps_debuglog_callsaver]r]rzrrrrfr\rr&rr) rOrPdebug_call_datarSr) response_data result_jsonrrrr~r error_msgs rrr sH  "#   OR%222222 >   <mU;;; ;.. j ,UE::M/2=3D3DVR3P3P3T3TUZ\^3_3_/`/`OO ,*]15QQQK585E5EO1 2 OO- ? ? ? KKMMM  e  'u55M/2=3D3DVR3P3P3T3TUZ\^3_3_/`/`OO ,*]15QQQK585E5EO1 2 OO- ? ? ? KKMMM  h   KK95% H H H!("5"~~',"' --C =SAAM/2=3D3DVR3P3P3T3TUZ\^3_3_/`/`OO ,*]15QQQK585E5EO1 2 OO- ? ? ? KKMMM  =ueLLL(**112   2(;; K((  -}==={  ,9(jquMMM 14[1A1A-. )?;;;  %%%4CFF44  T9%%%#, )?;;; )$$$$$$$$%s9 M*Bxxd?z cc<td@| i|$dAkr7|d= |#tdB| tdC| n|r7|s5tdD|d6 dE|d1gD]W}"|"d:d;} t|"dFd }%tdG| |%XdH|d1gD}&d1|&i}'|'d1gkrtQdI}(tS|(})n&t j|'dJdK}(tS|(})t|)|dL<|d6 dMtT+dN|tT,|)S#tD$r}*dOtI|*}+t#dP|+|+|dQ<tT+dN|tT,tQ|+cYd}*~*Sd}*~*wwxYw)Ra# Extract content from specific web pages using available extraction API backend. This function provides a generic interface for web content extraction that can work with multiple backends. Currently uses Firecrawl. Args: urls (List[str]): List of URLs to extract content from format (str): Desired output format ("markdown" or "html", optional) use_llm_processing (bool): Whether to process content with LLM for summarization (default: True) model (Optional[str]): The model to use for LLM processing (defaults to current auxiliary backend model) min_length (int): Minimum content length to trigger LLM processing (default: 5000) Security: URLs are checked for embedded secrets before fetching. Returns: str: JSON string containing extracted content. If LLM processing is enabled and successful, the 'content' field will contain the processed markdown summary instead of raw content. Raises: Exception: If extraction fails or API key is not set r) _PREFIX_RE)unquoteFz_Blocked: URL contains what appears to be an API key or token. Secrets must not be sent in URLs.)rr^)r^rrrrN)r}r^pages_extractedpages_processed_with_llmrrcompression_metricsprocessing_appliedz!Extracting content from %d URL(s)r*:Blocked: URL targets a private or internal network addressr~rrr^r+r.r-zTavily extract: %d URL(s)rx)r^rrmarkdownhtmlrRrTrbz%Blocked web_extract for %s by rule %srrulerPsourcerrrr~rrr^blocked_by_policyz Scraping: %s)r~formatsru)rwz!Firecrawl scrape timed out for %sucScrape timed out after 60s — page may be too large or unresponsive. Try browser_navigate instead.rrrrrz0Blocked redirected web_extract for %s by rule %sr~rrrr^rrzScrape failed for %s: %s)r~rrrr^rzExtracted content from %d pagesrrz3Processing extracted content with LLM (parallel)...rllm_processingcK|dd}|dd}|ddp|dd}|s|ddfSt|}t||| d{V}|r3t|}|d kr||z nd }||d<||d<|||| d }||d fS|||d dd d}||dfS)zHProcess a single result with LLM and return updated result with metrics.r~ Unknown URLrr*rrN no_contentrrr~ original_sizeprocessed_sizer model_used processedcontent_too_shortr~rrrrreason too_shortr$rr) rr~rrrrrrmetricsrrs rprocess_single_resultz/web_extract_tool..process_single_resultdsRjj 66 7B//$jj;;Xvzz)UW?X?X "6!455 #K 0 0 #;e_j## 8%(^^NJWZ[J[J[(F(Fad%)2F9%,7F=) #)6*8->&5 G"7K77 #)6*7-0&*"5 G"7K77rc&g|] }|SrFrF)rrrs rr/z$web_extract_tool..s%NNNv**622NNNrr~rrrrr%s (processed)r&%s (no processing - content too short)%s (no content to process)PLLM processing requested but no auxiliary model available, returning raw contentllm_processing_unavailabler%s (%d characters)cg|]h}|dd|dd|dd|ddd|vr d|diniiSr~r*rrr^rrr$rrs rr/z$web_extract_tool..     uuUB''w++55B//w    GZ]^F^F^)1-@+ABBdf      rz%Content was inaccessible or not foundr rrbase64_image_removalweb_extract_toolzError extracting content: rr^)- agent.redactrrrr\rvrr]rzrrrr9r{rgrrr[rSrrwait_for to_threadrfscrape TimeoutErrorrrr$rrrrrr&rrrcheck_auxiliary_modelr5rrGrrr).r^rrrrrr_urlr safe_urls ssrf_blockedr~rr)rr_is_interruptedblockedrscrape_payloadrrcontent_markdown content_html final_url final_blockedchosen_content scrape_errrrauxiliary_available results_listr7processed_resultsrrstatuscontent_lengthtrimmed_resultstrimmed_responsercleaned_resultrrrrs. ` @@rrrs4 >('''''$$$$$$   T " " j&7&7 &F&F : =    "4$   $%"# ! O"K% 7TCCC -/  & &Cs## &##Y%%   %%%%w GG"nnG*$$ 1) < <<<<<<<E!!&y11H$$ 7YHHH%i%&+226cXaHi RS gijjj&(Z'')lGGv%%%hGG *62G13MMMMMM$VVC&((!s]UW'X'XYYY 3377G! $KWU[_^efl^mnnn#&%,Y%7:A&/SZ[aSbnuv~n2A2A(( !F NC888%292B ' 1$9$;$;$B(+,3!"!"!" )+ 333------MM '3%%%"NN+NPSTTT#NN'*RB*O,,%H %*A)O)O#1#5#5j"#E#E "+9+=+=j+I+I('5'9'9&'A'A  *(D99.&x>>.+3+>+>+@+@!(:!>!>.+3+<+-!) Wb 9 9%-LLc$B$B (KF>S]jkq]rLMUV6W6W,, %?E >R>RW]WejzWe)9)9COCiScCigi#,%*'5+9(0 ((% %?jQQQ#&%'')+-%(__ ((  -"W,Gw'hll9b99:: 5GGG-<)*47 88L8L4M4M01B#@#B#B355 I G"5I G KKM N N N 0 1 8 89I J J J) 8) 8) 8) 8) 8) 8X$<< 266LNNNNNNNE&-ne&< < < < < < < ,= F F'jj 66[((#$9:AA'JJJ#$>???1D???KK 0#6666{**#$9:AA'JJJKK H#NNNNNN#?EEEE F" [*= [qrrr 45<<=YZZZ",,y"55 G Gjj 66!$VZZ r%B%B!C!C 0#~FFFF   \\)R00    &7    * *b 0 0$%LMMK0==NN*%5aeTTTK0==N14^1D1D-.,-445KLLL *O<<<  %%%9Q99  T9%%%#, *O<<< )$$$$$$$$%s5G`Q%$AJ'&Q%'AK/+Q%,`.K//EQ%4`6-Q%#`% R>/AR93`9R>>M` b$%A4bb$b$basic instructionsdepthc 4,-.K|||||ddddddggd} |p t-t}t}|dkr|dsd|}t |st jd|d d d d gid St|} | rct d| d| dt jd|d d | d| d| d| dddgid Sddl m } | rtdd St d||d|d} |r|| d<td| } t| |} d| i}t|dg}t d|||d<tt j||d <|r|rt d!|d"d#-fd$,,fd%|dgD}t%j|d{V}|D]7\}}}|d&kr+|d'||d(xxd)z cc<8|r7|s5td*|d"d+d,|dgD}t jd|id-d .}t+|}t||d/<t,d0|t,|St3s)t jd1t5d2d d3d S|ds d|}t d4||rd5|d6nd }t d7||t |st jd|d d d d gid St|} | rct d| d| dt jd|d d | d| d| d| dddgid Sdd8d9gid:}|rt d;ddl m } | rtdd S t7jddd<|i|}n.#t:$r!}td=|d}~wwxYwg}g}t?|d>r|j r|j ng}t d?tC|d@dAt dBt||stdCdDtE|Dtd?tC|d@dEtdFtC|dGdEtdHtC|dIdEntG|tHrd>|vr|d>g}ntdJtdKtK|t?|dLr?tdMtM|j'(|D]}dN}d } d}!d}"i}#t?|dOrU|)}$|$d9}!|$dP}"|$dQi}#nt?|dLrtC|d9d}!tC|dPd}"tC|dQi}%t?|%dOr|%)}#nt?|%dLr|%j'}#nptG|%tHr|%}#nXi}#nUtG|tHr@|d9}!|dP}"|dQi}#tG|#tHs?t?|#dOr|#)}#nt?|#dLr|#j'}#ni}#|#dR|#d._process_tavily_crawl8s%zz%??H"JJw33E$jjB77G":%t\99$'LLM&>wRWYhjt&u&u u u u u u uI <07}-,5y)*2]fijsftftZg8pI8V8VmpAP#Q#Q%w ;;&.bo47tWjllG!7K77rc&g|] }|SrFrF)rrrs rr/z"web_crawl_tool..Js%WWWa..q11WWWrrrrrrrcg|]h}|dd|dd|dd|ddd|vr d|diniiSrrrs rr/z"web_crawl_tool..UsOOOmn()uuUB'7'7!%%QSBTBTabafafgprtauauABAFAFGNAOAO i iFY]^F^F^)1-@+ABBdf iOOOrr rrweb_crawl_toolzFweb_crawl requires Firecrawl. Set FIRECRAWL_API_KEY, FIRECRAWL_API_URLz*, or use web_search + web_extract instead.rUz Added https:// prefix to URL: %sz with instructions: ''z Crawling %s%srr)rPscrape_optionsz;Instructions parameter ignored (not supported in crawl API)r~zCrawl API call failed: %srz Status: %srunknownzRetrieved %d pageszCrawlJob attributes: %sc<g|]}|d|Srr)rattrs rr/z"web_crawl_tool..s-8x8x8x$cgcrcrsvcwcw8x8x8x8xrzN/Az Total: %stotalz Completed: %s completedzUnexpected crawl result typezResult type: %srzResult attributes: %srrrrrrz"Blocked crawled page %s by rule %srrcpK|dd}|dd}|dd}|s|ddfSt|}t||| d{V}|r3t|}|dkr||z nd }||d <||d<|||| d }||d fS|||d dd d}||dfS)zNProcess a single crawl result with LLM and return updated result with metrics.r~rrr*rNrrrrrrrrrr) rrrrrrrrrrrs rprocess_single_crawl_resultz3web_crawl_tool..process_single_crawl_results<!::e];; 7B// **Y336!455 #G #;Xuoz## 8%(^^NJWZ[J[J[(F(Fad%-4F=)(1F9% ()6*8->&5 G"7K77 ()6*7-0&*"5 G"7K77rc&g|] }|SrFrF)rrrs rr/z"web_crawl_tool..4s%TTTV0088TTTrrrrrrrcg|]h}|dd|dd|dd|ddd|vr d|diniiSrrrs rr/z"web_crawl_tool..NrrrzError crawling website: rr^rF)+rrr9rrrvrrr]rzr[rSrrrrr$rrr5rrGrrrr;rSrfrr&rrrrdirrrtyperrkeysrr)/r~rrrrrrrr)r_is_intrsrrrrr7rrrrrrrinstructions_text crawl_params crawl_resultrpages data_listrrrrrr item_dict metadata_obj page_blockedrrrrrrrrs/ ` @@@rrrs@("4$   $%"# ! O$j%B#@#B#B355.. h  >>"9:: '&&&s## rz9sRTVY0[0[/\#]kprrrr+3//G Q A76?T[\bTcdddz9sRTVahiras29&/7SY?fmnvfw)x)x0z0z/{#|KPQQQQ B A A A A Awyy @!-???? KK*C 0 0 0!&''G  7*6'!'733C1#CHHHG!7+H  Y ; ;<EEgNNN'(BCCCqHCCC! [*= [qrrr 45<<=YZZZOOrzr~r~@IKMsNsNOOOO*i%A!Z_```K0==N585H5HO1 2 OO,o > > > KKMMM! !'(( #:i:<<iii " ### #~~566 A"S""C KK:C @ @ @GS[CLCCCCY[ OS*;<<<3 n:y3PRU,W,W+XYglnnn n's++  M KK=wvPWX^P_ ` ` `:y3PR]den]o.5fowvbijrbs%t%t,v,v+wxGLMMM MJ<    W KKU V V V====== 799 <mU;;; ; 80228LL    LL4a 8 8 8  ')  < ( ( Z-9->F ))BI KK glHi&P&P Q Q Q KK,c)nn = = = Y 68x8x#lJ[J[8x8x8xyyy \7<5+Q+QRRR [',*O*OPPP _glKQV.W.WXXX  d + + Z,0F0F$((44II NN9 : : : LL*D,>,> ? ? ?|Z00 Z 4d<;P;U;U;W;W6X6XYYYE E D$HE# LHt\** 4 OO-- #,==#<#< (}}V44 $==R88z** 4#*4T#B#B &tVT::  'tZ<< <66"+6688HH\:66"+4HH d33"+HH!HHD$'' 4#'88J#7#7 #xx// 88J33h-- "8\22"'2244HHXz22"'0HH!H ||Ke]1S1STTHLL"--E099L  @,vBVXdekXlmmm #eSU))42>v2FP\]cPdp|~FqG*H*H '<,<"G LL"&$     u%HLLB7788  & 666+8(47 88L8L4M4M01 I L"5I L KKK L L L 0 1 8 89I J J J) 8) 8) 8) 8) 8) 8X$<< 266LTTTT|TTTE&-ne&< < < < < < < ,= K K'!::e];;[((#$9:AA'JJJ#$>???1D???KK 0(;;;;{**#$9:AA'JJJKK H(SSSSNN#?JJJJ K" [*= [qrrr 45<<=YZZZ",,y"55 L L!::e];;!$VZZ 2%>%>!?!? 0(NKKKK   \\)R00    &7j!1!%PPP ,[9914^1D1D-.,-445KLLL (/:::  %%%7s1vv77  T9%%%#, (/::: )$$$$$$$$%ssA9rA3r r)H+r6r BrA3rArS*)r* T4TT]8r tA4t ttc:tp tS)a Check whether the Firecrawl backend is available. Availability is true when either: 1) direct Firecrawl config (`FIRECRAWL_API_KEY` or `FIRECRAWL_API_URL`), or 2) Firecrawl gateway origin + Nous Subscriber access token (fallback when direct Firecrawl is not configured). Returns: bool: True if direct Firecrawl or the tool-gateway can be used. )rMr5rFrrr;r;ss ( ) ) E-C-E-EErctdd}|dvrt |St ddDS)z6Check whether the configured web backend is available.r)r*)r.r+r,r-c34K|]}t|VdSr)r<)rr)s r z$check_web_api_key..s+hh'$W--hhhhhhr)r'r$r4rr<any)r6s rcheck_web_api_keyr sm!##'' 266<<>>DDFFJ???$Z000 hh=ghhh h hhrc.t\}}}|duS)zICheck if an auxiliary text model is available for LLM content processing.Nr)rrs rrrs133LFAq  r__main__u 🌐 Standalone Web Tools Modulez(========================================r/r0u✅ Web backend: r.z! Using Exa API (https://exa.ai)r+z+ Using Parallel API (https://parallel.ai)r-z( Using Tavily API (https://tavily.com)z Using self-hosted Firecrawl: r>z# Using direct Firecrawl cloud APIz! Using Firecrawl tool-gateway: z0 Firecrawl backend selected but not configuredu$❌ No web search backend configuredzWSet EXA_API_KEY, PARALLEL_API_KEY, TAVILY_API_KEY, FIRECRAWL_API_KEY, FIRECRAWL_API_URLu;❌ No auxiliary model available for LLM content processingzVSet OPENROUTER_API_KEY, configure Nous Portal, or set OPENAI_BASE_URL + OPENAI_API_KEYuK⚠️ Without an auxiliary model, LLM content processing will be disabledu✅ Auxiliary model available: ru!🛠️ Web tools ready for use!u+🧠 LLM content processing available with z& Default min length for processing: z charsu&🐛 Debug mode ENABLED - Session ID: z Debug logs will be saved to: z/web_tools_debug_z.jsonu=🐛 Debug mode disabled (set WEB_TOOLS_DEBUG=true to enable)z Basic usage:zI from web_tools import web_search_tool, web_extract_tool, web_crawl_toolz import asyncioz # Search (synchronous)z/ results = web_search_tool('Python tutorials')z$ # Extract and crawl (asynchronous)z async def main():z? content = await web_extract_tool(['https://example.com'])zC crawl_data = await web_crawl_tool('example.com', 'Find docs')z asyncio.run(main())z LLM-enhanced usage:zC # Content automatically processed for pages >5000 chars (default)zA content = await web_extract_tool(['https://python.org/about/'])z# # Customize processing parametersz$ crawl_data = await web_crawl_tool(z 'docs.python.org',z 'Find key concepts',z, model='google/gemini-3-flash-preview',z min_length=3000z )z # Disable LLM processingzY raw_content = await web_extract_tool(['https://example.com'], use_llm_processing=False)z Debug mode:z # Enable debug loggingz export WEB_TOOLS_DEBUG=truez # Debug logs capture:z$ # - All tool calls with parametersz # - Original API responsesz # - LLM compression metricsz # - Final processed resultsz3 # Logs saved to: ./logs/web_tools_debug_UUID.jsonuL 📝 Run 'python test_web_tools_llm.py' to test LLM processing capabilities)registryr web_searchzrSearch the web for information on any topic. Returns up to 5 relevant results with titles, URLs, and descriptions.objectstringz&The search query to look up on the web)rr)r propertiesrequired)rrr}ruExtract content from web page URLs. Returns page content in markdown format. Also works with PDF URLs (arxiv papers, documents, etc.) — pass the PDF link directly and it converts to markdown text. Pages under 5000 chars return full markdown; larger pages are LLM-summarized and capped at ~5000 chars per page. Pages over 2M chars are refused. If a URL fails or times out, use the browser tool to access it instead.arrayrz:List of URLs to extract content from (max 5 URLs per call))rrrmaxItemsr"c Lt|dddS)NrOr*rh)rP)rr$argskws rr2r2$s!txx/D/DANNNru🔍r)rtoolsetschemahandlercheck_fn requires_envemojimax_result_size_charsc tt|dtr|dgddngdS)Nr^rhr)rrr$rrs rr2r2.sP/$.txx/?/?$F$FNRaR  BPZ \ \ru📄) rrrrrris_asyncrr)rN)r*r)rFr*)rN)rh)m__doc__rvloggingrrBrtypingrrrrr{r,rrrr r tools.debug_helpersr tools.managed_tool_gatewayr r rKrtools.tool_backend_helpersrrtools.url_safetyrtools.website_policyr getLogger__name__r]rrrrr'r9r<rarbr4rDrGr5rMrQrSrr[rfrkrorlrprrxrrrrrrr$DEFAULT_MIN_LENGTH_FOR_SUMMARIZATIONrrrrrrrrrGrKrMr]rgrtr{rrrr;r rprint web_availabletool_gateway_availablerfirecrawl_key_availablefirecrawl_url_availablenous_availabledefault_summarizer_modelr)rBexitactive session_idlog_dirtools.registryr rWEB_SEARCH_SCHEMAWEB_EXTRACT_SCHEMAregisterrFrrr<s''R  ,,,,,,,,,,,,  -,,,,, SRRRRRRR((((((555555  8 $ $ %3%4%%%%$c6 3 4    @huT#s(^U3PXY\P]_ghk_lKlEm5m/n&o@@@@"1C1111 ggggg 6d6666    49*$$$P$"""&29.0HIIcDT(;t;;;;;"%%$%c%4PTUXZ]U]P^K_%%%%PCC. 3 4S#X+?    #$tCH~2F> 3 4S>    (,$LcLdLLLL / /(3- /5RUX`adXegkloqtltguIuCv / / / /x} k+< = = = : aa a a a C= a  a c] aaaaP qq qq C=q q  q  qc]qqqqhP PP C=P P  P c] PPPPfccF *;;s;3;;;;;8tCyT$sCx.%9D;;C;;D;;;;>($s)(T#s(^0D((((V%%3%s%3%%%%H#: E%E% s)E% E%E% C= E%  E%  E%E%E%E%T #: Z%Z% Z%Z% Z% Z% C= Z%  Z% Z%Z%Z%Z%| F F F F Fi4iiiit z E ,--- E(OOO&%''M3355"d929-@"#E#E#K#K#M#MNN"d929-@"#E#E#K#K#M#MNN**,,N<<>> ,.. +'++,,, e   E5 6 6 6 6  " " E? @ @ @ @  E< = = = =& JmCV9W9W9]9]9_9_9f9fgj9k9kmmnnnn( J;<<<<' JX:T:T:V:VXXYYYYHIIII 4555  2--// 2 2   L KLLL fggg [\\\\ J0HJJKKK  Q E -...e V