U Ad @sdZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZddlmZmZmZmZmZmZmZddlZdd lmZmZdd lmZdd l m!Z!m"Z"m#Z#zdd l$m%Z&e&Z%Wne'k re(Z%YnXe)e*Z+d Z,dZ-ddddZ.dZ/dZ0dZ1dZ2dZ3dZ4dZ5dZ6dZ7ervddlZddl8m9Z9m:Z:Gddde:Z;neee>Z?dd#d$e?ee@e@e@d%d&d'ZAGd(d)d)eBZCd*d+ZDded?ZMdenz)_format_schema_problems..)joinmap)r9r7r8Z formattedr)r)r*_format_schema_problemshs r@cs@eZdZdZd eeeedfdd ZedddZZ S) SchemaValidationErrorzz in version .  z **)get capitalizestrip)r\Zkey_descriptionvmsgannotaterZr)r*format_messages   z:_add_deprecated_changed_or_new_msg..format_messagerchangednewrEr)r&r>r?r`rstrip)rZrf filter_keyrgZ filter_keysZchanged_new_deprecatedrr)rer*"_add_deprecated_changed_or_new_msgs  rl)rschemark error_typeccs"|rt|d|gd}||VdS)zJsonschema validator for `deprecated` items. It raises a instance of `error_type` if deprecated that must be handled, otherwise the instance is consider faulty. T)rfrkN)rl) _validatorrZ _instancermrkrnrdr)r)r*ros ro)rkrhrnc #sddlm}g}g}t|D]d\}} t|j|| |d} ttfdd| } ttfdd| } | sv|| q|| q|d|f|dV|Ed Hd S) zJsonschema validator for `anyOf`. It treats occurrences of `error_type` as non-errors, but yield them for external processing. Useful to process schema annotations, as `deprecated`. rrZ schema_pathcs t| Sr:rVerpr)r*r<r=z_anyOf..cs t|Sr:rrrsrpr)r*r<r=.%r is not valid under any of the given schemascontextN)rSr enumeratelistdescendfilterextend) validatoranyOfrX_schemarnr all_errorsall_deprecationsindex subschemaall_errserrs deprecationsr)rpr*_anyOfs(   rc#sddlm}t|}g}g}|D]h\} } tj| | d} ttfdd| } ttfdd| } | s~| }|| q|| q |df|dVfd d |D}|r||d d d |D}|d|fVn |EdHdS)zJsonschema validator for `oneOf`. It treats occurrences of `error_type` as non-errors, but yield them for external processing. Useful to process schema annotations, as `deprecated`. rrrqcs t| Sr:rrrsrpr)r*r<r=z_oneOf..cs t|Sr:rrrsrpr)r*r<r=rurvcs g|]\}}|r|qSr))is_valid).0is)rXr}r)r* (s z_oneOf..r5css|]}t|VqdSr:)repr)rrmr)r)r* +sz_oneOf..z%r is valid under each of %sN) rSrrxryrzr{r|appendr>)r}oneOfrXrrnrZ subschemasrrrrrrrZ first_validZ more_validZreprsr))rnrXr}r*_oneOfs:      rc Csddlm}m}ddlm}t|j}d|d<ddi|dd <i}t|d rd|j dt }d |i}n|j }t t f|d<d |i}t|j}t|t<t|d <t|d<t|d<|f||dd|}ddd} | |_||fS)zGet metaschema validator and format checker Older versions of jsonschema require some compatibility changes. @returns: Tuple: (jsonschema.Validator, FormatChecker) @raises: ImportError when jsonschema is not present r)rQ FormatChecker)createFZadditionalPropertiestyperR propertieslabelrU type_checkerZ default_typesrhrr~Zdraft4)Z meta_schema validatorsversionNc[s$tdd|||}t|ddkS)zgOverride version of `is_valid`. It does ignore instances of `SchemaDeprecationError`. cSs t|t Sr:)rVr+rsr)r)r*r<rsz.is_valid..N)r{ iter_errorsnext)r3rXr__errorsr)r)r*rls  z*get_jsonschema_validator..is_valid)N)rSrQrZjsonschema.validatorsrrZ META_SCHEMAhasattrrUZredefinerYZ DEFAULT_TYPESr&rWdictZ VALIDATORS_validator_deprecatedDEPRECATED_KEY_validator_changedrrr) rQrrstrict_metaschemaZvalidator_kwargsrtypesrcloudinitValidatorrr)r)r*get_jsonschema_validator3s@      rTrmc Cszddlm}z||WnZ|k rt}z<|rXttddd|jD|jgd|t d|W5d}~XYnXdS) a Validate provided schema meets the metaschema definition. Return strict Validator and FormatChecker for use in validation @param validator: Draft4Validator instance used to validate the schema @param schema: schema to validate @param throw: Sometimes the validator and checker are required, even if the schema is invalid. Toggle for whether to raise SchemaValidationError or log warnings. @raises: ImportError when jsonschema is not present @raises: SchemaValidationError when the schema is invalid r) SchemaError.cSsg|] }t|qSr)r&rr;r)r)r*rsz3validate_cloudconfig_metaschema..rCzGMeta-schema validation failed, attempting to validate config anyway: %sN) Zjsonschema.exceptionsrZ check_schemarAr,r>r-r.LOGwarning)r}rmthrowrerrr)r)r*validate_cloudconfig_metaschema~s$ r)rZrmstrictr log_detailslog_deprecationscCs&|dkrt}z t\}}|r,t||ddWn tk rNtdYdSX|||d}g} g} t||dddD]D} d d d | j D} t | | j f} t | t r| | 7} q|| | 7} q||r| rt| d d d}t||r| s| rt| | | r"|rt| dd d}nd}t|dS)aValidate provided config meets the schema definition. @param config: Dict of cloud configuration settings validated against schema. Ignored if strict_metaschema=True @param schema: jsonschema dict describing the supported schema definition for the cloud config module (config.cc_*). If None, validate against global schema. @param strict: Boolean, when True raise SchemaValidationErrors instead of logging warnings. @param strict_metaschema: Boolean, when True validates schema using strict metaschema definition at runtime (currently unused) @param log_details: Boolean, when True logs details of validation errors. If there are concerns about logging sensitive userdata, this should be set to False. @param log_deprecations: Controls whether to log deprecations or not. @raises: SchemaValidationError when provided config does not validate against the provided schema. @raises: RuntimeError when provided config sourced from YAML is not a dict. NF)rz5Ignoring schema validation. jsonschema is not present)Zformat_checkercSs|jSr:)r-rsr)r)r*r<r=z-validate_cloudconfig_schema..r[rcSsg|] }t|qSr)rrr)r)r*rsz/validate_cloudconfig_schema..z"Deprecated cloud-config provided:  r6zInvalid cloud-config provided: zeInvalid cloud-config provided: Please run 'sudo cloud-init schema --system' to see the schema errors.) get_schemarrrTrdebugsortedrr>r-r,r.rVr+r@rrA)rZrmrrrrrrr}rrZ schema_errorr-problemr.Zdetailsr)r)r*validate_cloudconfig_schemas^         rc @seZdZeeedddZeeeeedddZ e ddd Z edeeeeeee ee d d d Z eeeeeedddZe e eeefdddZdS) _Annotator) cloudconfigoriginal_content schemamarkscCs||_||_||_dSr:) _cloudconfig_original_content _schemamarks)r3rrrr)r)r*rIsz_Annotator.__init__)rcontentr0cCsd|}d|d|dS)Nr# z: ------------- rF)r>)rrZbodyr)r)r* _build_footers z_Annotator._build_footer)r9cCsztt}|D]h\}}td|}|rD|\}}|t||nd}||j|||dk r dj|||d}q |S)Nz&format-l(?P\d+)\.c(?P\d+).*zLine {line} column {col}: {msg})linecolrd) rryrematchgroupsintrrr4)r3r9errors_by_liner-rdrrrr)r)r*_build_errors_by_lines   z _Annotator._build_errors_by_linerE)problemslabelsfooterr label_prefixr0cCsB|D]8}||}|||d|d||d7}q|S)Nrr1)r)rrrrrrrr)r)r* _add_problemss    z_Annotator._add_problems)linesrdeprecations_by_liner0c sg}g}g}d}d}t|dD]p\} } || } || } | s>| rg} j| | ||dd}j| | ||dd}|| dd| q|| q|tfddtd dd |fd |ff|S) NrE)rDz # ,cs j|Sr:)rseqr2r)r*r<Ar=z._Annotator._annotate_content..cSs t|dS)Nr)rLrr)r)r*r<Cr=ErrorsZ Deprecations)rxrrr>r|r?r{)r3rrrannotated_contentZ error_footerZdeprecation_footerZ error_indexZdeprecation_indexZ line_numberrrrrr)r2r*_annotate_content!sL   z_Annotator._annotate_content)rCrDr0cCsp|s|s|jS|jd}t|jtsDd||ddggS||}||}| |||}d|S)Nrrz&# E1: Cloud-config is not a YAML dict.) rdecodesplitrVrrr>rrr)r3rCrDrrrrr)r)r*rfMs*   z_Annotator.annotateN)rE)r#r$r%rrWrI staticmethodr&rrrOrrrrr rfr)r)r)r*rs4  . rrB)rrrrCrDr0cCst||||pg|pgS)aReturn contents of the cloud-config file annotated with schema errors. @param cloudconfig: YAML-loaded dict from the original_content or empty dict if unparseable. @param original_content: The contents of a cloud-config file @param schemamarks: Dict with schema marks. @param schema_errors: Instance of `SchemaProblems`. @param schema_deprecations: Instance of `SchemaProblems`. @return Annotated schema )rrf)rrrrCrDr)r)r*annotated_cloudconfig_filegs rc Cst|dd}|tsTtdd|tg}t|}|rPtti|i|j d|z&|rjt |\}}nt |}i}Wnt jk rN}zd} } d} t|drt|drt|d} nt|d rt|d rt|d } | r| jd} | jd} td j| | d d |t|g}t|}|r8tti|i|j d||W5d}~XYnXt|tsj|sjtd zt||dddWnrtk r}zR|rtt||||j |jdn |jrt|jddd} t| |rW5d}~XYnXdS)aValidate cloudconfig file adheres to a specific jsonschema. @param config_path: Path to the yaml cloud-config file to parse, or None to default to system userdata from Paths object. @param schema: Dict describing a valid jsonschema to validate against. @param annotate: Boolean set True to print original config file with error annotations on the offending lines. @raises SchemaValidationError containing any of schema_errors encountered. @raises RuntimeError when config_path does not exist. F)rz format-l1.c1z"File {0} needs to begin with "{1}"rrNZ context_markZ problem_markzformat-l{line}.c{col})rrzFile {0} is not valid yaml. {1}z Cloud-config is not a YAML dict.T)rrrBrGr5r6)r startswithCLOUD_CONFIG_HEADERr,r4rrAprintrrCrZload_with_marksloadyamlZ YAMLErrorrgetattrrcolumnr&rVr RuntimeErrorrrDr@rM) Z config_pathrmrfrrrrZmarksrtrrZmarkr.r)r)r*validate_cloudconfig_files            rcCs|dkr dS|dkrdSdS)zProvide a sorting weight for documentation of property types. Weight values ensure 'array' sorted after 'object' which is sorted after anything else which remains unsorted. Zarrayobjectrrr))valuer)r)r*_sort_property_orders rccs<|D]2}t|tr0t|ttfs0t|EdHq|VqdSr:)rVrr&rW_flatten)Zxsxr)r)r*rsr) property_dict multi_keyr0cCsng}||iD]P}|tr q|drH|dd|dgDq|dr||dqtt|S)NenumcSs g|]}dt||dqS`` _YAML_MAPr`)rZ enum_valuer)r)r*rsz,_collect_subschema_types..r)r`rr|rryr)rrproperty_typesrr)r)r*_collect_subschema_typess    r)rdefsr0c Csxt|||dg}t|ts&|g}|drDdd|dD}n6|dr`|t|dn|drz|t|dt|dkr|d}n|jtd d |}|d i}|dg}t|ts|g}t |}t |di|diD]*}t ||}|r|d krq| |q|rnt|dkrB|d |dS|jtd dd |d} |d | S|pvd S)zNReturn a string representing a property type from a given jsonschema. rrcSs g|]}dt||dqSrrrkr)r)r*r sz&_get_property_type..rr~rrr[/itemsZ UNDEFINEDz of ())_flatten_schema_refsr`rVryr|rlensortrr>rLr_get_property_typer) rrrZ property_typerZsub_property_typesZprune_undefinedsub_itemZsub_typeZsub_property_docr)r)r*rsB                rr/cCsB|d}tdd|}tdd||}tdd||}|S)aParse description from the meta in a format that we can better display in our docs. This parser does three things: - Guarantee that a paragraph will be in a single line - Guarantee that each new paragraph will be aligned with the first paragraph - Proper align lists of items @param description: The original description in the meta. @param prefix: The number of spaces used to align the current description z (\S)\n(\S)z\1 \2z\n\nz\n\n{}z\n( +)-z\n{}-)rsubr4)rr7Zlist_paragraphr)r)r*_parse_description+s r)src_cfgrcCsd|kr(|ddd}|||d|krd|dkrd|dddd}|d||d|dkr|ddD],}d|kr||ddd}|||q|t|dg|dg|dgD],}d|kr|ddd}|||qdS) zEFlatten schema: replace $refs in src_cfg with definitions from $defs.z$refz#/$defs/rErrr~allOfN)popreplaceupdaterr`)rrZ reference sub_schemar)r)r*rAs(     r)rcCs,|dd}|sdS|D]}||qdS)zFlatten schema: Merge allOf. If a schema as allOf, then all of the sub-schemas must hold. Therefore it is safe to merge them. r N)r r )rZ sub_schemasr r)r)r*_flatten_schema_all_of[s  r) prop_configr0cCs|tttddd}|di}|di}g}g}||||t||D]}||||qHdt||}|rxd|}|S)aGReturn accumulated property description. Account for the following keys: - top-level description key - any description key present in each subitem under anyOf or allOf Order and deprecated property description after active descriptions. Add a trailing stop "." to any description not ending with ":". rZ descriptionsdeprecated_descriptionscSs<tt|jdr |t|n|dr8|t|dS)N)Zdeprecated_versionZchanged_versionZ new_versionr)anyr?r`rrlrr)r)r*assign_descriptionsss z6_get_property_description..assign_descriptionsrr~r]r^)rryr`rr>)rrrr~rrrrr)r)r*_get_property_descriptionhs      r )rmrr0c s|d}g}ddkrdSfdddD}fdd|D}|D]T}|D]D\}} t| |t| | ddkrqTt| } | d |} |tj|| t| |t | |d | d } | rtt| || d s| d r|t j|| d|d7}|t | ||d| dgD]N} | d s@| d r$|t j|| d|d7}|t | ||dq$d | ksd | krT|t | ||dqTqFd |S)zDReturn restructured text describing the supported schema properties.rhiddenTrEcs$g|]}dks|dkr|qS)rr)rr\rr)r*rs z%_get_property_doc..)rpatternPropertiescsg|]}|iqSr))r`rrr)r*rsr)r7 prop_namerZ prop_typerrr)r7r)rr7rrF) r`rrrrrSCHEMA_PROPERTY_TMPLr4rrSCHEMA_LIST_ITEM_TMPL_get_property_docr>)rmrr7Z new_prefixrZ property_keysZproperty_schemasZ prop_schemaZprop_keyrrrrZ alt_schemar)rr*rs       r)metar0cCsj|d}|sdSt}t|D]F\}}t|dd}|tkrV|dt|d|d |7}q|S)zAReturn restructured text describing the meta examples if present.r rErrrr) r`SCHEMA_EXAMPLES_HEADERrxtextwrapindentrinsertSCHEMA_EXAMPLES_SPACER_TEMPLATEr4r>)rr Z rst_contentcountZexampleZindented_linesr)r)r* _get_exampless  r%cCs2|dsdSddd|dD}d|dS)Nr"rEr5css|]}d|dVqdS)rNr)rr)r)r*rsz3_get_activate_by_schema_keys_doc..z**Activate only on keys:** rF)r`r>)rZ schema_keysr)r)r* _get_activate_by_schema_keys_docs   r&)rrmr0c Csl|dkrt}|r|stdt|}ddddddd h}d h}d }||r^d ||}n|||rxd ||}|rt|tt|}d |d<|di}||dr||di}t t|}zt ||d|d<Wn(t k rt dd |d<YnX|drt|d<t||d<d|d|d<tdd|d |d<t||d <tjf|}|S)zReturn reStructured text rendering the provided metadata. @param meta: Dict of metadata to render. @param schema: Optional module schema, if absent, read global schema. @raise KeyError: If metadata lacks an expected key. Nz"Expected non-empty meta and schemarrr r!rrrr"rEz(Missing required keys in module meta: {}z3Additional unexpected keys found in module meta: {}Zproperty_header$defs)rZ property_docz3Unable to render property_doc due to invalid schemar5r-Ztitle_underbar)r ValueErrorsetkeysr4KeyErrorrrr`r rAttributeErrorrrSCHEMA_PROPERTY_HEADERr%r>rrr&SCHEMA_DOC_TMPL) rrmr+Z required_keysZ optional_keysZ error_messageZ meta_copyrtemplater)r)r* get_meta_docs^          r1cCstjtjt}t|Sr:)osr-dirnameabspath__file__r)Z configs_dirr)r)r* get_modules1sr6)requested_modulesr0cCsd}ttdg}t|t|}|rLtdt|d|dd|D]J}d|ksd||krPt |dgdg\}}|rPt |d }||j pd7}qP|S) zaLoad module docstrings Docstrings are generated on module load. Reduce, reuse, recycle. rEallz+Invalid --docs value {}. Must be one of: {}r5Tsys_exitcloudinit.configrr) ryr6valuesr* differencerr4r>r find_module import_modulerN)r7docsZ all_modulesZ invalid_docsmod_namemod_locs_modr)r)r*load_doc6s,rEcCstjtjtjtdS)NZschemas)r2r-r>r3r4r5r)r)r)r*get_schema_dirQsrFc Csxtjtt}d}ztt|}Wn0tk rV}zt d||W5d}~XYnX|stt d|idgd}|S)z?Return jsonschema coalesced from all cc_* cloud-config modules.Nz$Cannot parse JSON schema file %s. %szCNo base JSON schema files found at %s. Setting default empty schemaz'http://json-schema.org/draft-04/schema#)r'z$schemar ) r2r-r>rFUSERDATA_SCHEMA_FILEjsonloadsr Exceptionrr)Z schema_file full_schemartr)r)r*rUs  rcCsVt}tD]@\}}t|dgdg\}}|rt|d}|j||jd<q|S)zr?r)Z full_metarCrArBrDr)r)r*get_metatsrLcCs\|stjddd}|jdddd|jdd d d d |jd dddd|jdd d dd |S)z0Return a parser for supported cmdline arguments.cloudconfig-schemaz.Validate cloud-config files or document schema)progrz-cz --config-filez.Path of the cloud-config yaml file to validate)helpz--system store_trueFz)Validate the system cloud-config userdata)actiondefaultrOz-dz--docs+zCPrint schema module docs. Choices: all or space-delimited cc_names.)nargsrOz --annotatez/Annotate existing cloud-config file with errors)argparseArgumentParser add_argumentparserr)r)r* get_parsers: rZc Cs|j|j|jg}tdd|Ddkr2tddd|jrJ|jrJtdddt}|jrhtt|jdS|jr|d |jff}nt d krtd ddt gd }|j d d|j d}|stddddSd |ff}d|j dfd|j dff}|D]&\}} | rt j| r||| ff7}qt j|d dsTtd|d dddddd} tt|dk} | rtdddd|Dd} g} t|dD]\} \}}| rtd| d |d!|d"zt|||jWntk r:}z<|js*t| d#|tt|| d$d%| |W5d}~XYnvtk r}z4t| d#|tt|| d$d%| |W5d}~XYn$X|jr|n|}t| d&|q| rtdd'd| Dd(dddS))z@Handle provided schema args and perform the appropriate actions.cSsg|] }|r|qSr)r))rargr)r)r*rsz&handle_schema_args..rz;Expected one of --config-file, --system or --docs argumentsTr9z;Invalid flag combination. Cannot use --annotate with --docsNz user-datarzNUnable to read system userdata or vendordata as non-root user. Try using sudo.)Zds_depsZtrust)existingZ cloud_configz;Unable to obtain user data file. No instance data availablez vendor-dataZvendor_cloud_configz vendor2-dataZvendor2_cloud_configz Config file z does not existz Error: {})fmtr:rEz!Found cloud-config data types: %sr5css|]\}}|VqdSr:r))rcfg_typerCr)r)r*rsz%handle_schema_args..z rr]z at :zInvalid cloud-config z Error: {} )r]zValid cloud-config: css|] }|VqdSr:r))rrnr)r)r*rsz'Error: Invalid cloud-config schema: {} )Z config_filer@systemrrrfrrrEr2getuidrZfetchpathsZ get_ipathr-existsrLr>rxrrAr&rr)rargsZexclusive_argsrKZ config_filesZinitZ userdata_fileZvendor_config_filesr^Z vendor_fileZnested_output_prefixZmulti_config_outputZ error_typesidxZcfg_filertZcfgr)r)r*handle_schema_argss       rfcCst}td|dS)zDTool to validate schema of a cloud-config file or print schema docs.rMr)rZrf parse_argsrXr)r)r*mainsrh__main__)FN)T)NFFTF)F)r)N)N)irNrUrHZloggingr2rsysr  collectionsrZcollections.abcrcopyr functoolsr itertoolsrr(rrr r r r r rZ cloudinitrrZcloudinit.stagesrZcloudinit.utilrrrrSrZ_ValidationErrorrTrJZ getLoggerr#rZVERSIONED_USERDATA_SCHEMA_FILErGrrr/r.rrrr#rZDEPRECATED_PREFIXZtyping_extensionsrrrrr+r,rOr&r@r)rArYrlrLrorrrrrrrrrWrrrrrrrrrrrr%r&r1r6ryrErFrrLrZrfrhexitr)r)r)r*s     $       # )    ) /K% Qz  Z , *H > $X