/usr/local/lib/python3.6/site-packages/datasets/utils/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/datasets/utils/__pycache__/readme.cpython-36.pyc (8900B)
3
<%Eg1
@ s0 d dl Z d dlmZ d dlmZmZmZ d dlZyd dlj Z
W n e k
rZ d dl Z
Y nX ddl
m Z dZe de Ze jeZeeeef dd d
Zed \ZZd d
dgZeeee ee f ZG dd dZG dd deZ edkr,d dl m Z e ddZ e j d e j! Z"ee"j#Z#e j$e#Z%dS ) N)Path)AnyListTuple ) resourceszDhttps://github.com/huggingface/datasets/tree/main/src/datasets/utils/)resourcereturnc C s$ t jt| }tj|t d| fS )Nz /resources/)
pkg_resources read_textr yamlZ safe_load BASE_REF_URL)r content r ?/usr/local/lib/python3.6/site-packages/datasets/utils/readme.pyload_yaml_resource s r zreadme_structure.yamlz[Needs More Information]z[More Information Needed]zj(https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)c @ sT e Zd Zdeeee edddZdedddZee d d
d Z
ed d
dZ dS )SectionNF)namelevellinessuppress_parsing_errorsc C sJ || _ || _|| _d| _d| _i | _g | _g | _| jd k rF| j|d d S )N T)r ) r r r text
is_empty_textr parsing_error_listparsing_warning_listparse)selfr r r r r r r __init__) s
zSection.__init__)r c C s d}g }d}xj| j D ]}|jddkr,qq|jdd d dkrJ| }q|j d | jd kr| r|dkrt|| jd || j|< g }n>|g kr| jdj|j 7 _| jdkr| jtkrd| _ g }dj|j d d jd}q|j
| qW |dkr:|| jkr | j j
d
| d t|| jd || j|< n@|g krz| jdj|j 7 _| jdkrz| jtkrzd| _ | jdkr| r| j g ks| j g krd jd
d | j | j D }}d| j
d| }t|d S )Nr Fz
z```r # r z)Multiple sections with the same heading `z:` have been found. Please keep only one of these sections.
c s s | ]}d | V qdS )z- Nr ).0xr r r
[ s z Section.parse..z=The following issues were found while parsing the README at `z`:
)r stripsplitr r r r join FILLER_TEXTr appendr r r
ValueError)r r Zcurrent_sub_levelZ
current_linesZ
code_startlineerrors error_stringr r r r 5 sB
z
Section.parse) structurer
c C s g }g }|d dkr8| j r8| ji kr8|jd| j d |d dkr^| j r^|jd| j d |d d k r| ji krd
d |d D }|jd | j d
djdd |D d ndd |d D }d}x~t|D ]r\}}|| jkr|jd | j d| d d}q| jdkrqn | j| j|d | \} }
} ||
7 }|| 7 }qW |rx4| jD ]*}||krX|jd| j d| d qXW |ri ||fS | j ||fS d S )aC Validates a Section class object recursively using the structure provided as a dictionary.
Args:
structute (:obj: `dict`): The dictionary representing expected structure.
Returns:
:obj: `ReadmeValidatorOutput`: The dictionary representation of the section, and the errors.
Z allow_emptyFz"Expected some content in section `z` but it is empty.Zallow_empty_textz Expected some text in section `z4` but it is empty (text in subsections are ignored). subsectionsNc S s g | ] }|d qS )r r )r$
subsectionr r r
} s z$Section.validate..z Section `z&` expected the following subsections: z, c S s g | ]}d | d qS )`r )r$ r% r r r r3 s z. Found 'None'.c S s g | ] }|d qS )r r )r$ r2 r r r r3 s z` is missing subsection: `z`.Tz###r4 z ` has an extra subsection: `z[`. Skipping further validation checks for this subsection as expected structure is unknown.) r r r+ r r) enumerater validateto_dict) r r0
error_list warning_listvaluesZstructure_namesZhas_missing_subsectionsidxr _Zsubsec_error_listZsubsec_warning_listr r r r6 _ sD
(
zSection.validate)r
c C s$ | j | j| jdd | jj D dS )z3Returns the dictionary representation of a section.c S s g | ] }|j qS r )r7 )r$ valuer r r r3 s z#Section.to_dict..)r r r r1 )r r r r r: )r r r r r7 s zSection.to_dict)NF)F) __name__
__module__ __qualname__strr boolr r dictReadmeValidatorOutputr6 r7 r r r r r ( s *Gr c s e Zd Zdeee eed fdd
Zdd Ze de
eedd d
Z e deeeed d
dZ ded fdd
Z
dd Zdd Z ZS )ReadMeNF)r r r0 r c sB t j|dd || _d| _d| _|| _| jd k r>| j|d d S )Nr )r r r )r )superr r0 yaml_tags_line_count tag_countr r )r r r r0 r ) __class__r r r s
zReadMe.__init__c C sx | j d kr | jt\}}}n| j| j \}}}|g ks>|g krtdjttdd || }d| j d| }t|d S )Nr# c S s d| S )Nz- r )r% r r r s z!ReadMe.validate..z3The following issues were found for the README at `z`:
)r0 _validatereadme_structurer) listmapr r, )r r r8 r9 r. r/ r r r r6 s
zReadMe.validate)pathr0 r c
C s0 t |dd}|j }W d Q R X | ||||dS )Nzutf-8)encoding)r )open readlines)clsrQ r0 r fr r r r from_readme s zReadMe.from_readmeroot)stringr0 root_namer c C s |j d}| ||||dS )Nr# )r )r( )rU rY r0 rZ r r r r r from_string s
zReadMe.from_string)r c s d}xL| j D ]B}| jd7 _|jddkrF| jd7 _| jdkrFP |d7 }q W | jdkrr| j |d d | _ n| j | jd | _ t j|d d S )Nr r z
z---rF )r )r rI r' rJ rH r )r r
line_countr- )rK r r r s
z ReadMe.parsec C s t | j S )z>Returns the string of dictionary representation of the ReadMe.)rA r7 )r r r r __str__ s zReadMe.__str__c C s g }g }| j dkr |jd n*| jdkr4|jd n| jdkrH|jd t| jj }|dkr|jddjdd t| jj D d
nj|dk r|jd nVt| jj d }|jd r| j| j |d
d \}}}||7 }||7 }n
|jd |ri ||fS | j
||fS d S )Nr z-Empty YAML markers are present in the README.z*No YAML markers are present in the README.r z2Only the start of YAML tags present in the README.z-The README has several first-level headings: z, c S s g | ]}d | d qS )r4 r )r$ r% r r r r3 s z$ReadMe._validate..zL. Only one heading is expected. Skipping further validation for this README.zmThe README has no first-level headings. One heading is expected. Skipping further validation for this README.zDataset Card forr1 zuNo first-level heading starting with `Dataset Card for` found in README. Skipping further validation for this README.) rI r+ rJ lenr keysr) rO
startswithr6 r7 ) r rN r8 r9 Znum_first_level_keysZ start_keyr< Zsec_error_listZsec_warning_listr r r rM s4
*
zReadMe._validate)NF)NF)NrX F)F)r> r? r@ rA r rC rB r r6 classmethodr rW r[ r r] rM
__classcell__r r )rK r rE s
rE __main__)ArgumentParserz?Validate the content (excluding YAML tags) of a README.md file.)usagereadme_filepath)&loggingpathlibr typingr r r r
Zimportlib.resourcesr r ImportErrorZimportlib_resourcesr r __file__Zthis_url getLoggerr> loggerrA r rN Zknown_readme_structure_urlr* rC rD r rE argparserd Zap add_argument
parse_argsargsrf rW Zreadmer r r r s6
c