/usr/local/lib/python3.6/site-packages/datasets/features/__pycache__
Edit: /usr/local/lib/python3.6/site-packages/datasets/features/__pycache__/audio.cpython-36.pyc (11473B)
3
<%Eg5 @ s d dl Z d dlmZmZ d dlmZ d dlmZmZm Z m
Z
m Z m Z d dl
Zd dlZd dlmZ ddlmZ ddlmZ dd lmZ dd
lmZmZ erd d l m Z eG d
d dZ dS ) N) dataclassfield)BytesIO)
TYPE_CHECKINGAnyClassVarDictOptionalUnion)version )config)xopen)
array_cast)no_op_if_value_is_nullstring_to_dict ) FeatureTypec @ sN e Zd ZU dZdZee dZedZ e dZ
ee
dZ e
e ejej ej dZe
e ed dddZe dd Zee ef ed
d d Zd!eeee ee edf f ed
ddZedee df f dddZ eej ej f ej dddZ d"ej eej dddZ d#eee ee edf f ddd Z!d$d d Z"d d Z#dS )%Audioa Audio Feature to extract audio data from an audio file.
Input: The Audio feature accepts as input:
- A :obj:`str`: Absolute path to the audio file (i.e. random access is allowed).
- A :obj:`dict` with the keys:
- path: String with relative path of the audio file to the archive file.
- bytes: Bytes content of the audio file.
This is useful for archived files with sequential access.
- A :obj:`dict` with the keys:
- path: String with relative path of the audio file to the archive file.
- array: Array containing the audio sample
- sampling_rate: Integer corresponding to the samping rate of the audio sample.
This is useful for archived files with sequential access.
Args:
sampling_rate (:obj:`int`, optional): Target sampling rate. If `None`, the native sampling rate is used.
mono (:obj:`bool`, default ``True``): Whether to convert the audio signal to mono by averaging samples across
channels.
decode (:obj:`bool`, default ``True``): Whether to decode the audio data. If `False`,
returns the underlying dictionary in the format {"path": audio_path, "bytes": audio_bytes}.
Example:
```py
>>> from datasets import load_dataset, Audio
>>> ds = load_dataset("PolyAI/minds14", name="en-US", split="train")
>>> ds = ds.cast_column("audio", Audio(sampling_rate=16000))
>>> ds[0]["audio"]
{'array': array([ 2.3443763e-05, 2.1729663e-04, 2.2145823e-04, ...,
3.8356509e-05, -7.3497440e-06, -2.1754686e-05], dtype=float32),
'path': '/root/.cache/huggingface/datasets/downloads/extracted/f14948e0e84be638dd7943ac36518a4cf3324e8b7aa331c5ab11541518e9368c/en-US~JOINT_ACCOUNT/602ba55abb1e6d0fbce92065.wav',
'sampling_rate': 16000}
```
NTdict)bytespathF)defaultinitreprc C s | j S )N)pa_type)self r A/usr/local/lib/python3.6/site-packages/datasets/features/audio.py__call__G s zAudio.__call__)valuereturnc C s y ddl }W n, tk
r8 } ztd|W Y dd}~X nX t|trNd|dS d|krt }|j||d |d dd |j ddS |jd dk otj j
|d rV|d j d
rD|jddkrt d |jd rt
j|d t
jd
jt
jd }n t
j|d dddjt
jd }tt }|j|||d dd |j ddS d|jd dS nF|jd dk sv|jd dk r|jd |jd dS td| ddS )zEncode example into a format for Arrow.
Args:
value (:obj:`str` or :obj:`dict`): Data passed as input to Audio feature.
Returns:
:obj:`dict`
r Nz;To support encoding audio data, please install 'soundfile'.)r r array
sampling_rateZwav)formatr ZpcmzBTo use PCM files, please specify a 'sampling_rate' in Audio objectr )dtypei hr)r% modezUAn audio sample should have one of 'path' or 'bytes' but they are missing or None in .) soundfile ImportError
isinstancestrr writegetvaluegetosr isfileendswithKeyErrornpZ
frombufferZint16ZastypeZfloat32Zmemmapr
ValueError)r r sferrbufferZ bytes_valuer r r encode_exampleJ s2
"
zAudio.encode_example)r token_per_repo_idr! c C s | j std|d dk r.|d t|d fn
|d df\}}|dkr`|dkr`td| dn|dk r|jdr| j|r~|n|\}}nd|dk r|jdr|r| j|d\}}q| j|d|d \}}n&|r| j|\}}n| j||d \}}|||d
S ) a Decode example audio file into audio data.
Args:
value (:obj:`dict`): a dictionary with keys:
- path: String with relative audio file path.
- bytes: Bytes of the audio file.
token_per_repo_id (:obj:`dict`, optional): To access and decode
audio files from private repositories on the Hub, you can pass
a dictionary repo_id (str) -> token (bool or str)
Returns:
dict
zMDecoding is disabled for this feature. Please use Audio(decode=True) instead.r Nr zJAn audio sample should have one of 'path' or 'bytes' but both are None in r) mp3opus)r; )r r" r# )decode RuntimeErrorr r6 r3 _decode_mp3_decode_non_mp3_file_like_decode_non_mp3_path_like)r r r; r filer" r# r r r decode_examplex s 0zAudio.decode_exampler )r! c C s, ddl m} | jrtd|d|ddS )z[If in the decodable state, raise an error, otherwise flatten the feature into a dictionary.r )Valuez'Cannot flatten a decoded Audio feature.binarystring)r r )featuresrE r> r6 )r rE r r r flatten s
z
Audio.flatten)storager! c C s( t jj|jrJt jdgt| t j d}t jj||gddg|j d}nt jj
|jr~|jj dr~t jdd |j D }nt jj
|jr |jj
dd kr|jd}n t jdgt| t j d}|jj
dd kr|jd}n t jdgt| t j d}t jj||gddg|j d}t|| jS )
a Cast an Arrow array to the Audio arrow storage type.
The Arrow types that can be converted to the Audio pyarrow storage type are:
- pa.string() - it must contain the "path" data
- pa.struct({"bytes": pa.binary()})
- pa.struct({"path": pa.string()})
- pa.struct({"bytes": pa.binary(), "path": pa.string()}) - order doesn't matter
Args:
storage (Union[pa.StringArray, pa.StructArray]): PyArrow array to cast.
Returns:
pa.StructArray: Array in the Audio arrow storage type, that is
pa.struct({"bytes": pa.binary(), "path": pa.string()})
N)typer r )maskr" c S s$ g | ] }|d k r t j|nd qS )N)r r: ).0xr r r
s z&Audio.cast_storage..r )patypes is_stringrK r" lenrF StructArray from_arraysis_nullZ is_structZget_all_field_indices to_pylistZget_field_indexr rG r r )r rJ bytes_array
path_arrayr r r cast_storage s zAudio.cast_storage)rJ
drop_pathsr! c s t dd tj fdd|j D tj d}|rPtjdgt| tj dn|jd}tjj ||gddg|j
d }t || j S )
av Embed audio files into the Arrow array.
Args:
storage (pa.StructArray): PyArrow array to embed.
drop_paths (bool, default ``True``): If True, the paths are set to None.
Returns:
pa.StructArray: Array in the Audio arrow storage type, that is
pa.struct({"bytes": pa.binary(), "path": pa.string()})
c S s" t | d}|j }W d Q R X |S )Nrb)r read)r fbytes_r r r
path_to_bytes s z*Audio.embed_storage..path_to_bytesc s8 g | ]0}|d k r0|d d kr( |d q2|d nd qS )Nr r r )rM rN )r` r r rO s z'Audio.embed_storage..)rK Nr r )rL )
r rP r" rW rF rS rG r rT rU rV r r )r rJ r[ rX rY r )r` r
embed_storage s
* zAudio.embed_storage)r; c
" C s y dd l }W n, tk
r8 } ztd|W Y d d }~X nX |p@i }|dkrpdd l}tj|jtjdk rptd
|jdd }y t|t j
d }|| } W n t t fk
r d } Y nX t
|d
| d }
|j|
| j| jd \} } W d Q R X | | fS )Nr z:To support decoding audio files, please install 'librosa'.r= z1.0.30z4Decoding .opus files requires 'libsndfile'>=1.0.30, zPit can be installed via conda: `conda install -c conda-forge libsndfile>=1.0.30`z::r repo_idr\ )use_auth_token)srmonozDecoding .opus files requires 'libsndfile'>=1.0.30, it can be installed via conda: `conda install -c conda-forge libsndfile>=1.0.30`)librosar+ r* r parse__libsndfile_version__r? splitr r
ZHUB_DATASETS_URLr6 r4 r loadr# re )
r r r$ r; rg r8 r*
source_urlrb rc r^ r" r# r r r rB s(
"z Audio._decode_non_mp3_path_likec C s ydd l }dd l}W n, tk
r@ } ztd|W Y d d }~X nX |dkrhtj|jtjdk rhtd |j|\}}|j}| j r|j
|}| j r| j |kr|j ||| j dd}| j }||fS )
Nr zJTo support decoding audio files, please install 'librosa' and 'soundfile'.r= z1.0.30z4Decoding .opus files requires 'libsndfile'>=1.0.30, zPit can be installed via conda: `conda install -c conda-forge libsndfile>=1.0.30`Z kaiser_best)Zres_typezDecoding .opus files requires 'libsndfile'>=1.0.30, it can be installed via conda: `conda install -c conda-forge libsndfile>=1.0.30`)
rg r* r+ r rh ri r? r] Tre Zto_monor# Zresample)r rC r$ rg r7 r8 r" r# r r r rA s$
z Audio._decode_non_mp3_file_likec C s ydd l }dd lj}W n, tk
rB } ztd|W Y d d }~X nX y|jd W n, tk
r~ } ztd|W Y d d }~X nX |j|dd\}}| jr| j|krt| d s| j j
|kr|j || j| _ | j |}| j}|j }| j
r|jdd}||fS ) Nr zCTo support decoding 'mp3' audio files, please install 'torchaudio'.Zsox_ioz idr- r% r rP structrF rG r r r _typer r
r r: r rD rI Z StringArrayrT rZ ra rB rA r@ r r r r r s$
( / % ! &
r ) r1 Z dataclassesr r ior typingr r r r r r
rq r5 ZpyarrowrP packagingr r
Z#download.streaming_download_managerr tabler Zutils.py_utilsr r rH r r r r r r s