/usr/local/lib64/python3.6/site-packages/torch/autograd/__pycache__/
/usr/local/lib64/python3.6/site-packages/torch/autograd/__pycache__
NameSizeModeActions
anomaly_mode.cpython-36.pyc51490644editdlrm
forward_ad.cpython-36.pyc42590644editdlrm
function.cpython-36.pyc225870644editdlrm
functional.cpython-36.pyc315430644editdlrm
gradcheck.cpython-36.pyc504520644editdlrm
grad_mode.cpython-36.pyc96020644editdlrm
graph.cpython-36.pyc63160644editdlrm
profiler.cpython-36.pyc257530644editdlrm
profiler_legacy.cpython-36.pyc73220644editdlrm
profiler_util.cpython-36.pyc269410644editdlrm
variable.cpython-36.pyc8150644editdlrm
__init__.cpython-36.pyc110010644editdlrm
Edit: /usr/local/lib64/python3.6/site-packages/torch/autograd/__pycache__/gradcheck.cpython-36.pyc (50452B)
3 Eg @sddlZddlmZddlZddlmZddlZddlmZddl Z ddl m Z m Z m Z mZmZmZmZddlmZddlZGdddeZd d Zeeejd fd d dZdeeejd fdddZde ejeejfeeejdddZddZdeeejd fdddZdddZdd Z deejdd!d"Z!ee"eejfeejd fd#d$d%Z#deje ejejd&d'd(Z$dddd)d*Z%deejd fdd+d,Z&ddd-eeejd fd fdd.d/Z'd0d1Z(dd2d3Z)d4d5Z*d6d7Z+d8d9Z,deejdd:d;Z-dd?Z/eeejd feefdd@dAZ0dBZ1deejd fddCdDZ2dEdFZ3ddHdIZ4dJdKZ5eee ejddLdMZ6eee ejddNdOZ7eddPdQZ8dddRdSZ9eddTdUZ:dVdWZ;dXj<Z=dYdZZ>edd[d\Z?edd]d^Z@edd_d`ZAdadbZBdcddZCdeDddedfZEdgdhZFdidjZGdkdlZHdmdnZIddddodpdqZJdrdsZKdtduZLdvdwZMddxdyZNdzd{ZOd|d}ZPd~j<ZQddZRddZSddZTddddZUddddoddZVddddddGdddddd e d e efeeWeWeWeeeWeeeeeedddZXddZYdddddddGddddd e d efee eeWeWeWeeeWeeeeedddZZdS)N)_TensorOrTensors)is_tensor_like)product)CallableUnionOptionalIterableListTupleDict)vmapc@s eZdZdS)GradcheckErrorN)__name__ __module__ __qualname__rrD/usr/local/lib64/python3.6/site-packages/torch/autograd/gradcheck.pyr sr cCst|o|jp|jS)N)ris_floating_point is_complex)objrrr_is_float_or_complex_tensorsr.) input_tensorsreturncCsFg}x8|D]0}t|r |jr |j|j|j|ftjdq Wt|S)N)layout)r requires_gradappend new_zerosnumeltorchstridedtuple)rZ numel_outputouttrrr_allocate_jacobians_with_inputss  "r#)output_tensorsrcCsLg}||tjd}x0|D](}t|r|j|j||jff|qWt|S)N)dtypedevicer)rrrrrrr )r$Z numel_inputr%r&r!optionsr"rrr _allocate_jacobians_with_outputs#s   r(F)xonly_requiring_gradrccsbt|r|js| r^|VnBt|tjjr^t|t r^x&|D]}xt||D] }|VqLWq.get_stridecSsg|] }t|qSr)r5).0mrrr `sz _iter_tensor..r2c3s|]}||VqdS)Nr)r<k)indicesx_striderr bsz_iter_tensor..cSsg|] }t|qSr)r5)r<r=rrrr>escSsg|] }t|qSr)r5)r<r=rrrr>ms) is_sparseZ_nnzlistr6_indicesr"_valuesdatar5rtolistsumr3rr_mkldnn enumerateto_dense) Zx_tensorr;Zx_nnzZx_sizeZ x_indicesx_valuesr:Zx_valueZx_idxd_idxZx_tensor_denser)r@rAr _iter_tensor=s*  $ $$rOMbP?)rc Csg}|dkrt|t|}| r.zsExpected output to be non-complex. get_numerical_jacobian no longer supports functions that return complex outputs.cSs"g|]\}}t|r|jr|qSr)rr)r<r:arrrr>sz+_get_numerical_jacobian..T)input is_forward_ad) _as_tupleany ValueErrorrKzipr/)get_numerical_jacobian_wrt_specific_input) fninputsoutputstargetepsrT jacobians inp_indicesr:inpinp_idxrrr_get_numerical_jacobianqs"rc?csHtjd|dkrtdfdd}t||d||}tdd|DS) aDeprecated API to compute the numerical Jacobian for a given fn and its inputs. Args: fn: the function to compute the Jacobian for (must take inputs as a tuple) input: input to `fn` target: the Tensors wrt whom Jacobians are calculated (default=`input`) eps: the magnitude of the perturbation during finite differencing (default=`1e-3`) Returns: A list of Jacobians of `fn` (restricted to its first output) with respect to each input or target, if provided. Note that `target` may not even be part of `input` to `fn`, so please be **very careful** in this to not clone `target`. a?get_numerical_jacobian was part of PyTorch's private API and not meant to be exposed. We are deprecating it and it will be removed in a future version of PyTorch. If you have a specific use for this or feature request for this to be a stable API, please file us an issue at https://github.com/pytorch/pytorch/issues/newg?zaExpected grad_out to be 1.0. get_numerical_jacobian no longer supports values of grad_out != 1.0.cs|S)Nr)Zinps)rZrr fn_pack_inpssz,get_numerical_jacobian..fn_pack_inpsNcss|]}|dVqdS)rNr)r<Zjacobian_for_each_outputrrrrBsz)get_numerical_jacobian..)warningswarnrWrcr )rZr[r]r^grad_outrer_r)rZrget_numerical_jacobians   ricsd|j}|j|||}|j|||}|j|fddtfddt||DS)Ncs(||||d}|jjdS)Nr2)detachreshape)rRbret)nbhd_checks_fnnorm_vrrcomputes z,_compute_numerical_gradient..computec3s|]\}}||VqdS)Nr)r<rRrn)rrrrrBsz._compute_numerical_gradient..)clonecopy_r rX)rZentryvrqrporigZoutaZoutbr)rrrprqr_compute_numerical_gradients rxc Csg}|t|tr|dn|}|rt|tr<||ddn ||d}xft||D],\}}|j sjt||d} |j| qTWn*x(|D] }|s|j st|j|qW|S)Nrr2y?)r+r rXrAssertionErrorr) jvp_fndeltaZinput_is_complexrTjvpsZ ds_dx_tupZ ds_dy_tupZds_dxZds_dyZconj_w_drrr*_compute_numerical_jvps_wrt_specific_inputs&  r})jacobians_colsrc CsXt|||jjr|jndd}x6t|D]*\}}x |jD]\}}||||<q8Wq&W|S)N)r%)r(r%rrKitems) r~r\rSrr_r:jacobianr?rvrrr_combine_jacobian_colss r)rSmaybe_perturbed_inputrcCsL|jtjkr"|dk r|jS|Sn&|jtjkrD|r>|dk r>|S|Sn|SdS)N)rrrJZ to_mkldnn sparse_coo)rSr fast_moderrr_prepare_inputs   rc Csx|dk r dnd}|j|jksBtd|d|d|jd|jd |j|jksttd|d|d |jd|jd dS) Nzon index {idx} zPExpected `func` to return outputs with the same shape when inputs are perturbed zby z, but got: shapes z and .zPExpected `func` to return outputs with the same dtype when inputs are perturbed z, but got: dtypes )shaperyr%)Zoutput1Zoutput2r^idxZon_indexrrr"check_outputs_same_dtype_and_shapes $rcCsi}|dkr||n|}|js"tx`t|D]T\}} } t||||} || } tjt| |d} t| | || }t|||j ||| <q,Wt ||||j S)N)rr^) rryrO_with_prepare_inputs functoolspartialr_get_numerical_jvp_fnr}rrr)rZ input_idxr[r\r^rSrTZ jacobian_colsr)rrN wrapped_fninput_to_perturbrprzrrrrYs rY)check_grad_dtypesall_uc stjj}tdd|D}tdd|Dr4td|rPtfdd|D}ntfdd|D}|jBg}g} xft|D]Z\} } t| r| j r| j tj krtd|j | tj | } |j|j| d| j| qW|rxtt||D]\} \} } | j| j| t|| }tt|}xt|D]~\}}|j|\}}|rj|j|jkrjtd || |jd |d kr|| |jn|| |j|jdq4W| jqWnxt|D]\} } xttd d | jDD]\}}d| |<t|| }tt|}xt|D]t\}}|j|\}}|rX|j|jkrXtd |d krx|| ||jn|| ||j|jdq"Wd| |<qWqWWd QRX|S)aComputes the analytical Jacobian using forward mode AD of `fn(inputs)` using forward mode AD with respect to `target`. Returns N * M Jacobians where N is the number of tensors in target that require grad and M is the number of non-integral outputs. Contrary to other functions here, this function requires "inputs" to actually be used by the function. The computed value is expected to be wrong if the function captures the inputs by side effect instead of using the passed ones (many torch.nn tests do this). Args: fn: the function to compute the jacobian for inputs: inputs to `fn` outputs: provide precomputed outputs to avoid one extra invocation of fn check_grad_dtypes: if True, will check that the gradient dtype are valid all_u (optional): if provided, the Jacobian will be right multiplied with this vector Returns: A tuple of M N-tuples of tensors css |]}t|r|jr|VqdS)N)rr)r<r:rrrrB:sz6_get_analytical_jacobian_forward_ad..css|]}|jVqdS)N)r)r<r:rrrrB<szJExpected inputs to be non-complex for _get_analytical_jacobian_forward_ad.c3s|]}tdVqdS)r2N)r()r<r:)r\rrrB@sc3s|]}t|jVqdS)N)r(r)r<r:)r\rrrBBsz7MKLDNN inputs are not support for forward AD gradcheck.r2z'Forward AD gradient has dtype mismatch.rNcSsg|] }t|qSr)r5)r<r=rrrr>hsz7_get_analytical_jacobian_forward_ad..g?grkrk)rautogradZ forward_adr rVrWZ dual_levelrKrrrrJZ make_dual zeros_likerZ unpack_dualrXrtZview_asrUfilterrrr Zsqueeze_zero_rmrr6)rZr[r\rrZfwADZ tensor_inputsr_Zfw_gradsZ dual_inputsr:raZfw_graduZ raw_outputsZ dual_outputsZindex_oZd_ovalresZlin_idxZgrad_idxr)r\r#_get_analytical_jacobian_forward_ad$sZ     $   "rcCs6|jtjkr|j}n|jtjkr,|j}n|j}|S)N)rrrJrLrrsrG)rSrrrr_get_input_to_perturbys     rcsfdd}|S)Ncs<tfddttD}tddt|DS)Nc3s6|].\}}t|r*t||kr"ndn|VqdS)N)rr)r<r:rR)rrrrrrBsz;_with_prepare_inputs..wrapped_fn..css|]}|jVqdS)N)rs)r<rRrrrrBs)r rKrU)ra)rrZrrr[rrrsz(_with_prepare_inputs..wrapped_fnr)rZr[rrrrr)rrZrrr[rrsrcsfdd}|S)Ncst|S)N)rx)r{)r^rrprrrrzsz%_get_numerical_jvp_fn..jvp_fnr)rrr^rprzr)r^rrprrrsrcCsRt|tr8|djtjkrN|dj||dj|fSn|jtjkrN|j|S|S)Nrr2)r+r rrrrm)rrrrr_reshape_tensor_or_tuples    rcCs.t|tr"||d||dfS||SdS)Nrr2)r+r )rr?rrr_mul_tensor_or_tuples rc Csd||}t|}t||||d}tjt|d} t|||| } t||j}t||}t | ||j |S)NT)r^) rrrrrrrrrr}r) rZrr[rr^rTrSrrrprzrrr%_get_numerical_jvp_wrt_specific_inputs  rcCsg}xtt||D]\} \} } t|| || ||} g} t|}t| t|ksRtx*t| |D]\}}t|r^| j|q^q^W|dk rg}x&t|| D]\}}|jt||qW|j|q|j| qW|S)N) rKrXrrUr3ryrr_dot_with_type_promotion)rZr[r`func_outrall_vr^rTreduced_jacobiansr:rbrZall_JuZ filtered_JuZJuoutputjacobian_scalarsrvrrr_get_numerical_vJus"  rcCs@x:t||D],\}}|jdkr ||jj|kr dSq WdS)NrFT)rXrabsmax)Zj1Zj2atolZj1_xZj2_xrrr_check_jacobians_equals rcCst||}tt|d}d}d}xt|D]\}}||} ||} xt|D]\} } | dk rr| j| jkrrd}n| dk r| j| jkrd}| dkr| dd| fjqL| jtj ks| j n| } | dd| fj | j kst | j d| dd| f<qLWq*W|||fS)NTFr2rk)r#rDr/rKr6r%rrrrrLrryrm)Zlist_of_list_of_tensorsr[Z numel_outputsZ out_jacobiansdiff_input_listZcorrect_grad_sizesZcorrect_grad_typesr:Z tensor_listraZ out_jacobianjtensorZdenserrr_stack_and_check_tensorss$  ra NOTE: If your op relies on non-deterministic operations i.e., it is listed here: https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html this failure might be expected. If you are adding a new operator, please file an issue and then use one of the workarounds. The workaround depends on how your test invokes gradcheck/gradgradcheck. If the test - manually invokes gradcheck/gradgradcheck, then call gradcheck/gradgradcheck with `nondet_tol=` as a keyword argument. - is OpInfo-based (e.g., in test_ops.py), then modify the OpInfo for the test to have `gradcheck_nondet_tol=`. - is a Module test (e.g., in common_nn.py), then modify the corresponding module_test entry to have `gradcheck_nondet_tol=` cstt|dfdd}|rBt|j|}t|j|}nt|j}t|j}|sjjnd} t||| \} } } t||| \} }}t| | |}| r|rtd| std|std|dt | S) NTcstjj|dddS)NT) retain_graph allow_unused)rrgrad) grad_output)rrrrvjp_fn s z5_check_analytical_jacobian_attributes..vjp_fnr2zGradient has dtype mismatchz&Analytical gradient has incorrect sizezBackward is not reentrant, i.e., running backward with same input and grad_output multiple times gives different values, although analytical gradient matches numerical gradient.The tolerance for nondeterminism was r) rDr/(_get_analytical_vjps_wrt_specific_outputrs!_compute_analytical_jacobian_rowsrrrr FAILED_NONDET_MSG)r[r nondet_tolrrrvrvjps1vjps2 output_numel jacobians1types_oksizes_ok jacobians2_ reentrantr)rrr%_check_analytical_jacobian_attributess&   rc Csg}xt||D]\}}t||||d|d} g} xt| |D]z\} } | jjd} | jrtj| j} | jdd}| jdd}| j |j | dd|j | dq<| j | j | q.vjp_fn) rfrgrWrrDr/rrsrrr)r[rrrhrrrrrrrrrrr)rrrget_analytical_jacobian<s  rcCs t|||tddd}||S)NinfF)rr)rfloat)r[r\r output_idxr_rrr_get_analytical_jacobian[s rc Cstj|tjd}|jd}g}xzt|jD]j}|jd||<||}xLt|D]@\}}|dkrl|jg||t |tj r|j ndg7<qRWq,W|S)N) memory_formatr2g?rrk) rrlegacy_contiguous_formatviewr5rrrKrr+Tensorrs) r sample_outputZ grad_out_baseZ flat_grad_outZjacobians_rowsr grad_inputsr:Zd_xrrrrcs  .rcCsFg}||j|j}x,|D]$}|jt|tjr6|jndgqW|S)N)rmrrr+rrrs)rrrvZvjpsrvjprrrrws  $rcCs| r tdd|Dr tdd}xt|D]\}}t|r.|jr.|jtjkpZ|jtjksnt j d|d|j r||j n|}|j tjk rtddt|j|jDstd|d d }|jq.W|std d S) Ncss |]}t|tjr|jVqdS)N)r+rrrC)r<r"rrrrBsz _check_inputs..zTgradcheck expects all tensor inputs are dense when check_sparse_nnz is set to False.FzInput #z requires gradient and is not a double precision floating point or complex. This check will likely fail if all the inputs are not of double precision floating point or complex. css"|]\}}|dkp|dkVqdS)rr2Nr)r<stszrrrrBszThe zth input has a dimension with stride 0. gradcheck only supports inputs that are non-overlapping to be able to compute the numerical gradients correctly. You should call .contiguous on the input before passing it to gradcheck.Tzngradcheck expects at least one input tensor to require gradient, but none of the them have requires_grad=True.)rVr rKrrr%rfloat64 complex128rfrgrCrFrrJallrXr9r6 RuntimeErrorZ retain_gradrW) tupled_inputscheck_sparse_nnzZany_input_requiring_gradrracontentrrr _check_inputss&   rcCs8tdd|Drtdtdd|Dr4tddS)Ncss&|]}t|tjr|jtjkVqdS)N)r+rrrr)r<r"rrrrBsz!_check_outputs..zjSparse output is not supported at gradcheck yet. Please call to_dense() on the output of fn for gradcheck.css&|]}t|tjr|jtjkVqdS)N)r+rrrrJ)r<r"rrrrBszjMKLDNN output is not supported at gradcheck yet. Please call to_dense() on the output of fn for gradcheck.)rVrW)r\rrr_check_outputssrcCsLt||||d}x6|D].}x(|D] }tj|djdkr tdq WqWdS)N)r^rz3Numerical gradient for function expected to be zeroT)rcrnerIr )funcr[rr^Zjacobians_all_inputs_outputsZ%jacobians_all_outputs_and_fixed_inputrrrr _check_no_differentiable_outputss   rc Csjxdt||D]V\}}t|||||} x<| D]4} | jdkr.vjp..)r rX)rvresults)rrrrrsz_test_batched_grad..vjpcsg|]}tjqSr)r randn_like)r<r)rrrr>sz&_test_batched_grad..rjcsg|] }|qSrr)r<ZgO)rrrr>scSsg|]}tj|qSr)rstack)r<Zshardsrrrr>signorezBatching rule not implemented)messagez'torch.vmap is an experimental prototypez(While computing batched gradients, got: z )rDr/rrrrrr5rXrfcatch_warningsfilterwarningsr rrr rrKallcloser) rSrr grad_outputsexpectedr1exrrrr)rrrrr_test_batched_grads& .rcCs|tt|d}|stdtjj||dd|Ddd}xsz5_test_backward_mul_by_grad_output..)rzgrad is incorrect layout (z is not )z3grad is sparse tensor, but has incorrect sparse_dimz2grad is sparse tensor, but has incorrect dense_dimz&backward not multiplied by grad_outputrzgrad is incorrect typezgrad is incorrect size)rDr/r rrrrXr+rrrr.rZ sparse_dimZ dense_dimrLrreqrr%r&rCr6)r\r[rr grads_inputgiZdirrr!_test_backward_mul_by_grad_outputs:   $  * rcstt|dstdddfddddt||Dg}t|d d krx.warn_bc_breakingc sdd|D}ytjj||dd}Wn"tk rHtdYnXxCszN_test_undefined_grad..check_undefined_grad_support..T)rzExpected backward function to handle undefined output grads. Please look at "Notes about undefined output gradients" in "tools/autograd/derivatives.yaml"rzExpected all input grads to be undefined or zero when all output grads are undefined or zero. Please look at "Notes about undefined output gradients" in "tools/autograd/derivatives.yaml")rrrrr rXrr)output_to_checkZ grads_outputrrr:)rrrrcheck_undefined_grad_supportBs   z:_test_undefined_grad..check_undefined_grad_supportcSs(g|] }t|tjrtjjj|qSr)r+rr_C _functions UndefinedGrad)r<rQrrrr>Ysz(_test_undefined_grad..rr2cs,g|]$\}}|kr$tjjj|n|qSr)rrrr)r<rrQ)undef_grad_idxrrr>csc3s|]}|VqdS)Nr)r<r)rrrrBfsz'_test_undefined_grad..) rDr/r _differentiable_outputsr3r5rrKr)rr\r[Zoutputs_to_checkrr)rrrrr_test_undefined_grad6s  rcCs*t|tr|St|tr t|S|fSdS)N)r+r rD)r)rrrrUis   rUcCstddt|DS)Ncss|]}|jr|VqdS)N)r)r<rQrrrrBssz*_differentiable_outputs..)r rU)r)rrrrrsrc Csz| o|o||k}|o |o ||k}|r*dnd} |r6dnd} |p@|sFdnd| d| d} |r`d nd} | d | ||||fS) NZ imaginaryrealr[r\rzWhile considering the z part of complex z only, zcomputed with forward mode zWJacobian %smismatch for output %d with respect to input %d, numerical:%s analytical:%s r) analytical numericalrrcomplex_indices test_imagrTZout_is_complexZinp_is_complexpartelementprefixmoderrr_get_notallclose_msgvs    r cCs tt|S)N)rDrX)Zmatrix_of_tensorsrrr _transposesrcCs dd}||tj||tjfS)Ncsfdd}|S)Ncs"t|}tfdd|DS)Nc3s"|]}|jr|n|VqdS)N)r)r<rQ) fn_to_applyrrrBszU_real_and_imag_output..apply_to_c_outs..wrapped_fn..)rUr )r[Zouts)rZrrrrs zB_real_and_imag_output..apply_to_c_outs..wrapped_fnr)rZrrr)rZrrapply_to_c_outssz._real_and_imag_output..apply_to_c_outs)rrimag)rZrrrr_real_and_imag_outputsrcs(fdd}||dd||ddfS)Ncsfdd}|S)Ncs2t|}xD]}||||<qWt|S)N)rDrU)r[ new_inputsZshould_be_complex)complex_inp_indicesrZrrrrs zA_real_and_imag_input..apply_to_c_inps..wrapped_fnr)rZrr)r)rZrrapply_to_c_inpssz-_real_and_imag_input..apply_to_c_inpscSs|dS)Nry?yr)r)rrrsz&_real_and_imag_input..cSs|dS)Ny?r)r)rrrrsr)rZrrr)rr_real_and_imag_inputs rc Cstddt|D} tddt|D} | rt|\} }||}t|}|||||||||| | dd | |}t|}|| |||||||| | d n|||||||||| | rpddt|D}|rTt||\} }d d|D}||}t|}|||||||||| |ddd d d|D}| |}t|}|| |||||||| |dd n|||||||||| dd dS)NcSsg|]\}}|jr|qSr)r)r<r:rQrrrr>sz(_gradcheck_real_imag..css|]}|jVqdS)N)r)r<rQrrrrBsz'_gradcheck_real_imag..T)rr)rcSs$g|]\}}t|r|jr|qSr)rr)r<r:rarrrr>scSs&g|]}t|r|jr|jn|qSr)rrr)r<rarrrr>s)rruse_forward_adcSs&g|]}t|r|jr|jn|qSr)rrr)r<rarrrr>s)rr)r)rKrVrUrrr) gradcheck_fnrrrr\r^rtolrrcheck_forward_adrZcomplex_out_indicesZhas_any_complex_outputZreal_fnZimag_fnZ imag_func_outZ imag_outputsZ real_func_outZ real_outputsrZ imag_inputsZdiff_imag_func_outZ real_inputsZdiff_real_func_outrrr_gradcheck_real_imagsB            r)rrrc  Cs&t|}|st||||Stt||||| d} | rt||||d} xt| D]Z\}}xPt|D]D\}}| ||}t||j|j||s`t t ||||| | ddq`WqNWntxrt|D]f\}}t ||||}xNtt || |D]8\}\}}t||j|j||st t ||||| | qWqWdS)N)r^rT)rT)rT) rUrrrcrrK_allclose_with_type_promotiontor&r r rrX)rrrr\r^rrrrrrrrZanalytical_forwardr:Z n_per_outrnrRrQrrrr_slow_gradchecks$  r cCs(|jdkr|jdkst||jS)Nr2)r7ryrI)rrvrrrrsrcCs8tj|j|j}|j|d}|j|d}tj||||S)N)r%)rZ promote_typesr%rr)rRrnrrZ promoted_typerrrrs  rcCs(|tjkrtjS|tjkr tjS|SdS)N)rrrZ complex64Zfloat32)r%rrr_to_real_dtypes   r!cCs|jtjkrr|j}|r"t|jn|j}tj|j|dj||j dj |j }||j }tj |j||j}n>|rt|jn|j}tj|j|dj||j d}||j }|S)N) generator)r%r&)rrrrFr!r%Zrandrrr&rmrZnormZsparse_coo_tensorrEr6)r)r"Zdowncast_complexrMr%valuesZvecrrr_vec_from_tensors    r$cCs.ddt|D}dd|Ddd|DfS)NcSs&g|]\}}t|r|jr||fqSr)rr)r<r:r"rrrr>sz$_get_inp_tensors..cSsg|] }|dqS)rr)r<tuprrrr>scSsg|] }|dqS)r2r)r<r%rrrr>s)rK)rZ inp_idx_tuprrr_get_inp_tensorssr&cCsvt|tr|dn|}|jtjkr.tjj|n|j}|dkrBdn|jtjkrZtjj|n|j}|t|t|S)Nrg?)r+r rrrsparserIr)rrrvZsum_uZsum_vrrr_adjusted_atols  ,r(a Fast gradcheck failed but element-wise differences are small. This means that the test might've passed in slow_mode! If you are adding a new operator, please file an issue and then use one of the workarounds. The workaround depends on how your test invokes gradcheck/gradgradcheck: If the test - manually invokes gradcheck/gradgradcheck, then call gradcheck/gradgradcheck with `fast_mode=False` as a keyword argument. - is OpInfo-based (e.g., in test_ops.py), then modify the OpInfo for the test to have `gradcheck_fast_mode=False` - is a Module test (e.g., in common_nn.py), then modify the corresponding module_test entry to have `gradcheck_fast_mode=False` cst||d}|rPfdd} t| f|fdd} nt|} || jj} tj| |||} d|d| d| d} | r| t7} | S) N)rTcs t}||<t|S)N)rDrU)rar)rrrrrrnew_fn:sz,_run_slow_mode_and_get_error..new_fnra The above quantities relating the numerical and analytical jacobians are computed in fast mode. See: https://github.com/pytorch/pytorch/issues/53876 for more background about fast mode. Below, we recompute numerical and analytical jacobians in slow mode: Numerical: z Analytical: z1 The max per-element difference (slow mode) is: z. )rcrrrrrrFAST_FAIL_SLOW_OK_MSG)rrr\rrrrrTZslow_numericalr)Zslow_analyticalZ slow_max_diffZ slow_allclosemsgr)rrrrr_run_slow_mode_and_get_error6s"r,cCs"|jtjkr|jjdS|SdS)Nr2rk)rrrrLrm)rrrr_to_flat_dense_if_sparseSs r-c stjg}g}xn|D]f}t|d}t|}|jrht|d}|j||ft|} |j|| fq|j||j|qW|rdnfdd|D} | ||fS)NTcsg|]}t|qSr)r$)r<r!)g_cpurrr>jsz!_make_vectors..)r Generatorr$r-rr) inp_tensorsr\rr all_u_denseraurZur_denseZuiZui_denserr)r.r _make_vectorsZs    r3)rTc  Csxt|D]\} } xt| D]\}}| r6|| |}n ||| }|j|jd}t| || |rf||nd}t||j|j||st|||| ||| | }tt|||| || | |qWq WdS)N)r&)rKrr&r(rr,r r )Zall_analyticalZ all_numericalrrr\rrrrrrrTr:Zall_numerical_for_input_irrrRZ updated_atolZ jacobians_strrrr!_check_analytical_numerical_equalns r4c Cst|\} } t| || d\}}}t||| ||||| d}| r`|dksHtt||t|||d}n*|sxt|||| |||t||||||}t||| |||||||| | d dS)N)r)rT)rrT) r&r3rryrrUrrr4)rrr[r\r^rrrrrrrZinp_tensors_idxr0rrr1Z numerical_vJuZanalytical_vJurrr_fast_gradcheck~s    r5gư>gh㈵>T) r^rrraise_exceptionrrcheck_undefined_gradrcheck_batched_gradrr)rr[r^rrr6rrr7rr8rrrc CsTtj} | jd|sFy tf| Stk rB}zdSd}~XqPXn tf| SdS)a Check gradients computed via small finite differences against analytical gradients w.r.t. tensors in :attr:`inputs` that are of floating point or complex type and with ``requires_grad=True``. The check between numerical and analytical gradients uses :func:`~torch.allclose`. For most of the complex functions we consider for optimization purposes, no notion of Jacobian exists. Instead, gradcheck verifies if the numerical and analytical values of the Wirtinger and Conjugate Wirtinger derivatives are consistent. Because the gradient computation is done under the assumption that the overall function has a real-valued output, we treat functions with complex output in a special way. For these functions, gradcheck is applied to two real-valued functions corresponding to taking the real components of the complex outputs for the first, and taking the imaginary components of the complex outputs for the second. For more details, check out :ref:`complex_autograd-doc`. .. note:: The default values are designed for :attr:`input` of double precision. This check will likely fail if :attr:`input` is of less precision, e.g., ``FloatTensor``. .. warning:: If any checked tensor in :attr:`input` has overlapping memory, i.e., different indices pointing to the same memory address (e.g., from :func:`torch.expand`), this check will likely fail because the numerical gradients computed by point perturbation at such indices will change values at all other indices that share the same memory address. Args: func (function): a Python function that takes Tensor inputs and returns a Tensor or a tuple of Tensors inputs (tuple of Tensor or Tensor): inputs to the function eps (float, optional): perturbation for finite differences atol (float, optional): absolute tolerance rtol (float, optional): relative tolerance raise_exception (bool, optional): indicating whether to raise an exception if the check fails. The exception gives more information about the exact nature of the failure. This is helpful when debugging gradchecks. check_sparse_nnz (bool, optional): if True, gradcheck allows for SparseTensor input, and for any SparseTensor at input, gradcheck will perform check at nnz positions only. nondet_tol (float, optional): tolerance for non-determinism. When running identical inputs through the differentiation, the results must either match exactly (default, 0.0) or be within this tolerance. check_undefined_grad (bool, optional): if True, check if undefined output grads are supported and treated as zeros, for ``Tensor`` outputs. check_batched_grad (bool, optional): if True, check if we can compute batched gradients using prototype vmap support. Defaults to False. check_forward_ad (bool, optional): if True, check that the gradients computed with forward mode AD match the numerical ones. Defaults to False. fast_mode (bool, optional): Fast mode for gradcheck and gradgradcheck is currently only implemented for R to R functions. If none of the inputs and outputs are complex a faster implementation of gradcheck that no longer computes the entire jacobian is run; otherwise, we fall back to the slow implementation. Returns: True if all differences satisfy allclose condition r6FN)localscopypop_gradcheck_helperr )rr[r^rrr6rrr7rr8rrargserrr gradchecksJ   r?c  Cst|} t| ||| } t| }t|| r2tnt}t||| | |||||| |d x$t|D]\}}| r^t| ||q^Wt || ||rt ||| dS)N)rrT) rUrrrr5r rrKrrr)rr[r^rrrrr7rr8rrrrr\rr:rQrrrr<s    r<) r^rrgen_non_contig_grad_outputsr6rr7rr8r)rr[rr^rrr@r6rr7rr8rrc  st|} |dkr@fddt| }tfdd|D}nt|}t|fdd}t|| ||||||| | | | d S) a Check gradients of gradients computed via small finite differences against analytical gradients w.r.t. tensors in :attr:`inputs` and :attr:`grad_outputs` that are of floating point or complex type and with ``requires_grad=True``. This function checks that backpropagating through the gradients computed to the given :attr:`grad_outputs` are correct. The check between numerical and analytical gradients uses :func:`~torch.allclose`. .. note:: The default values are designed for :attr:`input` and :attr:`grad_outputs` of double precision. This check will likely fail if they are of less precision, e.g., ``FloatTensor``. .. warning:: If any checked tensor in :attr:`input` and :attr:`grad_outputs` has overlapping memory, i.e., different indices pointing to the same memory address (e.g., from :func:`torch.expand`), this check will likely fail because the numerical gradients computed by point perturbation at such indices will change values at all other indices that share the same memory address. Args: func (function): a Python function that takes Tensor inputs and returns a Tensor or a tuple of Tensors inputs (tuple of Tensor or Tensor): inputs to the function grad_outputs (tuple of Tensor or Tensor, optional): The gradients with respect to the function's outputs. eps (float, optional): perturbation for finite differences atol (float, optional): absolute tolerance rtol (float, optional): relative tolerance gen_non_contig_grad_outputs (bool, optional): if :attr:`grad_outputs` is ``None`` and :attr:`gen_non_contig_grad_outputs` is ``True``, the randomly generated gradient outputs are made to be noncontiguous raise_exception (bool, optional): indicating whether to raise an exception if the check fails. The exception gives more information about the exact nature of the failure. This is helpful when debugging gradchecks. nondet_tol (float, optional): tolerance for non-determinism. When running identical inputs through the differentiation, the results must either match exactly (default, 0.0) or be within this tolerance. Note that a small amount of nondeterminism in the gradient will lead to larger inaccuracies in the second derivative. check_undefined_grad (bool, optional): if True, check if undefined output grads are supported and treated as zeros check_batched_grad (bool, optional): if True, check if we can compute batched gradients using prototype vmap support. Defaults to False. fast_mode (bool, optional): if True, run a faster implementation of gradgradcheck that no longer computes the entire jacobian. Returns: True if all differences satisfy allclose condition Ncs@tj|js|jr|n|jtjd}r8tjj|}|jS)N)r) rrrrdoublerZtestingZmake_non_contiguousZrequires_grad_)r)y)r@rrrTs $ z!gradgradcheck..randn_likec3s|]}|VqdS)Nr)r<r))rrrrB[sz gradgradcheck..csf|d }| d}t|}tdd|D}tjj|||ddd}tdd|D}|S)Ncss |]}t|r|jr|VqdS)N)rr)r<r)rrrrBesz2gradgradcheck..new_func..T)Z create_graphrcss|]}|dk r|VqdS)Nr)r<grrrrBis)rr rrr)r=Z input_argsrr\r)r num_outputsrrnew_funcas zgradgradcheck..new_func) r^rrr6rr7rr8r)rUr r3r?)rr[rr^rrr@r6rr7rr8rrr\Ztupled_grad_outputsrEr)rr@rDrr gradgradcheck sE   rF)NN)F)NNrPF)NrPrd)F)F)N)NF)F)F)FN)rrd)FF)F)N)[rZ torch.typesrZ torch.testingZtorch.overridesrr, itertoolsrrftypingrrrrr r r Ztorch._vmap_internalsr rrr rrr#r(boolr/rOrcrirxr}intrrrrYrrrrrrrrrrrrrrrrrrrrrrrrrrrrUrr.r rrrrr rrr!r$r&r(r*r,r-r3r4r5rr?r<rFrrrrs   $  * 4% ! &T    "&%    $+"3    - ",F,