/usr/local/lib64/python3.6/site-packages/torch/distributed/launcher/__pycache__
Edit: /usr/local/lib64/python3.6/site-packages/torch/distributed/launcher/__pycache__/api.cpython-36.pyc (9314B)
3
Eg+) @ sf d dl Z d dlZd dlmZmZ d dlmZmZmZm Z m
Z
m Z m Z m
Z
d dljjjjZd dlmZmZ d dlmZmZ d dlmZ d dl m Z d dl m Z d d l m!Z! d d
l"m#Z# d d l$m%Z% e% Z&eG d d
d
Z'G dd dZ(e'ej)dddZ*e ee+df e e e+dddZ,e!e
e
e+ e
e- f dddZ.e'e ee+df e e ee-ef dddZ/dS ) N) dataclassfield)AnyCallableDictListOptionalUnioncastTuple)eventsmetrics)
WorkerSpec WorkerState)LocalElasticAgent)Std)ChildFailedError)RendezvousParameters)parse_rendezvous_endpoint)
get_loggerc @ s e Zd ZU dZeeedZe dZ
e
dZ e dZ e e
edZee ef d
ZedZedZed Ze d
Zee ejZe eeeef f ejZ e eeeef f e
edZ ee e f d d Z d
S ) LaunchConfiga
Creates a rendezvous config.
Args:
min_nodes: Minimum amount of nodes that the user function will
be launched on. Elastic agent ensures that the user
function start only when the min_nodes amount enters
the rendezvous.
max_nodes: Maximum amount of nodes that the user function
will be launched on.
nproc_per_node: On each node the elastic agent will launch
this amount of workers that will execute user
defined function.
rdzv_backend: rdzv_backend to use in the rendezvous (zeus-adapter, etcd).
rdzv_endpoint: The endpoint of the rdzv sync. storage.
rdzv_configs: Key, value pair that specifies rendezvous specific configuration.
rdzv_timeout: Legacy argument that specifies timeout for the rendezvous. It is going
to be removed in future versions, see the note below. The default timeout is 900 seconds.
rdzv_id: The unique run id of the job (if not passed a unique one will be
deduced from run environment - flow workflow id in flow - or auto generated).
role: User defined role of the worker (defaults to "trainer").
max_restarts: The maximum amount of restarts that elastic agent will conduct
on workers before failure.
monitor_interval: The interval in seconds that is used by the elastic_agent
as a period of monitoring workers.
start_method: The method is used by the elastic agent to start the
workers (spawn, fork, forkserver).
log_dir: base log directory where log files are written. If not set,
one is created in a tmp dir but NOT removed on exit.
redirects: configuration to redirect stdout/stderr to log files.
Pass a single ``Std`` enum to redirect all workers,
or a mapping keyed by local_rank to selectively redirect.
tee: configuration to "tee" stdout/stderr to console + log file.
metrics_cfg: configuration to initialize metrics.
..note:
`rdzv_timeout` is a legacy argument that will be removed in future.
Set the timeout via `rdzv_configs['timeout']`
Z default_roleZetcd)default_factory spawnNc C s4 d}| j dkr | j | jd< nd| jkr0|| jd< d S )Ni r timeout) rdzv_timeout rdzv_configs)selfZdefault_timeout r" J/usr/local/lib64/python3.6/site-packages/torch/distributed/launcher/api.py
__post_init__W s
zLaunchConfig.__post_init__r ) __name__
__module__ __qualname____doc__int min_nodes max_nodesnproc_per_noderun_idstrrole
rdzv_endpoint rdzv_backendr dictr r r r max_restartsmonitor_intervalfloat start_methodlog_dirr r NONE redirectsr tee metrics_cfgr$ r" r" r" r# r s$
) r c @ s2 e Zd ZdZeeeedf dddZdd Z dS )elastic_launcha
Launches an torchelastic agent on the container that invoked the entrypoint.
1. Pass the ``entrypoint`` arguments as non ``kwargs`` (e.g. no named parameters)/
``entrypoint`` can be a function or a command.
2. The return value is a map of each worker's output mapped
by their respective global rank.
Usage
::
def worker_fn(foo):
# ...
def main():
# entrypoint is a function.
outputs = elastic_launch(LaunchConfig, worker_fn)(foo)
# return rank 0's output
return outputs[0]
# entrypoint is a command and ``script.py`` is the python module.
ouptuts = elestic_launch(LaunchConfig, "script.py")(args)
ouptuts = elestic_launch(LaunchConfig, "python")("script.py")
N)config
entrypointc C s || _ || _d S )N)_config _entrypoint)r! r= r> r" r" r# __init__z s zelastic_launch.__init__c G s t | j| jt|S )N) launch_agentr? r@ list)r! argsr" r" r# __call__ s zelastic_launch.__call__)
r% r& r' r( r r r r. rA rE r" r" r" r# r< _ s r< )r= returnc C s6 | j | j| jd}tjdtjjttt tj
f |dS )N)r1 r- r/ z&torch.distributed.elastic.launch_agent)namesourcemetadata) r1 r- r/ r EventZ EventSourceZAGENTr
r r. ZEventMetadataValue)r= rI r" r" r# _construct_event s
rK )r> rD rF c C sF t | tr| jS t | tr>| tjkr8tdd |D dS | S ndS dS )a Retrive entrypoint name with the rule:
1. If entrypoint is a function, use ``entrypont.__qualname__``.
2. If entrypoint is a string, check its value:
2.1 if entrypoint equals to ``sys.executable`` (like "python"), use the first element from ``args``
which does not start with hifen letter (for example, "-u" will be skipped).
2.2 otherwise, use ``entrypoint`` value.
3. Otherwise, return empty string.
c s s | ]}|d dkr|V qdS )r -Nr" ).0argr" r" r#
s z'_get_entrypoint_name..r N)
isinstancer r% r. sys
executablenext)r> rD r" r" r# _get_entrypoint_name s
rT )rdzv_parametersrF c C sX | j dkrdS | j}|j }|s(tdt|dd\}}|d krPtd| d||fS )
NZstaticzKEndpoint is missing in endpoint. Try to add --master_addr and --master_portr ) default_portz port is missing in endpoint: z . Try to specify --master_port)NNr r )backendendpointstrip
ValueErrorr )rU rX master_addr master_portr" r" r# _get_addr_and_port s
r] )r= r> rD rF c
C s | j s*ttj j}tjd| || _ t||}tjd| d| j d| j
d| j d| j d| j d| j
d | j d
| j d | j d | j d
| j d tf | j | j
| j | j | j
d| j}d }tj|}t|\}} zyt| j| j |t||| j| j| j| j|| d }
| jr$t j | jnd } t j | t |
| j | jd}|j! } t"j#|j$t%j& | j' rxt(|| j)dn| j*S W nV t(k
r Y n@ t+k
r |rt"j#|j$t%j, nt"j#t-| Y nX W d |j. X d S )Nz*config has no run_id, generate a new one: zDStarting elastic_operator with launch configs:
entrypoint : z
min_nodes : z
max_nodes : z
nproc_per_node : z
run_id : z
rdzv_backend : z
rdzv_endpoint : z
rdzv_configs : z
max_restarts : z
monitor_interval : z
log_dir : z
metrics_cfg :
)rW rX r- r* r+ ) r/ Zlocal_world_sizer> rD rdzv_handlerr3 r4 r9 r: r[ r\ )specr6 r7 )rG failures)/r- r. uuiduuid4r) loggerwarningrT infor* r+ r, r1 r0 r r3 r4 r7 r; r
rdzv_registryZget_rendezvous_handlerr] r r/ tupler9 r: r
Z
MetricsConfigZinitialize_metricsr r6 runr recordZget_agent_status_eventr Z SUCCEEDEDZ is_failedr ra Z
return_values ExceptionZFAILEDrK shutdown)
r= r> rD r- Zentrypoint_namerU Zagentr_ r[ r\ r` cfgresultr" r" r# rB sb
f
rB )0rQ rb Z dataclassesr r typingr r r r r r r
r Z-torch.distributed.elastic.rendezvous.registryZ distributedZelasticZ
rendezvousregistryrg Ztorch.distributed.elasticr r
Z*torch.distributed.elastic.agent.server.apir r Z:torch.distributed.elastic.agent.server.local_elastic_agentr Z)torch.distributed.elastic.multiprocessingr Z0torch.distributed.elastic.multiprocessing.errorsr Z$torch.distributed.elastic.rendezvousr Z*torch.distributed.elastic.rendezvous.utilsr Z'torch.distributed.elastic.utils.loggingr rd r r< rJ rK r. rT r) r] rB r" r" r" r# s0 ( D'