Waiter_check_interval_seconds ( int) – Number of seconds between polling the state of the notebook.ĭefaults to 60 seconds. Waiter_countdown ( int) – Total amount of time the operator will wait for the notebook to stop.ĭefaults to 25 * 60 seconds. Waiter_delay ( int | None | ) – Number of seconds between polling the state of the notebook. Waiter_max_attempts ( int | None | ) – Maximum number of tries before failing. Tags ( list | None) – Optional list of key value pair to associate with the notebook execution. Group to associate with the master instance of the EMR cluster for this notebook execution. Master_instance_security_group_id ( str | None) – Optional unique ID of an EC2 security Security group to associate with the EMR notebook for this notebook execution. Notebook_instance_security_group_id ( str | None) – The unique identifier of the Amazon EC2 Notebook_params ( str | None) – Input parameters in JSON format passed to the EMR notebook at Notebook_execution_name ( str | None) – Optional name for the notebook execution. Service_role ( str) – The name or ARN of the IAM role that is used as the service roleįor Amazon EMR (the EMR role) for the notebook execution. Relative to the path specified for the EMR notebook.Ĭluster_id ( str) – The unique identifier of the EMR cluster the notebook is attached to. Relative_path ( str) – The path and file name of the notebook file for this execution, Start an EMR notebook execution ParametersĮditor_id ( str) – The unique identifier of the EMR notebook to use for notebook execution. EmrStartNotebookExecutionOperator ( editor_id, relative_path, cluster_id, service_role, notebook_execution_name = None, notebook_params = None, notebook_instance_security_group_id = None, master_instance_security_group_id = None, tags = None, wait_for_completion = False, aws_conn_id = 'aws_default', waiter_max_attempts = NOTSET, waiter_delay = NOTSET, waiter_countdown = 25 * 60, waiter_check_interval_seconds = 60, ** kwargs ) ¶ execute_complete ( context, event = None ) ¶ class .operators.emr. Refer to get_template_context for more context. Template_fields : Sequence = ('job_flow_id', 'job_flow_name', 'cluster_states', 'steps', 'execution_role_arn') ¶ template_ext : Sequence = ('.json',) ¶ template_fields_renderers ¶ ui_color = '#f9c915' ¶ operator_extra_links = () ¶ execute ( context ) ¶Ĭontext is the same dictionary used as when rendering jinja templates. This mode requires aiobotocore module to be installed. (default: True)ĭeferrable ( bool) – If True, the operator will wait asynchronously for the job to complete. Wait_for_completion – Whether to wait for job run completion. Wait_for_completion ( bool) – If True, the operator will wait for all the steps to be completed.Įxecution_role_arn ( str | None) – The ARN of the runtime role for a step on the cluster.ĭo_xcom_push – if True, job_flow_id is pushed to XCom with key job_flow_id. Steps ( list | str | None) – boto3 style steps or reference to a steps file (must be ‘.json’) to (templated)Ĭluster_states ( list | None) – Acceptable cluster states when searching for JobFlow id by job_flow_name.Īws_conn_id ( str) – aws connection to uses Exactly one cluster like this should exist or will fail. will search for id of JobFlow with matching name in one of the states in Job_flow_name ( str | None) – name of the JobFlow to add steps to. Job_flow_id ( str | None) – id of the JobFlow to add steps to. EmrAddStepsOperator ( *, job_flow_id = None, job_flow_name = None, cluster_states = None, aws_conn_id = 'aws_default', steps = None, wait_for_completion = False, waiter_delay = 30, waiter_max_attempts = 60, execution_role_arn = None, deferrable = conf.getboolean('operators', 'default_deferrable', fallback=False), ** kwargs ) ¶įor more information on how to use this operator, take a look at the guide: Operator to delete EMR Serverless application.Ĭlass .operators.emr. Operator to stop an EMR Serverless application. Operator to create Serverless EMR Application. An operator that adds steps to an existing EMR job_flow.Īn operator that starts an EMR notebook execution.Īn operator that stops a running EMR notebook execution.Īn operator that creates EMR on EKS virtual clusters.Īn operator that submits jobs to EMR on EKS virtual clusters.Ĭreates an EMR JobFlow, reading the config from the EMR connection.Īn operator that modifies an existing EMR cluster.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |