NAME

DESCRIPTION

USAGE

oci data-flow run submit [OPTIONS]

REQUIRED PARAMETERS

The OCID of a compartment.

The input used for spark-submit command. For more details see https://spark.apache.org/docs/latest/submitting-applications.html#launching-applications-with-spark-submit. Supported options include --class, --file, --jars, --conf, --py-files, and main application file with arguments. Example: --jars oci://path/to/a.jar,oci://path/to/b.jar --files oci://path/to/a.json,oci://path/to/b.csv --py-files oci://path/to/a.py,oci://path/to/b.py --conf spark.sql.crossJoin.enabled=true --class org.apache.spark.examples.SparkPi oci://path/to/main.jar 10 Note: If execute is specified together with applicationId, className, configuration, fileUri, language, arguments, parameters during application create/update, or run create/submit, Data Flow service will use derived information from execute input only.

OPTIONAL PARAMETERS

An Oracle Cloud Infrastructure URI of an archive.zip file containing custom dependencies that may be used to support the execution a Python, Java, or Scala application. See https://docs.cloud.oracle.com/iaas/Content/API/SDKDocs/hdfsconnector.htm#uriformat.

The Spark configuration passed to the running process. See https://spark.apache.org/docs/latest/configuration.html#available-properties. Example: { “spark.app.name” : “My App Name”, “spark.shuffle.io.maxRetries” : “4” } Note: Not all Spark properties are permitted to be set. Attempting to set a property that is not allowed to be overwritten will cause a 400 status to be returned. This is a complex type whose value must be valid JSON. The value can be provided as a string on the command line or passed in as a file using the file://path/to/file syntax.

The --generate-param-json-input option can be used to generate an example of the JSON which must be provided. We recommend storing this example in a file, modifying it as needed and then passing it back in via the file:// syntax.

Defined tags for this resource. Each key is predefined and scoped to a namespace. For more information, see Resource Tags <https://docs.cloud.oracle.com/iaas/Content/General/Concepts/resourcetags.htm>. Example: {“Operations”: {“CostCenter”: “42”}} This is a complex type whose value must be valid JSON. The value can be provided as a string on the command line or passed in as a file using the file://path/to/file syntax.

The --generate-param-json-input option can be used to generate an example of the JSON which must be provided. We recommend storing this example in a file, modifying it as needed and then passing it back in via the file:// syntax.

A user-friendly name that does not have to be unique. Avoid entering confidential information. If this value is not specified, it will be derived from the associated application’s displayName or set by API using fileUri’s application file name.

The VM shape for the driver. Sets the driver cores and memory.

The VM shape for the executors. Sets the executor cores and memory.

Free-form tags for this resource. Each tag is a simple key-value pair with no predefined name, type, or namespace. For more information, see Resource Tags <https://docs.cloud.oracle.com/iaas/Content/General/Concepts/resourcetags.htm>. Example: {“Department”: “Finance”} This is a complex type whose value must be valid JSON. The value can be provided as a string on the command line or passed in as a file using the file://path/to/file syntax.

The --generate-param-json-input option can be used to generate an example of the JSON which must be provided. We recommend storing this example in a file, modifying it as needed and then passing it back in via the file:// syntax.

Provide input to this command as a JSON document from a file using the file://path-to/file syntax.

The --generate-full-command-json-input option can be used to generate a sample json file to be used with this command option. The key names are pre-populated and match the command option names (converted to camelCase format, e.g. compartment-id –> compartmentId), while the values of the keys need to be populated by the user before using the sample file as an input to this command. For any command option that accepts multiple values, the value of the key can be a JSON array.

Options can still be provided on the command line. If an option exists in both the JSON document and the command line then the command line specified value will be used.

For examples on usage of this option, please see our “using CLI with advanced JSON options” link: https://docs.cloud.oracle.com/iaas/Content/API/SDKDocs/cliusing.htm#AdvancedJSONOptions

An Oracle Cloud Infrastructure URI of the bucket where the Spark job logs are to be uploaded. See https://docs.cloud.oracle.com/iaas/Content/API/SDKDocs/hdfsconnector.htm#uriformat.

The maximum time to wait for the resource to reach the lifecycle state defined by --wait-for-state. Defaults to 1200 seconds.

The OCID of OCI Hive Metastore.

The number of executor VMs requested.

The Spark version utilized to run the application. This value may be set if applicationId is not since the Spark version will be taken from the associated application.

The Spark application processing type.

Accepted values are:

BATCH, STREAMING

This operation creates, modifies or deletes a resource that has a defined lifecycle state. Specify this option to perform the action and then wait until the resource reaches a given lifecycle state. Multiple states can be specified, returning on the first state. For example, --wait-for-state SUCCEEDED --wait-for-state FAILED would return on whichever lifecycle state is reached first. If timeout is reached, a return code of 2 is returned. For any other error, a return code of 1 is returned.

Accepted values are:

ACCEPTED, CANCELED, CANCELING, FAILED, IN_PROGRESS, STOPPED, STOPPING, SUCCEEDED

Check every --wait-interval-seconds to see whether the resource to see if it has reached the lifecycle state defined by --wait-for-state. Defaults to 30 seconds.

An Oracle Cloud Infrastructure URI of the bucket to be used as default warehouse directory for BATCH SQL runs. See https://docs.cloud.oracle.com/iaas/Content/API/SDKDocs/hdfsconnector.htm#uriformat.

GLOBAL PARAMETERS

--auth-purpose, --auth, --cert-bundle, --cli-auto-prompt, --cli-rc-file, --config-file, --debug, --defaults-file, --endpoint, --generate-full-command-json-input, --generate-param-json-input, --help, --latest-version, --max-retries, --no-retry, --opc-client-request-id, --opc-request-id, --output, --profile, --query, --raw-output, --region, --release-info, --request-id, --version, -?, -d, -h, -i, -v

EXAMPLES

Please note this sample will only work in the POSIX-compliant bash-like shell. You need to set up the OCI configuration <https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/cliinstall.htm#configfile> and appropriate security policies <https://docs.oracle.com/en-us/iaas/Content/Identity/Concepts/policygetstarted.htm> before trying the examples.

    export compartment_id=<substitute-value-of-compartment_id> # https://docs.cloud.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/data-flow/run/submit.html#cmdoption-compartment-id
    export execute=<substitute-value-of-execute> # https://docs.cloud.oracle.com/en-us/iaas/tools/oci-cli/latest/oci_cli_docs/cmdref/data-flow/run/submit.html#cmdoption-execute
    oci data-flow run submit --compartment-id $compartment_id --execute $execute

AUTHOR

COPYRIGHT