	In Browser
	StumbleUpon
	del.icio.us
	Google
	Google Buzz
	reddit
	LinkedIn

	Facebook
	Twitter
	Linkedin
	E-Mail

Cloud Platforms > Google Cloud Platform (GCP) > Cloud Workflows

Cloud Workflows

Author: Venkata Sudhakar

Google Cloud Workflows is a fully managed, serverless orchestration platform that allows you to create, execute, and manage workflows that connect and automate Google Cloud and HTTP-based API services. It provides reliable, scalable execution without managing any infrastructure.

Key Features:

1. Serverless - No infrastructure to provision or manage. Pay only per step executed.

2. Built-in connectors - Native connectors for 200+ Google Cloud services (BigQuery, GCS, Pub/Sub, Cloud Functions).

3. Error handling - Built-in retry policies, exception catching, and fallback steps.

4. Parallel execution - Run multiple workflow branches concurrently for faster processing.

5. Long-running - Workflows can run for up to one year with built-in wait/callback mechanisms.

The below example shows a Cloud Workflow that orchestrates a data pipeline across multiple GCP services.

main:
  steps:
    - init:
        assign:
          - project: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
          - dataset: "sales_data"
          - bucket: "my-data-bucket"

- check_file_exists:
        call: http.get
        args:
          url: ${"https://storage.googleapis.com/" + bucket + "/daily-sales.csv"}
          auth:
            type: OAuth2
        result: file_check

- validate_file:
        switch:
          - condition: ${file_check.code == 200}
            next: run_dataflow_job
          - condition: true
            next: notify_failure

- run_dataflow_job:
        call: http.post
        args:
          url: ${"https://dataflow.googleapis.com/v1b3/projects/" + project + "/templates:launch"}
          auth:
            type: OAuth2
          body:
            jobName: "daily-sales-etl"
            gcsPath: "gs://my-templates/sales-etl-template"
            parameters:
              inputFile: ${"gs://" + bucket + "/daily-sales.csv"}
              outputTable: ${project + ":" + dataset + ".daily_summary"}
        result: job_response

- wait_for_job:
        call: sys.sleep
        args:
          seconds: 300

- notify_success:
        call: http.post
        args:
          url: "https://hooks.slack.com/services/YOUR/WEBHOOK"
          body:
            text: "Daily sales ETL completed successfully!"
        next: return_result

- notify_failure:
        call: http.post
        args:
          url: "https://hooks.slack.com/services/YOUR/WEBHOOK"
          body:
            text: "ERROR: Daily sales file not found!"

- return_result:
        return: ${job_response.body.job.id}

It gives the following output when executed,

Workflow execution started: executions/abc123
Step: init - SUCCEEDED
Step: check_file_exists - SUCCEEDED (HTTP 200)
Step: validate_file - SUCCEEDED
Step: run_dataflow_job - SUCCEEDED
Step: wait_for_job - SUCCEEDED
Step: notify_success - SUCCEEDED
Return value: "2024-01-15_daily-sales-etl_job_456"
Execution status: SUCCEEDED

Cloud Workflows vs Cloud Composer:

Cloud Workflows - Best for lightweight serverless orchestration of GCP services and HTTP APIs. No cluster to manage, very low cost, instant startup. Best for event-driven pipelines.

Cloud Composer (Apache Airflow) - Best for complex data engineering pipelines with rich scheduling, backfill, and dependency management. Better for large-scale DAG-based data workflows.

Send your comments, suggestions or queries regarding this site to [email protected].