certifications 21 Oct 2026

GCP ACE: BigQuery, Dataflow, Dataprep and Managed Service for Apache Spark

gcpacecertificationbigquerydataflowdataprocmanaged-services

Google Cloud has four services that all touch “data processing,” and the ACE exam leans hard on knowing which one a scenario is actually describing. BigQuery is a warehouse you query with SQL. Dataflow is a managed runner for Apache Beam pipelines. Dataprep by Trifacta is a point-and-click tool for cleaning messy data before it lands anywhere. And Managed Service for Apache Spark — the current name for what most of the industry still calls Dataproc — is where you run Spark, Hadoop, and Hive workloads with as much or as little cluster management as you want. They overlap at the edges, which is exactly where exam questions like to sit.

BigQuery: the serverless warehouse

BigQuery separates storage from compute entirely. You don’t provision anything to query a petabyte-scale table — you write SQL, BigQuery allocates compute slots behind the scenes, and you pay for what the query actually scans (on-demand) or for a block of reserved compute capacity (Editions).

Query limits that show up on the exam

Limit	Value
Maximum SQL query length	1,024 KB
Maximum unresolved GoogleSQL query length	1 MB
Maximum resolved query length (includes referenced views)	12 MB
Maximum resources referenced per query	1,000
Maximum request size	10 MB
Maximum response size	10 GB compressed (unlimited to a destination table)
Maximum row size	100 MB
Maximum partitions per table	4,000
Maximum clustering columns per table	4

Clustering sorts STRING columns using only their first 1,024 characters, and clustering order matters — put the column your queries filter on most often first. A table that would blow past 4,000 partitions at daily granularity is the standard cue to partition by month instead.

Pricing: on-demand vs. Editions

BigQuery retired flat-rate and Flex Slots in 2023 in favor of three Editions: Standard, Enterprise, and Enterprise Plus. All three bill compute in slot-hours, with a 1-minute minimum by default (or no minimum if you opt into BigQuery fluid scaling). On-demand pricing is still available and is what every project uses by default unless you assign it to a reservation: you pay per TiB scanned, and the first 1 TiB per month is free.

	On-demand	Standard	Enterprise	Enterprise Plus
Billing	Per TiB scanned	Slot-hours	Slot-hours	Slot-hours
Capacity limit	2,000 concurrent slots per project, 20,000 per organization	1,600 slots per reservation; up to 10 reservations per administration project and 16,000 slots per organization	Quota-based; up to 200 reservations	Quota-based; up to 200 reservations
Compute model	On-demand	Autoscaling only	Autoscaling + baseline	Autoscaling + baseline
Capacity commitments	—	Not available	1-year (20% discount) or 3-year (40% discount)	1-year (20% discount) or 3-year (40% discount)
BigQuery ML	Yes	No	Yes	Yes
CMEK and VPC Service Controls	Yes	No	Yes	Yes
Assured Workloads compliance controls	No	No	No	Yes
Monthly SLO	≥ 99.99%	≥ 99.9%	≥ 99.99%	≥ 99.99%

Two details people get wrong: the 16,000-slot organization cap belongs to Standard, not Enterprise, and CMEK and VPC Service Controls are already in Enterprise. What Enterprise Plus adds on top is compliance controls through Assured Workloads, not a better SLO.

The practical exam cue: “unpredictable, bursty query load” points to on-demand; “predictable, high, sustained query volume” points to Enterprise or Enterprise Plus with a capacity commitment (Standard can’t take commitments). “Need BigQuery ML” or “need VPC Service Controls” rules Standard out.

Creating a partitioned, clustered table

bq mk --table \
  --time_partitioning_type=DAY \
  --time_partitioning_field=event_date \
  --clustering_fields=country,product_id \
  my_project:analytics.events \
  schema.json

bq query --use_legacy_sql=false \
'SELECT country, COUNT(*) AS total
 FROM `my_project.analytics.events`
 WHERE event_date = "2026-10-20"
 GROUP BY country'

Partitioning on event_date means that second query only scans one day’s worth of data instead of the whole table — this is the main lever for controlling on-demand cost.

Dataflow: managed Apache Beam

Dataflow runs pipelines written against the Apache Beam SDK and unifies batch and streaming under the same programming model — a bounded PCollection is just a special case of an unbounded one. Autoscaling, worker provisioning, and (for streaming jobs) state management via Streaming Engine are all handled for you.

Windowing, triggers, and watermarks

These three concepts are the core of every Dataflow streaming question:

Windowing groups unbounded data into finite chunks by event time — fixed, sliding, session, or a single global window.
Watermarks are Beam’s estimate of “no more data older than this timestamp is coming.” They drive when a window is considered complete.
Triggers decide when to actually emit a window’s results — on watermark (the default), on a processing-time delay, after N elements, or some composite of these. Triggers are also how you handle late data arriving after the watermark has passed.

Templates: Classic vs. Flex

Template type	Parameterized at	Typical use
Classic	Pipeline construction time, limited runtime parameters	Simple, Google-provided templates (Pub/Sub → BigQuery, GCS → BigQuery)
Flex	Fully parameterized at runtime, packaged as a Docker image	Custom pipelines you want non-engineers to launch from the console or a scheduler

gcloud dataflow flex-template run "ingest-orders-$(date +%Y%m%d-%H%M%S)" \
  --template-file-gcs-location=gs://my-bucket/templates/orders-pipeline.json \
  --region=europe-southwest1 \
  --parameters inputSubscription=projects/my-project/subscriptions/orders-sub,outputTable=my-project:analytics.orders \
  --enable-streaming-engine

Streaming Engine offloads state and windowing computation from the worker VMs to the Dataflow service itself, which is what makes workers lighter and autoscaling faster. --enable-streaming-engine plus --additional-experiments is also the flag pair the exam expects when a question mentions streaming high availability — replicating a streaming job’s state across zones at the cost of extra watermark lag.

Dataprep by Trifacta: visual data wrangling

Dataprep is a serverless, point-and-click tool for exploring, cleaning, and structuring data before it’s analyzed — the thing you reach for when a data analyst needs to fix messy CSVs without writing code. Unlike the other three services here, it isn’t built by Google: it’s a partner service run by Alteryx (which acquired Trifacta), sold and billed through Google Cloud under Alteryx’s own terms of service. That’s why its documentation lives on Alteryx’s site rather than on docs.cloud.google.com. It connects natively to Cloud Storage and BigQuery, and under the hood a Dataprep “flow” compiles down to a Dataflow job that actually executes the transformation (recipe) at scale.

The exam-relevant distinction: Dataprep is for interactive, exploratory data cleaning done by a human looking at a sample of the data and building up a recipe of transformation steps. Dataflow is for engineered, repeatable pipelines a developer writes in code. If a scenario describes a business analyst visually wrangling data with no engineering involvement, that’s Dataprep; if it describes a pipeline triggered on a schedule or by an event, that’s Dataflow (possibly running a Dataprep-authored recipe).

Managed Service for Apache Spark (formerly Dataproc)

As of April 2026, Google unified its Spark offering under one brand: Managed Service for Apache Spark, replacing the names “Dataproc on Compute Engine” (cluster mode) and “Google Cloud Serverless for Apache Spark” / “Dataproc Serverless” (serverless mode). It still also runs Hadoop, Hive, Pig, and Presto. The important thing for the exam — and for your terminal — is that none of the tooling changed names: the gcloud dataproc command group, the dataproc.googleapis.com API, and gcloud services enable dataproc.googleapis.com are all exactly as they were. Expect exam language to use “Managed Service for Apache Spark,” but don’t go looking for a gcloud managed-spark command — it doesn’t exist.

Cluster mode vs. serverless mode

	Cluster mode	Serverless mode
Infrastructure	You size and manage a cluster (master + worker VMs)	No cluster to manage — submit a batch workload
Best for	Long-running or interactive workloads, custom cluster config, non-Spark engines (Hadoop, Hive, Presto)	One-off or scheduled Spark batch jobs
Billing	Cluster uptime	Only while the workload executes

Creating a cluster and submitting a job

gcloud services enable dataproc.googleapis.com

gcloud dataproc clusters create analytics-cluster \
  --region=europe-southwest1 \
  --worker-machine-type=n2-standard-4 \
  --num-workers=4 \
  --image-version=2.2-debian12

gcloud dataproc jobs submit spark \
  --cluster=analytics-cluster \
  --region=europe-southwest1 \
  --class=org.apache.spark.examples.SparkPi \
  --jars=file:///usr/lib/spark/examples/jars/spark-examples.jar \
  -- 1000

The standard pattern the exam expects is ephemeral clusters: spin a cluster up, run the job, tear it down — rather than keeping a cluster running 24/7 for occasional jobs. That’s also exactly what serverless mode removes the need to think about:

gcloud dataproc batches submit spark \
  --region=europe-southwest1 \
  --batch=orders-aggregation-$(date +%Y%m%d) \
  --class=com.example.OrdersAggregation \
  --jars=gs://my-bucket/jobs/orders-aggregation.jar \
  --subnet=default

Autoscaling a cluster

Autoscaling policies are defined in YAML and attached to a cluster, not configured with flags directly on clusters create:

# policy.yaml
workerConfig:
  minInstances: 2
  maxInstances: 10
secondaryWorkerConfig:
  maxInstances: 20
basicAlgorithm:
  cooldownPeriod: 4m
  yarnConfig:
    scaleUpFactor: 1.0
    scaleDownFactor: 1.0
    gracefulDecommissionTimeout: 1h

gcloud dataproc autoscaling-policies import orders-policy \
  --region=europe-southwest1 \
  --source=policy.yaml

gcloud dataproc clusters create analytics-cluster \
  --region=europe-southwest1 \
  --autoscaling-policy=orders-policy

gracefulDecommissionTimeout is the setting that matters most in practice: it gives a worker time to finish its current task (and, for YARN, let shuffle data drain) before the autoscaler removes it — set it too low and you lose in-flight work on scale-down.

Managed Service for Apache Spark vs. Dataflow

This is the comparison the exam asks about most directly: you already have Spark or Hadoop code (or a team that knows that ecosystem) → Managed Service for Apache Spark. You’re writing new pipelines from scratch and want a fully serverless, unified batch/streaming model → Dataflow. Lift-and-shift of an on-prem Hadoop cluster is Managed Service for Apache Spark; a greenfield streaming pipeline off Pub/Sub is Dataflow.

Choosing the right service

Scenario	Service
Ad hoc SQL analytics over huge datasets	BigQuery
Custom streaming or batch pipeline, new code, unified model	Dataflow
Analyst visually cleaning messy data before analysis	Dataprep by Trifacta
Existing Spark/Hadoop/Hive jobs, or need non-Beam engines	Managed Service for Apache Spark
One-off Spark job with no cluster to manage	Managed Service for Apache Spark, serverless mode

Exam Takeaways

BigQuery separates storage and compute; query limits worth memorizing: 1,024 KB max query length, 12 MB max resolved length, 1,000 resources per query, 100 MB max row size, 4,000 partitions and 4 clustering columns per table.
BigQuery pricing is either on-demand (per TiB scanned, first 1 TiB per month free, 2,000 concurrent slots per project) or one of three Editions — Standard, Enterprise, Enterprise Plus — billed in slot-hours; Editions replaced flat-rate and Flex Slots.
Standard has no BigQuery ML, no CMEK or VPC Service Controls, no capacity commitments, and a 99.9% SLO. Enterprise adds all of those with a 99.99% SLO; Enterprise Plus adds Assured Workloads compliance controls.
Dataflow unifies batch and streaming through Apache Beam; windowing groups data by event time, watermarks estimate completeness, and triggers decide when results are emitted — this trio is the core of every streaming question.
Flex Templates take runtime parameters and ship as a container image; Classic Templates are simpler but less flexible. Streaming Engine offloads state off the workers and is required for streaming high availability.
Dataprep by Trifacta is for interactive, visual data cleaning by analysts; it compiles recipes down to Dataflow jobs under the hood. It’s a partner service operated by Alteryx, not a Google-built product.
“Managed Service for Apache Spark” is the current brand name for what used to be Dataproc (cluster mode) and Dataproc Serverless (serverless mode) — but the CLI is still gcloud dataproc, and the API is still dataproc.googleapis.com.
Ephemeral clusters (spin up, run the job, tear down) are the standard pattern for cluster mode; serverless mode removes the need for this pattern entirely by billing only for execution time.
Pick Managed Service for Apache Spark when you have existing Spark/Hadoop/Hive code or need non-Beam engines; pick Dataflow for new pipelines built on the unified Beam model.

Sources

↗ LinkedIn