🚀   Portable does more than just ELT. Explore Our AI Orchestration Capabilities 

Preserving BigQuery Policy Tags with Portable: The Permissions You Actually Need

Sergio
CTO

When you connected BigQuery as a Portable destination, you created a service account with two roles, roles/bigquery.jobUser and roles/bigquery.dataEditor, and handed us a JSON key. That's all a sync needs: it can create tables, load data, and rewrite schemas all day without complaint.

Then someone on your data governance team tags a column with a policy tag. Your next full refresh runs, the load succeeds, and the policy tag is gone. No error, no warning. The column that was protected yesterday isn't protected today.

Portable has a setting for exactly this: Persist column tags on a full refresh sync, under the Policy Tags section of your BigQuery destination's settings. But flipping the switch isn't enough, because preserving policy tags requires two permissions that the standard Portable service account doesn't have. And one of them probably needs to be granted by a different team, in a different project. This post explains what to grant, where, and why.

Why Full Refreshes Eat Policy Tags

This isn't a bug, in BigQuery or in Portable. A full refresh exists to make your table converge to the source's current shape: columns added upstream appear, dropped columns disappear, type changes apply. To do that, Portable's load job replaces the table's schema wholesale with one derived from the source.

BigQuery treats that replacement literally. The load job's schema becomes the table's schema, and a schema without a policyTags block on a column means a column without policy tags. The omission is an instruction, not an oversight.

So when the preserve toggle is on, Portable does the dance for you:

  1. Capture. Before the load, we read the table's current schema and record which columns carry which policy tags.
  2. Reattach. We write the captured tags into the load job's schema, so the tags are reapplied as part of the same atomic load. There's no window where your data exists with the tags stripped.

The capture step needs nothing new, since reading a schema is plain table metadata access and Data Editor already grants it. The reattach step is where permissions get interesting.

One Attach, Two Resources, Two Projects

A policy tag on a column looks like part of the table, but the tag itself belongs to a taxonomy: a Data Catalog resource with its own project, its own location, and its own IAM policy. When anyone (including Portable's service account) attaches a policy tag to a column, Google Cloud checks both sides:

  1. bigquery.tables.setCategory, checked against the table: "are you allowed to change column-level security here?"
  2. datacatalog.taxonomies.get, checked against the taxonomy: "are you allowed to see the tag you're attaching?"

Neither is in the roles Portable's setup instructions ask for. setCategory lives in BigQuery Admin and BigQuery Data Owner, but notably not in Data Editor, even though Data Editor can drop and recreate the entire table. And taxonomies.get isn't a BigQuery permission at all.

The part that trips up almost everyone: the taxonomy grant must be made where the taxonomy lives. In most organizations, taxonomies sit in a central governance project owned by a security or platform team, not in the project that holds your analytics tables. No amount of IAM generosity in your data project will satisfy a check against a resource in another one.

There's an asymmetry that makes this confusing to debug. Reading a schema that contains policy tags requires nothing special, so Portable can see the tag and capture its full resource name. Attaching that exact same tag back is what triggers both checks. You have read access to the reference, not to the thing it references.

What to Grant

In the project that holds your tables (the one from Portable's destination settings):

  • Add bigquery.tables.setCategory to Portable's service account, either through a small custom role or by granting BigQuery Data Owner scoped to the dataset. Keep the existing Data Editor grant: the attach is mechanically a schema update, so bigquery.tables.update is still required, and Data Editor is what supplies it.

In the project that holds your taxonomy (ask whoever manages your policy tags if you're not sure):

  • Grant Data Catalog Viewer (roles/datacatalog.viewer) to Portable's service account, or add an IAM binding for datacatalog.taxonomies.get on the specific taxonomy if your governance team prefers tighter scoping. The Policy Tag Admin role also carries this permission, but it's far more than Portable needs, since it allows managing the taxonomy itself.
  • While you're in that project, confirm the Data Catalog API is enabled there. A correctly granted role on a disabled API still fails; the error message does tell you to enable the API, but it's easy to misread as a permissions problem when you're already debugging permissions.

Just as important, what Portable does not need:

  • Fine-Grained Reader. That role lets a principal read data in protected columns. Portable reattaches tags; it never needs to read what they protect. If a tool asks for Fine-Grained Reader to manage tags, that's an over-grant. Push back, including on us.
  • Policy Tag Admin. Attaching existing tags doesn't require the ability to create, modify, or delete taxonomies.

Verifying the Setup Before a 3 AM Surprise

Portable runs these checks for you: when you enable the preserve toggle, we test the table-side permissions against a real table in your dataset and taxonomy access against every taxonomy already attached to your tables, and a Verify permissions button in the same section re-runs them anytime. The manual route below is for digging deeper, or for checking grants before you've connected the destination at all.

You can check both grants without running a sync, and you can check them as the actual service account. Since you generated its key, you can authenticate as it locally:

gcloud auth activate-service-account --key-file=portable-sa-key.json

This switches your active gcloud account to the service account. When you're done, switch back to your own account, and consider revoking the service account's credentials from your machine while you're at it. Its key shouldn't linger in your local credential store:

gcloud auth list                                  # see your accounts; the active one is starred
gcloud config set account [email protected]   # switch back to yourself
gcloud auth revoke [email protected]

A note on the examples that follow: they use analytics-prod for the project that holds your tables, my_dataset.customers for a tagged table, and data-governance for the project that holds the taxonomy. None of these will exist in your setup. Swap in your own values as you go.

First, find where the taxonomy lives. The tag's full resource name is visible in the table schema, and reading it requires no special access:

# format is project:dataset.table — use one of YOUR tagged tables here
bq show --schema --format=prettyjson "analytics-prod:my_dataset.customers"

Look for the policyTags block on a tagged column:

{
  "name": "email",
  "type": "STRING",
  "policyTags": {
    "names": [
      "projects/data-governance/locations/us/taxonomies/123456789/policyTags/987654321"
    ]
  }
}

That projects/data-governance, not your data project, is where the datacatalog.taxonomies.get grant needs to exist.

Now test both halves with the testIamPermissions API, which any authenticated caller can use to ask "which of these permissions do I have on this resource?" Test against the specific resources, not the project, because a custom role granted at the dataset or table level won't show up in a project-level check:

# BigQuery side: can the service account set policy tags on this table?
# (swap in your own project, dataset, and table in the URL)
curl -s -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  "https://bigquery.googleapis.com/bigquery/v2/projects/analytics-prod/datasets/my_dataset/tables/customers:testIamPermissions" \
  -d '{"permissions": ["bigquery.tables.setCategory", "bigquery.tables.update"]}'

# Data Catalog side: can it see the taxonomy? The URL is built from the
# policyTags resource name you found in the schema above: project,
# location, and taxonomy ID, in that order
curl -s -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  "https://datacatalog.googleapis.com/v1/projects/data-governance/locations/us/taxonomies/123456789:testIamPermissions" \
  -d '{"permissions": ["datacatalog.taxonomies.get"]}'

Each call echoes back the subset of permissions the caller holds. An empty response means none of them. That's your missing grant, identified before any sync runs.

If Tags Still Aren't Sticking

The toggle is on, you've made the grants, and a full refresh still drops or fails to reattach tags. The remaining culprits, roughly in order of likelihood:

  1. The taxonomy grant landed in the wrong project. Re-read the resource name from the schema. The grant goes where the taxonomy lives, which is rarely the project you load into.
  2. Someone granted Fine-Grained Reader instead of Data Catalog Viewer. It sounds related; it does nothing for attaching tags (see above).
  3. Location mismatch. A taxonomy can only govern tables in its own regional location. A taxonomy in us can't tag a table in europe-west1. Google's answer is replicating the taxonomy across locations, but note that replicas keep the same IDs while their IAM bindings do not sync. If tags preserve fine in one region and fail in another, check the replica's bindings, not the original's.
  4. Crossing an organization boundary. Tables and policy tags must belong to the same Google Cloud organization, full stop. There is no grant that fixes a cross-org reference.

When a reattach does fail, the failure is recorded with the run, per table, so if you contact support we can tell you exactly which step couldn't complete and against which resource.

Checklist

  • Policy tag preservation is toggled on under your BigQuery destination's settings in Portable.
  • Portable's service account has bigquery.tables.setCategory (remember, Data Editor does not include it).
  • The service account keeps its Data Editor grant (the schema update itself needs bigquery.tables.update).
  • The service account has datacatalog.taxonomies.get in the project where the taxonomy lives. Find it via bq show --schema.
  • Taxonomy and table are in the same regional location; replicated taxonomies have their own IAM bindings.
  • Table and taxonomy are in the same Google Cloud organization.
  • You verified both grants with testIamPermissions as the service account, instead of inferring them from role names.

One quirk of the underlying APIs worth knowing: there's no way to ask "could this service account attach policy tags, hypothetically?" Permissions can only be tested against a specific taxonomy, whose name you only learn from a tagged table's schema. Portable's permission checks work within that constraint: they test against the taxonomies actually attached to your tables, so until a tagged table exists, the taxonomy check honestly reports that there's nothing to verify yet instead of guessing. The run history remains the authoritative record of what happened to your tags on each sync.

Using Snowflake instead? Tag preservation works there too, with a different permission model (GOVERNANCE_VIEWER to read tags, APPLY TAG to reattach). That one's getting its own post.

Questions, or a permission error that doesn't match anything above? Reach out. We've debugged this more times than we'd like to admit.