Download Google Cloud Dataflow

This page describes how to use the Dataflow connector forSpanner to import, export, and modify data in SpannerGoogleSQL-dialect databases and PostgreSQL-dialect databases.Dataflow is a managed service for transforming and enrichingdata. The Dataflow connector for Spanner lets you readdata from and write data to Spanner in a Dataflowpipeline, optionally transforming or modifying the data. You can also createpipelines that transfer data between Spanner and otherGoogle Cloud products.The Dataflow connector is the recommended method for efficientlymoving data into and out of Spanner in bulk. It's also therecommended method for performing large transformations to a database which arenot supported by Partitioned DML, such as table moves and bulk deletesthat require a JOIN. When working with individual databases, there are othermethods you can use to import and export data:Use the Google Cloud console to export an individual database fromSpanner to Cloud Storage in Avroformat.Use the Google Cloud console to import a database back intoSpanner from files you exported to Cloud Storage.Use the REST API or Google Cloud CLI to run export or importjobs from Spanner to Cloud Storage and back also usingAvro format.The Dataflow connector for Spanner is part of theApache Beam Java SDK, and it provides an API for performing the previousactions. For more information about some of the concepts discussed in this page,such as PCollection objects and transforms, see the Apache Beam programmingguide.Add the connector to your Maven projectTo add the Google Cloud Dataflow connector to a Mavenproject, add the beam-sdks-java-io-google-cloud-platform Maven artifact toyour pom.xml file as a dependency.For example, assuming that your pom.xml file sets beam.version to theappropriate version number, you would add the following dependency: org.apache.beam beam-sdks-java-io-google-cloud-platform ${beam.version}Read data from SpannerTo read from Spanner, apply the SpannerIO.readtransform. Configure the read using the methods in theSpannerIO.Read class. Applying the transform returns aPCollection, where each element in the collectionrepresents an individual row

Us-central1 VERSION: the version of the template that you want to useYou can use the following values: latest to use the latest version of the template, which is available in the non-dated parent folder in the bucket— gs://dataflow-templates-REGION_NAME/latest/ the version name, like 2023-09-12-00_RC00, to use a specific version of the template, which can be found nested in the respective dated parent folder in the bucket— gs://dataflow-templates-REGION_NAME/ GCS_FILE_PATH: the Cloud Storage path that is used to store datastream events. For example: gs://bucket/path/to/data/ CLOUDSPANNER_INSTANCE: your Spanner instance. CLOUDSPANNER_DATABASE: your Spanner database. DLQ: the Cloud Storage path for the error queue directory. API To run the template using the REST API, send an HTTP POST request. For more information on the API and its authorization scopes, see projects.templates.launch. POST "launch_parameter": { "jobName": "JOB_NAME", "containerSpecGcsPath": "gs://dataflow-templates-REGION_NAME/VERSION/flex/Cloud_Datastream_to_Spanner", "parameters": { "inputFilePattern": "GCS_FILE_PATH", "streamName": "STREAM_NAME" "instanceId": "CLOUDSPANNER_INSTANCE" "databaseId": "CLOUDSPANNER_DATABASE" "deadLetterQueueDirectory": "DLQ" } }} Replace the following: PROJECT_ID: the Google Cloud project ID where you want to run the Dataflow job JOB_NAME: a unique job name of your choice LOCATION: the region where you want todeploy your Dataflow job—for example, us-central1 VERSION: the version of the template that you want to useYou can use the following values: latest to use the latest version of the template, which is available in the non-dated parent folder in the bucket— gs://dataflow-templates-REGION_NAME/latest/ the version name, like 2023-09-12-00_RC00, to use a specific version of the template, which can be found nested in the respective dated parent folder in the bucket— gs://dataflow-templates-REGION_NAME/ GCS_FILE_PATH: the Cloud Storage path that is used to store datastream events. For example: gs://bucket/path/to/data/ CLOUDSPANNER_INSTANCE: your Spanner instance. CLOUDSPANNER_DATABASE: your Spanner database. DLQ: the Cloud Storage path for the error queue directory. Template source code Java What's next Learn about Dataflow templates. See the list of Google-provided templates.

To empty.transformationCustomParameters: String containing any custom parameters to be passed to the custom transformation class. Defaults to empty.filteredEventsDirectory: This is the file path to store the events filtered via custom transformation. Default is a directory under the Dataflow job's temp location. The default value is enough under most conditions.shardingContextFilePath: Sharding context file path in cloud storage is used to populate the shard id in spanner database for each source shard.It is of the format Map>.tableOverrides: These are the table name overrides from source to spanner. They are written in thefollowing format: [{SourceTableName1, SpannerTableName1}, {SourceTableName2, SpannerTableName2}]This example shows mapping Singers table to Vocalists and Albums table to Records. For example, [{Singers, Vocalists}, {Albums, Records}]. Defaults to empty.columnOverrides: These are the column name overrides from source to spanner. They are written in thefollowing format: [{SourceTableName1.SourceColumnName1, SourceTableName1.SpannerColumnName1}, {SourceTableName2.SourceColumnName1, SourceTableName2.SpannerColumnName1}]Note that the SourceTableName should remain the same in both the source and spanner pair. To override table names, use tableOverrides.The example shows mapping SingerName to TalentName and AlbumName to RecordName in Singers and Albums table respectively. For example, [{Singers.SingerName, Singers.TalentName}, {Albums.AlbumName, Albums.RecordName}]. Defaults to empty.schemaOverridesFilePath: A file which specifies the table and the column name overrides from source to spanner. Defaults to empty.shadowTableSpannerDatabaseId: Optional separate database for shadow tables. If not specified, shadow tables will be created in the main database. If specified, ensure shadowTableSpannerInstanceId is specified as well. Defaults to empty.shadowTableSpannerInstanceId: Optional separate instance for shadow tables. If not specified, shadow tables will be created in the main instance. If specified, ensure shadowTableSpannerDatabaseId is specified as well. Defaults to empty.Run the template Console Go to the Dataflow Create job from template page. Go to Create job from template In the Job name field, enter a unique job name. Optional: For Regional endpoint, select a value from the drop-down menu. The default region is us-central1. For a list of regions where you can run a Dataflow job, see Dataflow locations. From the Dataflow template drop-down menu, select the Cloud Datastream to Spanner template. In the provided parameter fields, enter your parameter values. Click Run job. gcloud In your shell or terminal, run the template: gcloud dataflow flex-template run JOB_NAME \ --project=PROJECT_ID \ --region=REGION_NAME \ --template-file-gcs-location=gs://dataflow-templates-REGION_NAME/VERSION/flex/Cloud_Datastream_to_Spanner \ --parameters \inputFilePattern=GCS_FILE_PATH,\streamName=STREAM_NAME,\instanceId=CLOUDSPANNER_INSTANCE,\databaseId=CLOUDSPANNER_DATABASE,\deadLetterQueueDirectory=DLQ Replace the following: PROJECT_ID: the Google Cloud project ID where you want to run the Dataflow job JOB_NAME: a unique job name of your choice REGION_NAME: the region where you want todeploy your Dataflow job—for example,

Folder in the bucket— gs://dataflow-templates-REGION_NAME/latest/ the version name, like 2023-09-12-00_RC00, to use a specific version of the template, which can be found nested in the respective dated parent folder in the bucket— gs://dataflow-templates-REGION_NAME/ STAGING_LOCATION: the location for staging local files (for example, gs://your-bucket/staging) INPUT_SUBSCRIPTION_NAME: the Pub/Sub subscription name TOKEN: Splunk's Http Event Collector token URL: the URL path for Splunk's Http Event Collector (for example, DEADLETTER_TOPIC_NAME: the Pub/Sub topic name JAVASCRIPT_FUNCTION: the name of the JavaScript user-defined function (UDF) that you want to useFor example, if your JavaScript function code ismyTransform(inJson) { /*...do stuff...*/ }, then the function name ismyTransform. For sample JavaScript UDFs, seeUDF Examples. PATH_TO_JAVASCRIPT_UDF_FILE: the Cloud Storage URI of the .js file that defines the JavaScript user-definedfunction (UDF) you want to use—for example, gs://my-bucket/my-udfs/my_file.js BATCH_COUNT: the batch size to use for sending multiple events to Splunk PARALLELISM: the number of parallel requests to use for sending events to Splunk DISABLE_VALIDATION: true if you want to disable SSL certificate validation ROOT_CA_CERTIFICATE_PATH: the path to root CA certificate in Cloud Storage (for example, gs://your-bucket/privateCA.crt) API To run the template using the REST API, send an HTTP POST request. For more information on the API and its authorization scopes, see projects.templates.launch.POST "jobName": "JOB_NAME", "environment": { "ipConfiguration": "WORKER_IP_UNSPECIFIED", "additionalExperiments": [] }, "parameters": { "inputSubscription": "projects/PROJECT_ID/subscriptions/INPUT_SUBSCRIPTION_NAME", "token": "TOKEN", "url": "URL", "outputDeadletterTopic": "projects/PROJECT_ID/topics/DEADLETTER_TOPIC_NAME", "javascriptTextTransformGcsPath": "PATH_TO_JAVASCRIPT_UDF_FILE", "javascriptTextTransformFunctionName": "JAVASCRIPT_FUNCTION", "batchCount": "BATCH_COUNT", "parallelism": "PARALLELISM", "disableCertificateValidation": "DISABLE_VALIDATION", "rootCaCertificatePath": "ROOT_CA_CERTIFICATE_PATH" }} Replace the following: PROJECT_ID: the Google Cloud project ID where you want to run the Dataflow job JOB_NAME: a unique job name of your choice LOCATION: the region where you want todeploy your Dataflow job—for example, us-central1 VERSION: the version of the template that you want to useYou can use the following values: latest to use the latest version of the template, which is available in the non-dated parent folder in the bucket— gs://dataflow-templates-REGION_NAME/latest/ the version name, like 2023-09-12-00_RC00, to use a specific version of the template, which can be found nested in the respective dated parent folder in the bucket— gs://dataflow-templates-REGION_NAME/ STAGING_LOCATION: the location for staging local files (for example, gs://your-bucket/staging) INPUT_SUBSCRIPTION_NAME: the Pub/Sub subscription name TOKEN: Splunk's Http Event Collector token URL: the URL path for Splunk's Http Event Collector (for example, DEADLETTER_TOPIC_NAME: the Pub/Sub topic name JAVASCRIPT_FUNCTION: the name of the JavaScript user-defined function (UDF) that you want to useFor example, if your JavaScript function code ismyTransform(inJson) { /*...do stuff...*/ }, then the function name ismyTransform. For sample JavaScript UDFs, seeUDF Examples. PATH_TO_JAVASCRIPT_UDF_FILE: the Cloud Storage URI of the .js file that defines the JavaScript user-definedfunction (UDF) you want to use—for example, gs://my-bucket/my-udfs/my_file.js BATCH_COUNT: the batch size to use for sending multiple events to Splunk PARALLELISM: the number of parallel requests to use for