tPubSubInput properties for Apache Spark Structured Streaming | Talend Components for Jobs Help
Skip to main content Skip to complementary content
Qlik Resources
Loading SearchUnify's search If you need assistance with your product, please contact Qlik Support.
Qlik Customer Portal
Loading SearchUnify's search If you need assistance with your product, please contact Qlik Support.
Qlik Customer Portal

tPubSubInput properties for Apache Spark Structured Streaming

Last updated: 9/30/2026
Feature in preview

These properties are used to configure tPubSubInput running in the Spark Structured Streaming Job framework.

The Spark Structured Streaming tPubSubInput component belongs to the Messaging family.

The streaming version of this component is available in Talend Real-Time Big Data Platform and in Talend Data Fabric.

Basic settings

Properties Description
Schema and Edit schema

A schema is a row description. It defines the fields (columns) processed by the component. When you create a Spark Job, avoid the reserved word line when naming fields.

  • Built-In: create and store the schema locally for this component only.

  • Repository: reuse a schema created and stored in the Repository across projects and Job designs.

Click Edit schema to modify the schema. If you modify a Repository schema, the available options include:

  • View schema: view the schema without modifying it.

  • Change to built-in property: change the schema to Built-in for local changes.

  • Update repository connection: modify the Repository schema and choose whether to propagate the changes to other Jobs.
Google Cloud configuration Select the tGoogleCloudConfiguration component that provides the Google Cloud connection settings for this component.
Message format Select the format of the Pub/Sub messages from the drop-down list:
  • String: decodes the binary message payload as a UTF-8 string.
  • byte[]: passes the raw binary payload without decoding.
  • Avro: deserializes the binary payload using the Avro schema defined in the Schema tab.
Pub/Sub topic Enter the name of the Pub/Sub topic to consume messages from.
Pub/Sub subscription Enter the name of the Pub/Sub subscription from which to consume messages. The subscription must be connected to the specified topic.
Enable watermarking Select this check box to activate watermarking and select the time mode to be used:
  • Event time: Select to use a timestamp column representing when each event occurred, then select the column in the Watermark column (timestamp) drop-down list.
  • Processing time: Select to use the time when Spark reads each record.

In the Watermark delay field, specify how long Spark waits for late data before finalizing each window.

Advanced settings

Properties Description
Max messages Enter the maximum number of Pub/Sub messages fetched per micro-batch.
Acknowledge deadline (seconds) Enter the acknowledgment deadline in seconds. Messages that are not acknowledged within this window are redelivered by Pub/Sub.
Pub/Sub options Add custom Pub/Sub connector options as key-value pairs by clicking the plus button.

Usage

Usage guidance Description
Usage rule

This component is used as a start component and requires an output link.

This component, along with the Spark Structured Streaming component Palette it belongs to, appears only when you are creating a Spark Structured Streaming Job.

Spark Connection

You need to use the Structured Streaming Configuration tab in the Run view to define the connection to a Spark cluster for the whole Job.

This connection is effective on a per-Job basis.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here