# tPubSubOutput properties for Apache Spark Streaming (deprecated) Availability-note[Deprecated](https://help.talend.com/r/en-US/Cloud/labels-on-talend-help-center/availability-deprecated) This content is deprecated. Receives messages serialized into byte arrays by its preceding component and issues these messages into a given PubSub service. Information noteDeprecation: Spark Streaming is deprecated beginning with the R2026-09 release. Use Spark Structured Streaming instead. These properties are used to configure tPubSubOutput running in the Spark Streaming Job framework. The Spark Streaming tPubSubOutput component belongs to the Messaging family. If you are using Dataproc 1.4 and onwards as your Spark cluster, make sure to select the Allow API access to all Google Cloud services in the same project check box at the cluster creation on Google Cloud Platform to be able to run PubService. This component is available in Talend Real-Time Big Data Platform and Talend Data Fabric. ## Basic settings | Properties | Description | | --- | --- | | Schema and Edit
schema | A schema is a row description. It defines the number of fields
(columns) to be processed and passed on to the next component. When you create a Spark
Job, avoid the reserved word line when naming the
fields.

Note that the schema of this component is read-only. It stores the
messages to be published. | | Define a Google Cloud configuration component | If you are using Dataproc as your Spark cluster, clear this check box.

Otherwise, select this check box to allow the Pub/Sub component to use the Google Cloud
configuration information provided by a
tGoogleCloudConfiguration component. | | Topic name | Enter the name of the topic you want to publish messages to. This topic must already
exist. | | Topic operation | Select the operation to be performed on the specified topic:

None: select this option if the topic to be used already exists.
Create if not exists: select this option if the topic to be used does not exist.

| ## Advanced settings | Properties | Description | | --- | --- | | Connection pool | In this area, you configure, for each Spark executor, the connection pool used to control
the number of connections that stay open simultaneously. The default values given to the
following connection pool parameters are good enough for most use cases.

Max total number of connections: enter the maximum number of connections (idle or active) that are allowed to stay open simultaneously.

The default number is 8. If you enter -1, you allow unlimited number of open connections at the same time.
Max waiting time (ms): enter the maximum amount of time at the end of which the response to a demand for using a connection should be returned by the connection pool. By default, it is -1, that is to say, infinite.
Min number of idle connections: enter the minimum number of idle connections (connections not used) maintained in the connection pool.
Max number of idle connections: enter the maximum number of idle connections (connections not used) maintained in the connection pool.

| | Evict connections | Select this check box to define criteria to destroy connections in the connection pool. The
following fields are displayed once you have selected it.

Time between two eviction runs: enter the time interval (in milliseconds) at the end of which the component checks the status of the connections and destroys the idle ones.
Min idle time for a connection to be eligible to eviction: enter the time interval (in milliseconds) at the end of which the idle connections are destroyed.
Soft min idle time for a connection to be eligible to eviction: this parameter works the same way as Min idle time for a connection to be eligible to eviction but it keeps the minimum number of idle connections, the number you define in the Min number of idle connections field.

| ## Usage | Usage guidance | Description | | --- | --- | | Usage rule | This component is used as an end component and requires an input link.

This component needs a Write component such as tWriteJSONField to define a *serializedValue* column in the input schema to send serialized data. | | Spark Connection | In the Spark
Configuration tab in the Run
view, define the connection to a given Spark cluster for the whole Job. In
addition, since the Job expects its dependent jar files for execution, you must
specify the directory in the file system to which these jar files are
transferred so that Spark can access these files:

Yarn mode (Yarn client or Yarn cluster):
- When using Google Dataproc, specify a bucket in the Google Storage staging bucket field in the Spark configuration tab.
- When using HDInsight, specify the blob to be used for Job deployment in the Windows Azure Storage configuration area in the Spark configuration tab.
- When using Altus, specify the S3 bucket or the Azure Data Lake Storage for Job deployment in the Spark configuration tab.
- When using on-premises distributions, use the configuration component corresponding to the file system your cluster is using. Typically, this system is HDFS and so use tHDFSConfiguration.
Standalone mode: use the configuration component corresponding to the file system your cluster is using, such as tHDFSConfiguration Apache Spark Batch or tS3Configuration Apache Spark Batch.

If you are using Databricks without any configuration component present in your Job, your business data is written directly in DBFS (Databricks Filesystem).

This connection is effective on a per-Job basis. | | PubSub access permissions | When you use Pub/Sub with a Dataproc cluster, ensure that this cluster
has the appropriate permissions to access the Pub/Sub service.

To do this, you can create the Dataproc cluster by checking
Allow API access to all Google Cloud services in
the same project in the advanced options on Google Cloud Platform, or
via the command line, assigning the scopes explicitly (the following
example is for a low-resource test cluster):

```
gcloud beta dataproc clusters create CLUSTER_ID \
--zone europe-west1-b \
--master-machine-type n1-standard-2 \
--master-boot-disk-size 50 \
--num-workers 2 \
--worker-machine-type n1-standard-2 \
--worker-boot-disk-size 50 \
--scopes 'https://www.googleapis.com/auth/cloud-platform' \
--project PROJECT_ID
``` |