Fetching run logs periodically for analysis
Use the /monitoring/executions/{runId}/logs endpoint and the /monitoring/executions/{runId}/logs/status endpoint to fetch run logs of a given period.
- Use the POST method at
/monitoring/executions/{runId}/logs to generate the full log of a
given period of a run as well as the token to access this log. This token is valid for
one hour after being generated.
To ensure the completeness of your log, check that the run for which you need the log has completed and then wait for 30 seconds before sending the API call.
- Use the POST method at /monitoring/executions/{runId}/logs/status to check whether this log is ready for download and generate a download URL accordingly.
Before you begin
-
Generate access tokens:
- For users, generate a personal access token by following Generating a Personal Access Token.
- For service accounts, generate a service access token by following Generating a service account token.
Once generated, a service account token expires after 30 minutes. If it expires, generate a new token using the POST method at the endpoint https://api.<env>.cloud.talend.com/security/oauth/token. For more information about generating a token, see Generating a service account token.
- The user or the service account to be used must have the Author, the Viewer and the Executor permissions for the workspace to which the task to be monitored belongs.
- You must know the ID of the task run you need to monitor.
This ID is available on the Task execution log page, reading as Task execution ID. Or you can obtain this ID using a GET call from the /processing/executables/tasks/{taskId}/executions endpoint.
About this task
In this example, Talend API Tester is used to issue API requests. For further information about Talend API Tester, see Talend Cloud API Tester User Guide.
Procedure
Results
You can then integrate these API requests in your own monitoring system to periodically generate and download the full log of a given time range for analytic purpose.
https://api.<env>.cloud.talend.com/processing/executables/tasks/{taskId}/executions
Then pass each of the IDs dynamically to the /monitoring/executions/{runId}/logs and the /monitoring/executions/{runId}/logs/status endpoints to prepare the logs to be used.
After all, you can develop a program, such as a Talend Job, to automate the whole process in your monitoring system.