Configuration of ElasticSearch Synchronization

Synchronization of metadata with ElasticSearch is configured by the following files in directory <TalendRuntimePath>\etc:

• org.talend.esb.registry.el.sync.server.cfg: This file controls activation of the ElasticSearch access by the service registry server.

  The el.enable property must be set true after ElasticSearch access has been properly configured in order to really enable it.

• org.talend.esb.registry.el.sync.client.cfg: This file contains access information for ElasticSearch including credentials and SSL keystore information.

  Property name	Description
  el.host	Hostname of ElasticSearch client node.
  el.port	HTTP or HTTPS Port of ElasticSearch client node.
  el.ssl.enable	Enable or disable HTTPS communication.
  el.authentication.enable	Enable or disable authentication settings.
  el.username	Username for BASIC authentication method.
  el.password	Password for BASIC authentication method.
  el.index.name	Elasticsearch index name for synchronization.
  el.ssl.keystore.password	SSL Keystore password.
  el.ssl.keystore.key.password	SSL Keystore Private key password.
  el.ssl.keystore.path	SSL Keystore with private key.
  el.ssl.keystore.type	SSL Keystore type.
  el.ssl.truststore.password	SSL Truststore password.
  el.ssl.truststore.path	SSL Truststore with Elasticsearch server certificate.
  el.ssl.truststore.type	SSL Truststore type.
  el.ssl.enable.cn.check	Enable or disable hostname verification.

• org.talend.esb.registry.server.metadata.xsd: This file is the schema XSD which is used to validate the input metadata XML.

  <?xml version="1.0" encoding="UTF-8"?>
  <schema
      targetNamespace="http://metadata.extension.registry.esb.talend.org/v1"
      elementFormDefault="qualified" xmlns="http://www.w3.org/2001/XMLSchema"
      xmlns:tns="http://metadata.extension.registry.esb.talend.org/v1"
      xmlns:xs="http://www.w3.org/2001/XMLSchema">
   
      <element name="metadata" type="tns:metadata"></element>
   
      <complexType name="metadata">
          <sequence>
          <element name="ServiceType" type="string"></element>
              <element name="ServiceStatus">
                  <simpleType>
                      <restriction base="string">
                          <enumeration value="active"></enumeration>
                          <enumeration value="inactive"></enumeration>
                      </restriction>
                  </simpleType>
              </element>
              <element name="ServiceVersion" type="string"></element>
              <element name="CreationDate">
                  <simpleType>
                      <restriction base="date">
                          <pattern value="[^:Z]*"></pattern>
                      </restriction>
                  </simpleType>
              </element>
              <element name="Description" type="string"></element>
              <element name="Usage">
                  <simpleType>
                      <restriction base="string">
                          <enumeration value="internal"></enumeration>
                          <enumeration value="external"></enumeration>
                      </restriction>
                  </simpleType>
              </element>
              <element name="ServiceOwner" type="string"></element>
              <element name="ServiceDesigner" type="string"></element>
              <element name="ServiceDevelopmentTeam" type="string"></element>
              <element name="Projectname" type="string"></element>
              <element name="ApplicationName" type="string"></element>
              <element name="ServiceMaintenanceOwner" type="string"></element>
              <element name="ServiceMaintenanceContact" type="string"></element>
              <element name="Tags" type="string" maxOccurs="unbounded"
                  minOccurs="0">
              </element>
          </sequence>
      </complexType>
  </schema>

• org.talend.esb.registry.el.sync.template.json: This file contains a JSON template with information for preparing and indexing of the metadata for search in ElasticSearch. This template must be consistent with the schema of the metadata.

  {
    "order" : 1,
    "template" : "talendregistry",
    "settings" : {
      "index.number_of_replicas" : "1",
      "index.number_of_shards" : "1",
      "analysis" : {
        "analyzer" : {
          "servicename" : {
            "type" : "pattern",
            "pattern" : "\\{.*\\}",
            "lowercase" : false
          },
          "namespace" : {
            "type" : "pattern",
            "pattern" : "(.*\\{)|(\\}.*)",
            "lowercase" : false
          }
        }
      }
    },
    "mappings" : {
      "metadata" : {
        "properties" : {
          "metadataExtension" : {
            "properties" : {
              "@schemaLocation" : {
                "type" : "string",
                "index" : "not_analyzed"
              },
              "ServiceName" : {
                "type" : "string",
                "analyzer" : "servicename",
                "fields" : {
                  "raw" : {
                    "index" : "not_analyzed",
                    "type" : "string"
                  },
                  "namespace" : {
                    "index" : "analyzed",
                    "analyzer" : "namespace",
                    "type" : "string"
                  }
                }
              },
              "metadata" : {
                "properties" : {
                  "CustomServiceType" : {
                    "type" : "string",
                    "index" : "not_analyzed"
                  },
                  "CustomCreationDate" : {
                    "type" : "date",
                    "format" : "yyyy-MM-dd"
                  },
                  "CustomDescription" : {
                    "type" : "string",
                    "index" : "analyzed"
                  },
                  "CustomServiceOwner" : {
                    "type" : "string",
                    "index" : "analyzed",
                    "fields" : {
                      "raw" : {
                        "index" : "not_analyzed",
                        "type" : "string"
                      }
                    }
                  },
                }
              }
            }
          }
        }
      }
    }
  }

Most of the template file needs to be taken over as it is. Only the content of the innermost metadata properties that starts with Custom need to be replaced by the properties matching the elements from the corresponding custom schema.

For the present example, matches are as follows:

• CustomServiceType: a simple string which should be indexed for search as it is without cutting it into further details.

• CustomCreationDate: a stringified date where the template contains the format for converting the string back into a date.

• CustomDescription: a string which should be cut into words for full-text search.

• CustomServiceOwner: a string which should be cut into words for full-text search, but at the same time the raw string should be searchable as a whole.