Indexing phone reviews with embeddings and vector database

• In the Directory field, enter or select: "<folder_path>/phone-reviews/"
• In the Files field, add a line and enter: "*.txt" to list all text files in the directory (LG.txt and Iphones.txt).

• In the Filename field, enter: ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH")) to read each phone review file from the list.

• chunk (String)

• Click Sync columns to retrieve the column schema from the previous component.
• In the Start code field, enter the following code:

  //String regex = "(?:\\d(?:\\.\\d)?[A-Z]{5})\\b";
  //Pattern pattern = Pattern.compile(regex);
  int nbParts=2;

• In the Main code field, enter the following code to convert byte content to string and split into smaller chunks:

   //System.out.println(part);
  String input = row7.content.toString();
  String[] lines = input.split("\\r?\\n");
  int nbLines=lines.length;
  int linesPerPart = nbLines / nbParts;
  int remainder = nbLines % nbParts;
  int linesProcessed = 0;
  //System.out.println("nb lines:"+nbLines); 
   for (int i = 0; i < nbParts; i++) {
              StringBuilder partBuilder = new StringBuilder();
              int linesInThisPart = linesPerPart + (remainder > 0 ? 1 : 0);
              remainder--;
              for (int j = 0; j < linesInThisPart; j++) {
              partBuilder.append(lines[linesProcessed]).append(" ");
              linesProcessed++;
              }
                             
              String body=partBuilder.toString();
              row8.chunk=body;
              //System.out.println("text:"+body);
              globalMap.put("id",((String)globalMap.get("tFileList_1_CURRENT_FILE"))+i);
              globalMap.put("chunk",body);
                          

  Information noteNote: This code splits the phone review text into smaller chunks to facilitate better analysis and more precise semantic search in the vector database.
• In the End code field, enter the following code:

  
  }

This component displays the chunked phone review text in the console as a table.

• Click Edit schema and verify the schema has the following column: embedding (List).
• In the Platform list, select Azure OpenAI.
• In the Model name field, click the [...] button and select text-embedding-3-small.
• In the Token/API Key field, click the [...] button and enter your Azure OpenAI API key, then click OK.
• In the Azure endpoint field, enter your Azure OpenAI endpoint (for example: https://your-resource-name.openai.azure.com/).
• In the Column for embedding list, select chunk.

This component generates vector embeddings for each phone review text chunk.

This component displays the generated embeddings in the console.

• id (String)
• vector (List) - Map it with the embedding input column
• text (String)

This mapping verifies that all metadata are correctly transferred to tPineconeClient. The id and values columns are required by Pinecone for upsert operations.

• Click Edit schema and verify the schema has the following column: upsertedCount (Int).
• In the API Key field, click the [...] button and enter your Pinecone API key, then click OK.
• In the Host field, enter your Pinecone index host (for example: "your-index-name.svc.environment.pinecone.io").
• In the Operation list, select Upsert to load the vectorized phone review data into the Pinecone index.
• In the Namespace field, enter the namespace name (for example: phones") or leave empty to use the default namespace.

  

This component displays the upserted records in the console, confirming successful loading into Pinecone.