Skip to main content
Sign in

Call a different query pipeline/query profile from Rest Query Stage

  • Updated

Goal

To call a different query pipeline/query profile from the current query pipeline using the Rest query stage, read/get the response and add the results of a REST (RPC) call to the context, request or response.

Environment

Fusion 5.x and Fusion 4.x

Guide

Query Use case:

After executing a REST Query call to another data source you want to add a document from the query to the external data source into the main document list from the actual query response.

Examples: Clients wanting to use the already existing pipelines based on conditions, or for optimizing the performance of query pipelines, etc.

We have also used this for examples like Rest Call delete jobs, Solr updates, etc.

Steps:

  • A query comes in
  • The REST Query query stage takes the query and executes a call to a data source that accepts REST calls (in this case I will make a call to another query pipeline)
  • Execute the Solr Query stage
  • Execute a JavaScript query stage that will read one or more of the documents returned from the REST Query call and insert them into the main document list in the response.

Reference : https://doc.lucidworks.com/managed-fusion/5.9/4rzzxv/rest

In this case:

  • I have 2 Apps
      • FirstApp (default collection - FirstApp)
      • SecondApp (default collection - SecondApp)
  • the query pipeline of FirstApp will call out to SecondApp
      • The REST Query stage in the query pipeline of FirstApp (configured to call out to query profile/pipeline)
        • Endpoint URI: "extservice://query/query-pipelines/SecApp/collections/SecApp/select",
        • Call Method: get
        • Query Parameters
          • q:${request.params.q}
          • fl:id,players_s
        • Results Location: Context
        • Results Key: queryRPC
  • the JavaScript stage in the query pipeline of FirstApp will pull out the documents in the response from the REST Query stage and put them into the document list of the main document response

FirstApp Query pipeline configuration json:

{

"id": "1b3488b6-9279-4b9d-8e85-603320f4e62b",
"mappingRules": [
{
"path": "//response/result/doc/*[@name=\"players_s\"]/text()",
"target": "a_newfield",
"targetLocation": "Context",
"append": false,
"xml": false
}],
"debug": true,
"resultsLocation": "Context",
"resultsKey": "queryRPC",
"params": {
"uri": "extservice://query/query-pipelines/SecApp/collections/SecApp/select",
"method": "get",
"queryParams": {
"q": "${request.params.q}",
"wt": "xml"
},
"headers": {}
},
"useIncomingRequestEntity": true,
"errorHandling": "ignore",
"hasNoSideEffects": true,
"type": "query-rpc",
"skip": false,
"legacy": false,
"secretSourceStageId": "1b3488b6-9279-4b9d-8e85-603320f4e62b"
},
{
"id": "a28d3a3c-f6ff-47ca-ac3d-f91986fa0a46",
"httpMethod": "POST",
"allowFederatedSearch": false,
"preferredReplicaType": "pull",
"type": "solr-query",
"skip": false,
"responseSignalsEnabled": true,
"secretSourceStageId": "a28d3a3c-f6ff-47ca-ac3d-f91986fa0a46"
},
{
"id": "327a4b50-c6f8-4a2d-a8b8-3c864d3ba514",
"script": "function(request,response, ctx) {\n // get the existing docs\n var docs = response.getInnerResponse().getDocuments();\n //var docs = response.initialEntity.getUnderlyingObject()\n logger.info('***** docs: ' + docs);\n \n // get the new docs\n var restQueryResponse = ctx.getProperty(\"queryRPC\"); // XML formatted string\n logger.info('*****restQueryResponse:' + restQueryResponse);\n\n\n\n var extractedValues = ctx.getProperty(\"a_newfield\"); // XML formatted string\n logger.info('*****extractedValues:' + extractedValues);\n\n // get and add the new docs to the list\n getNewDocs(docs, restQueryResponse, extractedValues);\n \n // update the response with all the new docs\n response.getInnerResponse().updateDocuments(docs);\n \n logger.info('\\n\\n***** END *****' + '\\n');\n}\n\n\nfunction getNewDocs(docList, xmlDoc, extractedValues) {\n logger.info('**** xmlDoc: ' + xmlDoc);\n logger.info('**** docList: ' + docList);\n\n \n factory = javax.xml.parsers.DocumentBuilderFactory.newInstance();\n logger.info('**** factory: ' + factory);\n\n try {\n\n logger.info('**** tryblock: ');\n var builder = factory.newDocumentBuilder();\n\n logger.info('**** builder: ' + builder);\n logger.info('**** newxmlDoc: ' + xmlDoc);\n var docs = builder.parse(new java.io.StringBufferInputStream(xmlDoc));\n \n logger.info('**** newdocs: ' + docs);\n\n // Create XPathFactory object\n var xpathFactory = javax.xml.xpath.XPathFactory.newInstance();\n\n // Create XPath object\n var xpath = xpathFactory.newXPath();\n var expr = xpath.compile(\"/response/result/doc\");\n \n // Each node is a single document\n var nodes = expr.evaluate(docs, javax.xml.xpath.XPathConstants.NODESET);\n logger.info('******** nodes: ' + nodes);\n \n info('*** nodes.getLength(): ' + nodes.getLength());\n for (var i = 0; i < nodes.getLength(); i++) { \n var node = nodes.item(i);\n \n expr = xpath.compile(\"./str[@name='id']\");\n var id = expr.evaluate(node, javax.xml.xpath.XPathConstants.STRING); \n logger.info('**** id: ' + id);\n\n expr = xpath.compile(\"./str[@name='players_s']\");\n var players = expr.evaluate(node, javax.xml.xpath.XPathConstants.STRING);\n logger.info('**** players: ' + players);\n \n expr = xpath.compile(\"./str[@name='author_s']\");\n var author = expr.evaluate(node, javax.xml.xpath.XPathConstants.STRING); \n logger.info('**** author: ' + author);\n\n expr = xpath.compile(\"./str[@name='genre_s']\");\n var genre = expr.evaluate(node, javax.xml.xpath.XPathConstants.STRING);\n logger.info('**** genre: ' + genre);\n\n //expr = xpath.compile(\"./str[@name='a_newField']\");\n //var a_newField = expr.evaluate(node, javax.xml.xpath.XPathConstants.STRING);\n //logger.info('**** a_newField: ' + a_newField);\n\n // Create a new doc and add it to the docList\n var docmap = new java.util.HashMap();\n \t docmap.put(\"id\", id);\n \t docmap.put(\"players\", players);\n \t docmap.put(\"author\", author);\n \t docmap.put(\"genre\", genre);\n\n var extractedValues = extractedValues; // XML formatted string\n \n logger.info('**** tryextractedValues: ' + extractedValues);\n\n docmap.put(\"a_newField\", extractedValues);\n \n info('**** docmap: ' + docmap);\n var newDoc = new com.lucidworks.apollo.solr.response.ResponseDocument(docmap, id);\n logger.info('**** newDoc: ' + newDoc);\n\n docList.add(newDoc); \n logger.info('**** docList: ' + docList);\n } \n } catch (e) {\n e.printStackTrace();\n }\n}\n\nfunction info(msg) {\n logger.info('\\n' + msg + '\\n');\n}",
"shareState": true,
"type": "javascript-query",
"skip": false,
"secretSourceStageId": "327a4b50-c6f8-4a2d-a8b8-3c864d3ba514"
}

 

Example : 

 

 

 

 

 

Output can be checked in query_pipeline pod logs :
fl=id, players_s

2024-08-30T15:21:53.195Z - INFO  [qtp1108674897-1089:com.lucidworks.apollo.pipeline.query.stages.RPCQueryStage@231] - RPC response entity: {
"response" : {
"docs" : [ {
 "id" : "55555555",
"players_s" : "Neeraj Chopra"
}, {
"id" : "44444444",
"players_s" : "Vinesh Phogat"
}, {
"id" : "66666666",
"players_s" : "Lakshya Sen"
} ],
"numFound" : 3,
"start" : 0,
"numFoundExact" : true
},
}

No fl (q=*:*)

2024-08-30T14:52:45.208Z - INFO  [qtp1108674897-1250:com.lucidworks.apollo.pipeline.query.stages.RPCQueryStage@231] - RPC response entity: {
"response" : {
"docs" : [ {
"name_s" : "a Thief",
"genre_s" : "fantasy",
"_lw_data_source_collection_s" : "SecApp",
"players_t" : "Neeraj Chopra",
"players_s" : "Neeraj Chopra",
"series_t" : "Percy Jackson and the Olympians",
"mrp_t" : "13.9",
"author_s" : "Rick Riordan",
"cat_ss" : [ "book", "hardcover" ],
"price_d" : 13.9,
"cat_txt" : [ "book", "hardcover" ],
"author_t" : "Rick Riordan",
"name_t" : "a Thief",
"id" : "55555555",
 "_version_" : 1808816885787000832
}, {
 "name_s" : "dhe Lightning Thief",
"genre_s" : "fantasy",
"_lw_data_source_collection_s" : "SecApp",
"players_t" : "Vinesh Phogat",
"players_s" : "Vinesh Phogat",
"series_t" : "Percy Jackson and the Olympians",
"mrp_t" : "9.8",
"author_s" : "Rick Riordan",
"cat_ss" : [ "book", "hardcover" ],
"price_d" : 9.8,
"cat_txt" : [ "book", "hardcover" ],
"author_t" : "Rick Riordan",
"name_t" : "dhe Lightning Thief",
"id" : "44444444",
"_version_" : 1808816885788049408
}, {
"name_s" : "Lucene in Action, Second Edition",
"genre_s" : "IT",
"players_t" : "Lakshya Sen",
"players_s" : "Lakshya Sen",
"mrp_t" : "30.50",
"author_s" : "Michael McCandless",
"cat_ss" : [ "book", "paperback" ],
"price_d" : 30.5,
"cat_txt" : [ "book", "paperback" ],
"name_t" : "Lucene in Action, Second Edition",
 "id" : "66666666",
"_version_" : 1808816885795389440
} ],
 "numFound" : 3,
"start" : 0,
"numFoundExact" : true
},

 

If required Xpath can also be given as below to fetch a particular field and values as below: 

XPath: //response/result/doc/*[@name="players_s"]/text() 

 

2024-08-31T14:30:47.652Z - INFO [qtp1108674897-2175:com.lucidworks.apollo.pipeline.query.stages.RPCQueryStage@365] - Rule MappingRule[path=//response/result/doc/*[@name="players_s"]/text(), target=a_newfield, targetLocation=Context, append=false, xml=false], extracted values: {a_newfield=[Neeraj Chopra, Vinesh Phogat, Lakshya Sen]}


Some of the differences between Fusion 4 and Fusion 5 Rest Query Stage configuration: 

Fusion 4 Fusion 5
service /extservice
/api /query
api.log query-pipeline pod log



 

 

 

 

Related articles

Comments

0 comments

Article is closed for comments.

Powered by Zendesk