Dynamically Assigning Documents to Collections

Issue:

How can I dynamically assign documents to collections? 

Environment:

Fusion

Resolution:

Some scenarios warrant the need for Fusion datasource indexing. Similarly, there are some use cases in which a datasource may not be needed for indexing. In the latter case, you might have your documents pushed directly to the Fusion index pipelines via the Index Pipelines REST-API. 

If you choose to use a datasource for indexing, that datasource has an associated index pipeline as well as an associated collection to which the index pipeline outputs. By default, datasources are contained within a collection and cannot be accessed from outside collections. 

If you choose to send your documents directly to an index pipeline outside the current pipeline, the request path would look like this:

/api/apollo/index-pipelines/<id>/collections/<collectionName>/index

The collection to which the pipeline outputs is declared in the API request.  This allows for more control over dynamically assigning documents to certain collections.

 In the event that you'd like to recycle a datasource and use it across multiple collections, you can use the JavaScript Index Stage in the Fusion index pipeline to reassign the collection which the pipeline will output to. 

In the following example, I create a Push Connector datasource and send documents directly to the endpoint opened up by the push connector. The connector then sends the documents to the index pipeline where I've defined a JavaScript index stage that resets the `collection` property. Note that you could use the conditional script to output to a certain collection based on whichever criteria you use.

Here are the instructions for recreating this example:

1. Create a push datasource - choose the pipeline which you will be using to index to all collections
2. in that index pipeline, add a javascript index stage with the following
var new_collection = ctx.setProperty('collection','COLLECTION_NAMEl');
replace COLLECTION_NAME with the collection to which you'd like to index.
3. Save the index pipeline
4. Send documents to the push connector endpoint and verify that they index to the collection that you specified in step 2

Cause: