FUSION JDBC Stage: Fetch records from DB for given Prepared SQL Statement and index to Solr

Fusion provides a general and flexible framework for indexing data - all of your data, any of your data; provisioning index pipelines frameworks and connectors framework.

JDBC Index Stage here pulls few of the needed records from DB and puts into pipeline context to further process or directly index to solr.

JDBC{Index|Query}Stage adds support for calling out to a database as part of a pipeline stage and injects the results into either the context/request or the pipeline document. For instance, if you wished to look up a user from a DB and add their profile information onto the request for downstream use in a pipeline, the JDBCQueryStage would facilitate that.

Use case
To fetch few of the records from database based on the given prepared statement and index them to solr.

Configuration steps
1. Go to Fusion admin page and click on pipelines [http://localhost:8764/admin/index-pipelines]
2. Add a pipeline(index-pipeline)
3. Add a JDBC Look up Stage, selecting it from the listed index stages.Add the sql statement and prepared statement keys [using Add+ button]. I have attached image of my config for reference.

4. Add a field mapping stage, if you need any field mappings [optional, based on requirement]

5. Add a Solr index stage.

Please place the jdbc jar in the following locations: 

//lucidworks/fusion/apps/jetty/api/webapps/api/WEB-INF/lib

//lucidworks/fusion/apps/jetty/connectors/webapps/connectors/WEB-INF/lib

 And restart the api and connectors component  - ./api restart  ./connectors restart

6. The API to fire a request from your client end which will duly pull the record from database for the ID you pass in the request and index to solr is as below.

curl -u user:pass -X POST -H "Content-Type: application/vnd.lucidworks-document" -d '[{"id": "myDoc1","fields": [{"name":"PersonID", "value": "5"}]}]' http://localhost:8764/api/apollo/index-pipelines/<pipeline_name/collections/<collection_name>/index

If your SQL statement is select * from Persons where PersonID=? , in the request api, you should have field name(PersonID) and value passed to it. This will pull the record from DB and indexes it.

 

Reference links

http://doc.lucidworks.com/fusion/2.0.0/Fusion_Documentation/Pipeline_Stages_Reference/JDBC-Query-Stage.html#JDBCQueryStage-Overview

https://docs.lucidworks.com/display/fusion/Index+Pipelines+API#IndexPipelinesAPI-SubmitaSetofDocumentstoanIndexPipeline

 

 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk