Delete a document in Fusion via API

Goal

A guide to delete documents from Fusion via the Index Pipeline API's.

Environment

Fusion 4 and Fusion 5

Guide

You can delete documents from Fusion using the Index Pipeline API.

For example, to delete the document with id = 1234 from Fusion, use the command below. This example includes a commit command, which is appropriate for a single document. If you are deleting many documents, see Deleting many documents (batch deletion) below before sending a commit with every document.

Note: In the command(s) below, update the id value and all the variables (prefaced with a $):

Fusion 4.x - 5.1.x

curl -u $username:$password -X POST -H \
"Content-Type: application/vnd.lucidworks-document" -d \
'[{ "id": "1234","commands": [{"name": "delete","params": {}}, {"name": "commit","params": {}}]}]' \
http://$fusionHost:$fusionPort/api/apollo/index-pipelines/$indexPipelineId/collections/$collection/index

Fusion 5.2.0+

curl -u $username:$password -X POST -H \
"Content-Type: application/vnd.lucidworks-document" -d \
'[{ "id": "1234","commands": [{"name": "delete","params": {}}, {"name": "commit","params": {}}]}]' \
http://$fusionHost:$fusionPort/api/index-pipelines/$indexPipelineId/collections/$collection/index

Note: In 5.2.0+ the REST API endpoint /api/apollo/.. in earlier versions has been replaced with /api/...

Deleting many documents (batch deletion)

The example above pairs a commit command with the delete command in the same document. This is appropriate when you are deleting a single document, but it is not suitable when you send a batch of documents to delete.

Each commit command in a document triggers a Solr hard commit on the collection. If you repeat the delete + commit structure for every document in a batch, you issue one commit per document. On a large collection, hard commits are expensive, so committing after every document is much more costly than committing once for the whole batch.

When deleting many documents, do not send a commit with every document. Instead, choose one of the following:

• Rely on the collection's automatic commit (recommended). Send only the delete command for each document, with no commit command. Deletes still take effect through the collection's automatic commit even when no explicit commit is sent per document.
• Send a single explicit commit. If you need to force a commit at the end of the batch, include the commit command on only the last document of the batch (or send a separate commit request once after the batch), rather than on every document.

Example: batch delete with a single commit on the last document (Fusion 5.2.0+)

curl -u $username:$password -X POST -H \
"Content-Type: application/vnd.lucidworks-document" -d \
'[{"id": "1234", "commands": [{"name": "delete", "params": {}}]},
  {"id": "1235", "commands": [{"name": "delete", "params": {}}]},
  {"id": "1236", "commands": [{"name": "delete", "params": {}}, {"name": "commit", "params": {}}]}]' \
http://$fusionHost:$fusionPort/api/index-pipelines/$indexPipelineId/collections/$collection/index

To rely on the automatic commit instead, remove the commit command from the last document.

The same applies if you delete documents one at a time at a high rate: sending a commit with every request has the same cost as sending a commit with every document in a batch.

Large requests that include many commits can exceed load balancer timeouts (HTTP 504). The request is usually still processed, so wait before retrying rather than resending the same request immediately.