Submit a Managed Fusion promotion request for a Production environment

Issue

How do I submit a Managed Fusion promotion request from Staging to Production?

Environment

Managed Fusion Production environments, post go-live.

Resolution

Pre-promotion steps

Prior to submitting the promotion request, the following steps should be followed:

• Implement and test changes in your Development environment
• Promote the tested changes to your Staging environment
• Confirm the tested changes in your Staging environment are stable and complete, and ready for promotion to Production

Promotion request steps

Submit a ticket to Support by following these steps:

Note: Promotion requests are typically fulfilled within 3 business days, assuming all of the required information (see below) has been included in the ticket description.

Note: Everything in Staging will be promoted to Production at the time of request unless specifically indicated as a partial promotion in the ticket description.

• Once the pre-promotion steps have been completed, visit the Support Portal and click Sign in in the upper-right corner
• Once signed in, click Submit a request:
• Select Managed Services Promotion Request and fill out the subject and description fields per the template below:                                          

•  
  • Ticket Template:
    • Ticket Subject: Promotion Request - [High level summary]

      • Example: Promotion Request - Add new datasource

         

  • Ticket Description:
    • Description:
      1. Nature of the change completed in lower environments:
      2. If multiple apps, which app(s) does the change apply to?:
      3. What part of the system is impacted?: (i.e. specific datasources/index vs query pipeline, files, rules, or profiles)
      4. Starting with Fusion version 5.9.8, Lucidworks introduced the Configuration Promotion Console to facilitate promotions. Using this tool, users can create Pull Requests to move objects between instances. For promotions to a Production environment, Managed Services will review, approve, and merge the pull request.
    • Testing: 
      1. Has this been tested in all lower environments?
      2. How was the new feature tested and validated? (e.g. set of queries this change was tested with, or set of fields that populated as expected)
      3. Is the Staging/Pre-prod environment in sync with Production for the area of the system being changed?
    • Validation:
      1. Is this a partial promotion? (e.g. making a change just to a stage rather than entire pipeline)
      2. How should Lucidworks Managed Services validate the change after promotion? (e.g. set of queries this change was tested with, or set of fields that populated as expected. Note that this is not a full QE exercise, just a quick way to validate the change. The expectation is that the change was fully tested in lower environments and that the implementation team is responsible for full validation)
    • Performance:
      1. Does this change impact performance? Was any load testing done? (e.g. substantial increase in the index with adding a new collection or introducing wildcards)

Promotion fulfillment steps

• Managed Services promotes outlined changes from Staging to Production.
   
  • The standard SLA for promotion requests is 3 business days. Signature Support clients may have a different SLA and can check in with their Client Success Manager regarding this SLA.
• Managed Services notifies client that changes are live in Production.
• Client confirms and Managed Services closes ticket.

Special considerations

• A request to clear a search facing collection , causing a search outage may require scheduling the delete for off-hours on weekdays,  depending on how long it would take to repopulate the collection. This is to be avoided unless absolutely necessary.  In many cases, a clear crawlDB might accomplish the desired outcome without the risk of an outage.
• If requesting a change to a datasource that will cause changes to an existing collection (ie updated start link, sql query, etc) please specify if that change will cause an increase in the number of processed documents which may impact current infrastructure.
• If there is a special request to do a promotion request during “off-hours”, please specify what the window of time is considered to be “off-hours” and the team will do their best to accommodate. Promotions are not supported during weekend hours.
• If the environments (Staging and Production) are out of sync, Managed Services will typically need confirmation for the promotion. This promotion may be delayed so that Managed Services can verify that the proper components are being promoted.
• If the request is a large promotion with multiple changes to multiple pipelines, datasources, and collections, or part of a new feature implementation, Managed Services will reach out and alert that the promotion may be delayed because it may require more time to review the changes before merging.
• If the request includes a schema field type change, this needs to be specifically called out. See special notes section below.
• If Managed services identifies code that may cause risk to the environment, we may choose to delay the promotion to discuss such risk.
• If Managed Services finds components that are not identified in the promotion request, Managed Services will need confirmation for how to proceed with the promotion. The promotion may be delayed in order to work through the concerns.
• Certain requests may not fit into the criteria for promotions and may require a formal handoff to Support and Managed Services:
  • Full App promotions - These may impact performance of the current environment and need to be reviewed by Cloud Operations.
  • Promotion of multiple collections and/or pipelines - Better described as a partial app promotion.
  • Large code promotions - Possibly part of a Phase 2 implementation or include multiple pipelines that considerably alter the current functionality.
• On rare occasions, the promotion may cause issues in the environment that will need troubleshooting and time to resolve.

Special implementation notes

Schema updates

In general, with any schema changes, the safe recommendation is to clear and reindex the complete collection but here are some guidelines for changing existing field definitions in production as this can lead to schema incompatibility in cases where the index is not properly cleared. 

• Adding or deleting a field

1. This is fairly simple, and MS can promote the change after confirming that the client has tested it in lower environments. Note that it is recommended that clients reindex the data fully since old documents will not have the new field, which may cause unexpected results.

2. Addition of documents with the new field without full reindexing typically should not produce any Solr error.

• Changing a field or fieldtype 

Recommended approaches to changing a field would be:

Approach 1: This can be adopted by clients who are using collections in their applications without aliases

1. Add a new field in schema with required changes and retire the old field after verification. that the new field is indexing the field value as expected.

2. Reindex all documents for this datasource.

Approach 2: This can be adopted by clients using collection aliasing in their applications

1. Create a new collection with the updated schema.
2. Fully Index all data to the new collection.
3. Point collection alias to new collection once it has been verified that the new collection is populated with the expected data.

Approach 3: If the two previous approaches are not available 

This approach also requires to fully reindex the data to ensure no traces of fields with old definition exist in the index. To ensure proper reindexing of documents, we recommend the following steps:

1. Clear the collection by deleting all documents.
2. Perform a hard commit. 
3. Apply updated schema with schema changes and reload schema.
4. Index few sample documents and ensure that the new fields are properly indexed in Solr.