Summary
Provide a supported, administrator-controlled maintenance mode that prevents IBM Voice Gateway from advertising availability to an external Session Border Controller (SBC), while allowing its deployment to run and satisfy the requirements for upgrade completion.
This setting must remain effective throughout IBM Software Hub/Cloud Pak for Data upgrades, including operator reconciliation, deployment scaling, and pod restarts.
Environment and current maintenance procedure
The client operates two production IBM Voice Gateway instances in separate geographic regions. Before upgrading IBM Software Hub/Cloud Pak for Data in either region, traffic is diverted to the other region:
watsonx Assistant digital traffic: The backends associated with the cluster being upgraded are removed from the Google Cloud external global Application Load Balancer.
Voice traffic: The Voice Gateway deployment in the maintenance region is scaled to zero replicas. Its SBC health checks subsequently fail, causing the SBC to route calls to the other region.
Problem
During the upgrade process, the Voice Gateway deployment must be scaled back to one replica so that the operator can complete upgrade reconciliation and report the custom resource as successfully upgraded.
When Voice Gateway starts, the SBC detects it as available and resumes forwarding calls to the region undergoing maintenance. This defeats the intended traffic diversion and exposes production calls to an environment whose dependent services may still be upgrading or unavailable.
The current procedure therefore creates a conflict between completing the product upgrade and keeping the maintenance region excluded from production voice routing.
Requested enhancement
Introduce an explicit maintenance-mode setting, exposed through a supported custom resource field, configuration flag, or environment variable.
When enabled, this setting should:
- Prevent the gateway from advertising availability to the SBC. Suppress responses to inbound SIP OPTIONS health checks, or return a configurable non-success response that the SBC recognizes as unavailable.
- Allow upgrade completion. Voice Gateway must be able to start, pass the internal checks required for upgrade validation, and allow the custom resource to report successful upgrade completion while remaining unavailable for external call routing.
- Persist throughout the maintenance window. The setting must survive pod restarts, replica changes, operator reconciliation, and upgrades. Newly created pods must honor it before responding successfully to external health checks.
- Require explicit reactivation. Completing the upgrade must not automatically restore SBC-facing availability. Administrators should disable maintenance mode after validating the region and approving its return to service.
- Expose maintenance status. Provide a clear status indication or log message distinguishing intentional maintenance from a service failure.
Where applicable, the feature should allow existing calls to drain without interruption while preventing new calls from being admitted. Document the behavior for new SIP INVITE requests received during maintenance, including requests that arrive before the SBC has detected the health-state change.
Business impact
This enhancement would support controlled regional maintenance, reduce the risk of production calls reaching partially upgraded services, and remove the need to use deployment shutdown as the mechanism for controlling SBC routing eligibility.
It would also make upgrade procedures more predictable by allowing administrators to manage service availability independently of the deployment state required by the upgrade process.
Acceptance criteria
With maintenance mode enabled in one region:
The SBC recognizes that region as unavailable and routes new calls to the alternate region after its configured health-check detection interval.
Voice Gateway can be scaled up, restarted, and upgraded without becoming eligible for production call routing.
The operator can complete reconciliation and report the upgrade as successful.
The region remains unavailable to the SBC until an administrator explicitly disables maintenance mode.
Already implemented.
https://github.ibm.com/WatsonOrchestrate/wo-tracker/issues/102619