Instance requests, approval, and safe host recovery

Release status: archiving and bounded host startup

The Delete record archive workflow and bounded startup checks have been released to production and staging. Optional AWS observation still depends on environment configuration. These controls report lifecycle state; they do not prove that a failed host, its data, or workspace access has been restored. Check readiness and client access separately.

Request and launch a new instance

A new managed host requires an explicit review and execution process. Creating a draft is not a launch, and approval is not execution. Direct launch and automatic new-host allocation are not offered by this admin workflow.

1. Create a draft: as PartnerAdmin or SuperAdmin, complete Request an instance. This saves only the draft configuration.
2. Submit for review: the original requester submits their Draft (or resubmits their Rejected request).
3. Approve or reject: a SuperAdmin reviews the submitted request. The approver must be different from both the original requester and the person who submitted it. A rejection reason is optional.
4. Publish / execute: after independent approval, a SuperAdmin separately starts execution. The publisher may be the requester or approver; a third distinct person is not required.
5. Verify launch and grant: inspect the resulting host and request state. A SuperAdmin must explicitly grant the launched host to the requesting partner before its PartnerAdmin can place clients on it.

Launching can be disabled in an environment. Drafts, submission, and approval may still work, but Publish / execute remains disabled with the API's reason. Approval does not override that environment-level block.

Instance request field reference

When comparing performance, use Instance type's vCPU and memory, not just the Small/Standard/Large label. See the tier and hardware explanation.

Read the instance-request table

• Request: name, EC2 type, plan size, client capacity, and retained host ID if available.
• Requested by: owning partner and the draft's creator.
• Status / decision: lifecycle status and the recorded submitter, approver, or rejector.
• Lifecycle: timestamps for request, review, decision, launch attempt, launch, and execution.
• Rejection reason: optional reviewer explanation. A dash means no reason was recorded.
• Actions: operations available for the current role and state. Some rows have no action.

History and concurrent changes

Expand History for ordered server audit entries: event, resulting status, version, actor, time, and any reason. Actions are guarded by request version. If another operation changes a record first, refresh the table to obtain its current state before acting again.

Statuses and next steps

The normal path is Draft → SubmittedForReview → Approved → Launching → Launched. Rejection returns the request to a review path through requester resubmission. LaunchFailed is an execution failure, not a review rejection or a signal that a replacement instance will be launched.

Delete record means archive, not infrastructure deletion

Delete record is a SuperAdmin-only archive/hide operation for an instance-request record. It hides the record from the default active list without erasing its immutable history, identity, or idempotency information. Hosts, clients, and disks are unchanged.

1. Check the record's identity, current status, and history, and confirm you intend to hide the request rather than remove its host.
2. As SuperAdmin, use Delete record only on an eligible request and read the confirmation.
3. Refresh the list and use the archived-record view to verify the preserved record and history.

Archiving does not reset launch idempotency, permit a duplicate execution, undo an approval, or imply that the linked host was stopped. An archived Draft or Rejected request cannot be submitted/resubmitted or published/executed. Do not treat archiving as a way to restart its workflow.

Show archived requests within the same partner scope

The release-dependent Show archived requests view makes archived records available for inspection while retaining role authorization and partner scoping. PartnerAdmins remain limited to their own partner's requests; archiving does not make another partner's history visible.

An archived LaunchFailed request still supports SuperAdmin Reconcile existing host and Delete host where each action is otherwise eligible. Open the archived view, verify the linked host and current state, and follow the same recovery checks or empty-managed-host deletion safeguards. Archiving itself neither disables safe failed-launch cleanup nor authorizes deletion.

Reconcile an existing host; do not retry the launch

Reconcile existing host compares an uncertain request with the identity of its already-existing host. It can recover the request's launched state when that host can be identified. It does not create a replacement host, repeat execution, or retry a failed launch.

1. As SuperAdmin, inspect the request, retained host ID, current state, and audit history.
2. For Launching, wait until the request has been in that state for at least 10 minutes. LaunchFailed requests can also be checked.
3. Use Reconcile existing host and read the result. Refresh the request and host records rather than assuming recovery succeeded.
4. If the request is recovered as Launched, verify the host and explicitly grant it. If recovery cannot be confirmed, follow your operator's recovery procedure instead of repeatedly publishing or launching another host.

Reconciliation and deletion have different goals. Reconcile preserves/checks existing identity; termination removes infrastructure. Deleting a host does not reconcile the request or erase its audit trail.

A corrected request status is not proof that a dead host has been restored or client routing repaired. Verify actual host readiness and client access separately; reconciliation is not a replacement instance launch or a routing repair operation.

Startup timeout: bounded Starting grace, not a relaunch

Host startup grace is separate from normal heartbeat monitoring. Cells:StartupTimeoutSeconds defaults to 900 seconds (15 minutes). A host in Starting receives a bounded grace period measured from StateChangedAt; non-ready heartbeats do not extend that startup window.

When the startup window expires without readiness, the host is marked Failed. Timeout does not relaunch it, delete its host record, terminate infrastructure, or delete disks or clients. This host startup timer is separate from the instance request's 10-minute Launching eligibility for reconciliation and from a client's idle-stop threshold.

Ordinary client wake requests do not automatically restart a Failed host. An explicit SuperAdmin host Start remains a separate recovery operation; it is not proof that the underlying failure has been resolved. Have the operator inspect the failure before attempting recovery.

AWS Running is not application readiness

AWS Running means the virtual machine is powered on. The Platform Portal can still show Starting while installation, containers, or dependencies are not ready. The operator should inspect cloud-init output, container health and the cell's configured private /health/ready endpoint. A liveness response or an HTML page returning 200 is not readiness evidence.

Cell readiness checks database, queue, search and storage. All four must pass. A storage failure may be a scoped permission or configuration problem, not a missing bucket; do not grant broad bucket permissions as a shortcut. Fix the reported dependency and verify a heartbeat from the registered instance before treating the host as Running.

Optional read-only AWS terminated-instance checks

The backend can optionally observe AWS termination using the host's exact EC2 instance ID, with host identity, managed-host, and environment tag checks. This is a read-only observation of matching infrastructure, not a broad search, automatic replacement launch, or termination command.

Only a matching observation of shutting-down or terminated permits early failure reporting. Missing instances, mismatched identity/tags, and observation errors do not prove termination; startup monitoring falls back to the bounded deadline rather than treating those results as confirmed deletion.

Failure/termination detection improves lifecycle reporting; it does not restore an actually terminated machine, recover its disk, or prove client routes have been repaired. Follow the operator's recovery procedure and verify host readiness and client access before declaring service restored.

Start, Stop, and Terminate are different

In Hosts, SuperAdmins can start or stop managed hosts. Stop shuts down compute without deleting the persistent disk; it is not termination. Idle-stop and Always on govern automatic availability behavior, as described in the client power settings guide.

Terminate removes an eligible managed host and deletes its disk. It requires zero clients. Registered host type, state, and server validation determine whether a control is available; a host ID on a failed request is not sufficient authorization to terminate it.

Delete an empty managed host safely

Destructive operation: host termination deletes the host disk. Daily snapshots already taken are retained for seven days under the documented platform policy. Do not assume a snapshot exists, that it includes recent writes, or that deletion is automatically reversible; confirm your deployment's backup policy with the operator.

1. Confirm you are SuperAdmin and have selected the intended host, not just a request with a similar name.
2. Verify the host is managed and has zero attached clients. A request-row Delete host action additionally requires a failed request linked to that verified empty managed host.
3. If the launch outcome is uncertain and you intend to recover it, use reconciliation first. Do not use deletion as a recovery shortcut.
4. Use Terminate in Hosts, or Delete host on an eligible failed request where that control is available in your release.
5. Read the confirmation prompt, verify the host identity and disk-deletion warning, then confirm only if removal is intended.
6. Check the API result and refreshed host list. The server can reject the operation if state changed or clients were attached after the page loaded.

Request-row Delete host availability

The request-row Delete host control is release-dependent. It is limited to a failed request's linked, verified empty managed host; it is not a general Delete request or Delete any host button. If absent, use Hosts → Terminate for an eligible host. If disabled, resolve the displayed state or loading error rather than bypassing it.

Reconcile remains a separate recovery action; the presence of Delete host does not turn reconciliation into deletion. Termination removes the linked host, not the instance-request record. Its retained host ID and history can remain visible as lifecycle/audit context.

Delete record versus Delete host

Troubleshooting

• Approve is unavailable: verify submitted status and independent SuperAdmin review. The requester/submitter cannot approve their own request.
• Publish / execute is disabled: read the environment's launch-disabled reason; approval does not override it.
• Status is stale or a version conflict appears: refresh the request table and history before acting again.
• Launch failed: do not assume no machine exists. Check retained identity with reconciliation; there is no automatic launch retry.
• Delete host is missing or disabled: check release availability, SuperAdmin access, failed request state, verified host identity, managed type, and zero clients. Host loading failure is not proof of emptiness.
• Delete record is missing or disabled: confirm that the archive release is deployed, you are SuperAdmin, and the status is Draft, Rejected, Launched, or LaunchFailed. SubmittedForReview/Pending, Approved, and Launching cannot be archived.
• An archived request disappeared from the active list: use Show archived requests within your authorized partner scope. Archived Draft/Rejected records cannot be resubmitted or published; archived failed requests retain eligible recovery/host cleanup actions.
• A host keeps reporting Starting: confirm whether the startup-timeout fix is deployed and inspect StateChangedAt/readiness with the operator. Non-ready heartbeats should not reset the 15-minute default grace. A Failed status does not mean the host was relaunched or client routing restored.
• Partner cannot select the new host: verify the explicit host grant and free capacity after launch.