Node shell (SSM)

Periscope can open an in-browser shell onto the EC2 host behind a Kubernetes node — for debugging kubelet, journald, containerd, or EBS-mounted volumes — without leaving the dashboard and without an SSH key or a bastion. The Node shell button on the node detail page opens an AWS Systems Manager (SSM) Session Manager session, streamed to an xterm.js terminal.

The load-bearing property is per-user AWS impersonation: the session is opened with the operator's own short-lived AWS credentials, minted from their OIDC id_token via sts:AssumeRoleWithWebIdentity — never Periscope's pod identity. An IAM trust policy, not Periscope config, is the source-of-truth gate, and CloudTrail records the session under the human's assumed-role identity. Even a fully compromised Periscope pod cannot open a node shell, because it has no SSM permissions of its own.

This page is the operator setup guide. It assumes no prior experience wiring AWS to OIDC. The design lands issue #105. For the user-facing tour — opening a shell, what to run on the host, and the attribution model — see docs/usage/node-shell.md.

1. What this is, and why it's safe

A node shell is opened in three steps, all of which must succeed:

[user clicks "Node shell"]
   -> Periscope takes the user's OIDC id_token from their session
   -> sts:AssumeRoleWithWebIdentity(role=periscope-node-shell,
                                     token=<the id_token>)
        AWS validates the token against the IAM OIDC provider and the
        role's TRUST POLICY. If the claims don't satisfy it, this fails
        and no session is ever created.
   -> ssm:StartSession(target=i-0abc...)  using the user's creds
   -> CloudTrail records: assumed-role/periscope-node-shell/periscope-<oidc-sub>

Why this is more secure than a single shared role:

• The trust policy is the gate, not Periscope. Periscope's own pod role has zero SSM permissions. The only way to reach a node is to present a live id_token that AWS itself validates. Compromising Periscope does not grant node access.
• Every session is attributed to a human. Because each session uses a per-user assumed role, CloudTrail and the SSM session history record who opened it. A shared bot role would make every session look identical.
• Defense in depth. Three independent gates must all pass: the IAM trust policy (AWS-side), Periscope's tier check (server-side), and the nodeShell.enabled Helm flag. Any one failing denies the shell.

2. Prerequisites

• An EKS (or self-managed) cluster whose nodes are EC2 instances with the SSM agent running and Online. EKS managed node groups and Bottlerocket/AL2023 AMIs ship the agent and register automatically; bare EC2 may need the agent installed and an instance profile with AmazonSSMManagedInstanceCore.
• Periscope running in auth.authorization.mode: tier with OIDC login configured (the node shell is unavailable in dev/shared mode — it needs a real id_token).
• The session-manager-plugin is bundled in the Periscope server image; no operator action needed.
• An OIDC IdP whose id_token Periscope already verifies at login.

Tested IdP matrix

3. Register your IdP as an OIDC provider in AWS IAM

AssumeRoleWithWebIdentity requires AWS to trust your IdP's issuer. This is a one-time, per-AWS-account step.

If you already federate this IdP into IAM (e.g. for IRSA with the same issuer, or an existing web-identity setup) — skip to step 4.

From scratch:

# <issuer> is your OIDC issuer WITH its trailing slash if it has one,
#   Auth0:   https://<tenant>.us.auth0.com/
#   Okta:    https://<org>.okta.com/oauth2/<server>
#   Cognito: https://cognito-idp.<region>.amazonaws.com/<pool-id>
# <client-id> is the OIDC application's client_id — this is the value
#   that appears in the id_token's `aud` claim (NOT the API audience).

aws iam create-open-id-connect-provider \
  --url "https://<tenant>.us.auth0.com/" \
  --client-id-list "<client-id>" \
  --thumbprint-list "$(: 'see note below')"

Note the resulting provider ARN — arn:aws:iam::<account>:oidc-provider/<issuer-host> — you'll reference it in the trust policy.

4. Create the per-user role

Two policies on one role: a trust policy (who may assume it) and a permission policy (what they may do).

4a. Trust policy — the gate

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::<account>:oidc-provider/<issuer-host>"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "<issuer-host>:aud": "<client-id>"
      }
    }
  }]
}

<issuer-host> is the issuer without the https:// scheme but with its trailing slash if present, e.g. <tenant>.us.auth0.com/.

To restrict to specific people at the AWS layer too, add a sub condition (this does work):

"Condition": {
  "StringEquals": {
    "<issuer-host>:aud": "<client-id>",
    "<issuer-host>:sub": ["auth0|abc123", "auth0|def456"]
  }
}

4b. Permission policy — what the role may do

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "StartSessionOnFleetInstances",
      "Effect": "Allow",
      "Action": "ssm:StartSession",
      "Resource": "arn:aws:ec2:<region>:<account>:instance/*",
      "Condition": {
        "StringEquals": {
          "ssm:resourceTag/eks:cluster-name": "<cluster-name>"
        }
      }
    },
    {
      "Sid": "StartSessionDocument",
      "Effect": "Allow",
      "Action": "ssm:StartSession",
      "Resource": "arn:aws:ssm:<region>:<account>:document/SSM-SessionManagerRunShell"
    },
    {
      "Sid": "ManageOwnSessions",
      "Effect": "Allow",
      "Action": ["ssm:TerminateSession", "ssm:ResumeSession"],
      "Resource": "arn:aws:ssm:*:*:session/${aws:userid}-*"
    },
    {
      "Sid": "Preflight",
      "Effect": "Allow",
      "Action": "ssm:DescribeInstanceInformation",
      "Resource": "*"
    }
  ]
}

Create the role with the trust policy as its assume-role policy and attach the permission policy. Note the role ARN — arn:aws:iam::<account>:role/periscope-node-shell.

4c. Multi-account fleets

ssm:StartSession is account-local — you cannot open a session on an instance in account B with a role in account A. So each AWS account whose nodes you want to shell needs its own OIDC provider + role. The same id_token federates into every account that trusts the issuer (one login, no re-auth). Wire each cluster's role per cluster (§5).

A copy-paste Terraform module that creates the OIDC provider + role + both policies lives in hack/poc-ssm-data-channel/terraform (set enable_oidc=true); it's the executable form of the JSON above.

5. Wire it into Periscope's Helm values

nodeShell:
  enabled: true
  # Single-account fleet: one global role covers every cluster's nodes.
  awsRoleArn: "arn:aws:iam::<account>:role/periscope-node-shell"
  # The id_token aud the trust policy expects = your OIDC client_id.
  oidcAudience: "<client-id>"
  region: "<region>"          # falls back to the cluster's region
  tiers: [admin]              # which Periscope tiers may open a shell
  idleSeconds: 600
  transcriptMaxBytes: 1048576

Multi-account: leave the global awsRoleArn empty and set it per cluster in your clusters config — each cluster points at the role in its account:

clusters:
  - name: prod-eu
    backend: agent
    nodeShell:
      awsRoleArn: "arn:aws:iam::<account-B>:role/periscope-node-shell"
      oidcAudience: "<client-id>"
      region: "eu-west-1"

6. Verify without the dashboard

Confirm the trust policy works using a real id_token, before touching Periscope. Obtain an id_token from your IdP (a normal OIDC login), then:

aws sts assume-role-with-web-identity \
  --role-arn "arn:aws:iam::<account>:role/periscope-node-shell" \
  --role-session-name "verify-$(whoami)" \
  --web-identity-token "file://id-token.txt" \
  --query 'AssumedRoleUser.Arn' --output text

A printed arn:aws:sts::...:assumed-role/periscope-node-shell/... means the OIDC provider + trust policy are correct. An AccessDenied means the token's aud/iss don't match the provider/trust policy — see §7.

To also confirm the SSM agent is reachable, export the returned credentials and run:

aws ssm describe-instance-information \
  --filters "Key=InstanceIds,Values=i-0abc..." \
  --query 'InstanceInformationList[0].PingStatus' --output text
# -> Online

Periscope's preflight (GET .../nodes/{name}/shell/preflight) runs exactly these two checks before opening the WebSocket, so a clean preflight means the real session will almost certainly succeed.

7. Troubleshooting

8. Limitations and threat model

Protects against:

• Cross-tier / unauthorized access — the trust policy (AWS-side) plus the tier check (Periscope-side) must both pass.
• Periscope pod compromise — the pod has no SSM permissions; node access requires a live, user-presented id_token AWS validates.
• Lost attribution — every session is tied to a human via the per-user assumed role (CloudTrail) and the audit session_id.

Does NOT protect against:

• A legitimately authorized operator — anyone whose tier allows the shell, and who can pass the trust policy, has full host access (the generic ssm-user, with whatever sudo the node grants it). Scope nodeShell.tiers accordingly.
• SSM agent vulnerabilities or AWS-side SSM issues.
• Whatever the operator does once inside the shell — the transcript (captured in the ssm_session_close audit row, capped at transcriptMaxBytes) is the forensic record, not a preventive control.

Audit. Two rows per session — ssm_session_open and ssm_session_close — with the assumed-role identity, instance, and (on close) the full transcript. Cross-reference session_id with CloudTrail for the AWS-side view. See audit and RFC 0003.