allow non-root user containers to use devices

[mythi]

What happened:
Detailed report in a google doc

TL;DR; CRI container runtimes do not follow securityContext (runAsGroup / runAsUser) when creating config.json for device mounts (coming from the device plugins).

This blocks using non-root users with containers that require device resources.

What you expected to happen:
non-root containers get the device access

Anything else we need to know?:
Discussed in sig-node June 16th. Follow-up June 23th.

The action is to verify security implications and if no issues identified, then move to fix this in both containerd and cri-o.

[mythi]

/sig node
/cc @mikebrow @mrunalp @RenaudWasTaken @kad

[mikebrow]

Note: The securitycontext (runAsGroup / runAsUser) is explicitly stated to be for the container processes. There is no directive or suggestion in the runtime spec or CRI api that the process uid/gid (linux) should be applied to each device uid/gid. And it is a valid issue to resolve, a means for assigning device uid/gid.

[kad]

@mikebrow OCI runtime spec expects that device node ownership is optional and can be provided. But we have a disconnect in conversion CRI -> OCI... so we are trying to find best way to solve it: either it is explicitly comes from CRI (and then some ways inside kubelet to populate it) or make it as expectation to runtimes (cri-o, containerd) to not assume host OS ownership, but look at container process ownership to satisfy a promise of using the assigned device.

[mikebrow]

right.. my concern is making sure we have a path to the end goal for how we want to fix the disconnect. For those reading here.. See ongoing thread/discussion in the google doc comments.

[mythi]

@kad @mikebrow @RenaudWasTaken @mrunalp I've updated the Google doc based on what we discussed earlier this week. We ran out of time in SIG-NODE this week but I'll follow-up with this topic in the next call.

Meanwhile, we could discuss and agree on the annotation to use. How about separate entries for user/group as follows:

io.kubernetes.cri.devices.uid: runAsUser
io.kubernetes.cri.devices.gid: runAsGroup

[mikebrow]

To set the security context for each individual device or use a wildcard to set all devices or use some other useful pattern. Suggest: annotation: io.kubernetes.cri.setHostDeviceSecurityContext value={path/wildcards uid=(int or runAsUser) gid=(int or runAsGroup)} where path is either the full or partial path of the host device supporting wildcards to cover case A, runAsUser meaning use the container's uid...

[mikebrow]

another alternative would be to not allow wildcard for the setHostDeviceSecurityContext annotation and use some other annotation for the basic default run as uid:gid of the container process.

[eero-t]

Here's more info on the problem statement:

User can access the device if container user or group happens to match host user/group setting for the device. This s information about the host operating system, and may allow fingerprinting it => potential security issue.

Host group IDs for devices differ between distributions, their versions (e.g. Ubuntu 20.04 added new "render" group for render node devices), and whether user has installed the distribution from scratch or upgraded it. I.e. even knowledge of the nodes' host distribution isn't reliable way to "guess" the correct group ID to access the devices as non-root.

Needing root for reliable device access is a regression compared to accessing them on the Linux host systems, where system arbitrates access for the current user (user at active console automatically gets access to GPU).

[mythi]

To set the security context for each individual device or use a wildcard to set all devices or use some other useful pattern.

Device plugins can set device node specific annotations so perhaps something like:
io.kubernetes.cri.setHostDeviceSecurityContext/dev-vfio-321: {"uid":"123","gid": "42"}

dev-vfio-321 (example) is derived from the file path.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale