Creating a Kubernetes Controller to Get GHS Token for a GitHub Application

Posted on
There are many ways to handle repeatable jobs in Kubernetes. For some cases, you can use a CronJob to run recurrent tasks. However, when you need to interact with Kubernetes objects, resources, or custom resources, implementing your controller is a more effective way to maintain the desired state with minimal effort.

In my case, I’ve created a special Kubernetes controller to work with a GitHub App to exchange the App JWT for an installation-specific and short-lived GHS token. This controller updates the token before it expires and updates some third-party integrations such as ArgoCD OCI repository credentials and Dockerconfig JSON secrets. Unfortunately, this controller is currently useless due to GitHub limitations: only personal access tokens (classic) can access private registries (see GitHub discussion for more details).

TL;DR

References to create your own Kubernetes controller:

My implementation:

GitHub discussion:

  • Using GitHub Apps to access private registry is here

Basic Understanding of Kubernetes Controllers

Kubernetes controllers are control loops that monitor the state of your cluster resources and make necessary adjustments. Kubernetes provides several built-in controllers to manage various resources such as pods, deployments, services, and replica sets. By design, controllers continuously reconcile the current state of the cluster resources with the desired state defined in the Kubernetes resource manifests.

To understand what is under the hood of Kubernetes Controllers, we can refer to an official controller example, sample-controller. This repository provides a comprehensive diagram that helps to understand the underlying components of a typical Kubernetes controller.

client-go and controller interaction

You can see that the client-go library covers a lot of interaction and components by itself.

From the custom Kubernetes controller implementation perspective, we can identify two main tasks in the controller workflow:

  • Use informers to watch add/update/delete events for the Kubernetes resources we want to know about.
  • Consume items from the workqueue and process them.

client-go provides informers for standard resources, for example, deployments: k8s.io/client-go/informers/apps/v1/DeploymentInformer. These informers contain everything needed at the top of the picture, they provide the reflector, indexer, and local storage.

But we are here not for watching some resources, we need to define and proceed with our own.

How I Created a Kubernetes Controller

Here is a high-level plan to achieve my goals:

  1. Create Custom Resource Definition (CRD) for ArgoCD repositories.
  2. Create CRD for Dockerconfig JSON.
  3. Create CRD for GHS tokens.
  4. Implement logic to manage ArgoCD repositories.
  5. Implement logic to manage Docker configurations.
  6. Implement logic to remove expired GHS tokens.

Preparation

Before we begin, we need to get code-generator. It helps us to automatically generate tons of code instead of manually writing it.

Then we need to update the hack/update-codegen.sh script to have something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/usr/bin/env bash
set -o errexit
set -o nounset
set -o pipefail

SCRIPT_ROOT=$(dirname "${BASH_SOURCE[0]}")/..
CODEGEN_PKG=${CODEGEN_PKG:-$(cd "${SCRIPT_ROOT}"; ls -d -1 ./vendor/k8s.io/code-generator 2>/dev/null || echo ../code-generator)}

source "${CODEGEN_PKG}/kube_codegen.sh"

THIS_PKG="github.com/technicaldomain/github-app-jwt2token-controller"

kube::codegen::gen_helpers \
    --boilerplate "${SCRIPT_ROOT}/hack/boilerplate.go.txt" \
    "${SCRIPT_ROOT}/pkg/apis"

kube::codegen::gen_client \
    --with-watch \
    --output-dir "${SCRIPT_ROOT}/pkg/generated" \
    --output-pkg "${THIS_PKG}/pkg/generated" \
    --boilerplate "${SCRIPT_ROOT}/hack/boilerplate.go.txt" \
    "${SCRIPT_ROOT}/pkg/apis"

Custom Resource Definition

Here, we need to create YAML definitions for all our custom resources:

CRD definition for the resource that be responsible for updating the password field for OCI repository by GHS token generated by the controller:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: argocdrepos.githubapp.technicaldomain.xyz
spec:
  group: githubapp.technicaldomain.xyz
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                privateKeySecret:
                  type: string
                argoCDRepositories:
                  type: array
                  items:
                    type: object
                    properties:
                      repository:
                        type: string
                      namespace:
                        type: string
            status:
              type: object
              properties:
                token:
                  type: string
      subresources:
        status: {}
  scope: Namespaced
  names:
    plural: argocdrepos
    singular: argocdrepo
    kind: ArgoCDRepo
    shortNames:
      - gha

CRD definition for the resource that be responsible for generating/updating docker config JSON with the actual GHS token:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: dockerconfigjsons.githubapp.technicaldomain.xyz
spec:
  group: githubapp.technicaldomain.xyz
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                privateKeySecret:
                  type: string
                dockerConfigSecrets:
                  type: array
                  items:
                    type: object
                    properties:
                      secret:
                        type: string
                      namespace:
                        type: string
                      registry:
                        type: string
                        default: "ghcr.io"
                      username:
                        type: string
                        default: "x-access-token"
            status:
              type: object
              properties:
                token:
                  type: string
      subresources:
        status: {}
  scope: Namespaced
  names:
    plural: dockerconfigjsons
    singular: dockerconfigjson
    kind: DockerConfigJson
    shortNames:
      - dcj

CRD definition to store all generated GHS tokens:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ghss.githubapp.technicaldomain.xyz
spec:
  group: githubapp.technicaldomain.xyz
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                token:
                  type: string
            status:
              type: object
              properties:
                expiresAt:
                  type: string
                  format: date-time
      subresources:
        status: {}
  scope: Namespaced
  names:
    plural: ghss
    singular: ghs
    kind: GHS
    shortNames:
      - ghs

Please pay attention, we are trying to create namespaced CRDs to limit access to them in the future if needed.

Then, we need to define types in pkg/apis/githubappjwt2token/v1/types.go to be able to work with defined CRDs. Also, we need to add special annotations in the code to tell code-generator what should be generated for this type. For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type ArgoCDRepo struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   ArgoCDRepoSpec   `json:"spec,omitempty"`
	Status ArgoCDRepoStatus `json:"status,omitempty"`
}

type ArgoCDRepoSpec struct {
	PrivateKeySecret   string               `json:"privateKeySecret"`
	ArgoCDRepositories []ArgoCDRepositories `json:"argoCDRepositories"`
}

type ArgoCDRepositories struct {
	Repository string `json:"repository"`
	Namespace  string `json:"namespace"`
}

type ArgoCDRepoStatus struct {
	Token     string      `json:"token,omitempty"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type ArgoCDRepoList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []ArgoCDRepo `json:"items"`
}

The full implementation you can find here - types.go

Right now we can run ./hack/update-codegen.sh to generate clientset, informers, and listers.
All generated codebases will be located under the pkg/generated path of the repo. And we can go to the implementation logic of our controllers.

Controller Logic

The logic for controller_argocdrepo.go and controller_dockerconfigjson.go is pretty similar and described in the diagram below:

controller argocdrepo dockerconfigjson

Every resource with kinds ArgoCDRepo and DockerConfigJson in the scope of githubapp.technicaldomain.xyz/v1 is processed by the appropriate controller.
The controller checks the resource status (subresource), in the status recorded md5 of the appropriate GHS custom resource. The controller checks for the existence of this GHS resource, if it exists do nothing, otherwise, the controller calls a special function to retrieve the GitHub App private key, GitHub App ID, and GitHub App Installation ID from the secret (name of the secret defined in the custom resource field).

Then the controller creates and signs JWT based on the GitHub App private key, GitHub App ID, and GitHub App Installation ID, after that the controller calls a special GitHub API endpoint to exchange this JWT for a GHS token. Then calculate md5 for this token, create a new GHS resource, and store inside this resource token and information about token expiration. md5 is used as the name of the GHS resource, also this name is recorded in ArgoCDRepo or DockerConfigJson status field.

After that, based on the type of the resource, the controller looks for all ArgoCD repositories defined in the ArgoCDRepo custom resource and updates the password field to the new token.
For DockerConfigJson the story is about the same, but in this case, the controller generates a docker config.

The logic implemented in controller_ghs.go is much more straightforward.

controller ghs

Here the Kubernetes controller just checks for the resource expiration time, and if it is less than 15 minutes - simply delete the resource. That’s it.

Local Run and Debug

To debug this controller, you should have a deployed and accessible Kubernetes cluster (k3s works, I’ve checked).

To run the controller locally, please ensure that you have updated your kubeconfig and that your cluster is set for debugging in the selected context.

Then simply run go run . -kubeconfig=$HOME/.kube/config. The controller will be compiled and will try to connect to your Kubernetes cluster.

To work with your favorite debugger and put some point break, please refer to your IDE guidelines and best practices.

Installation

This controller can be easily installed using the github-app-jwt2token-controller Helm chart.

Here is an example of an ArgoCD application:

Folder structure:

1
2
3
4
.
├── Chart.yaml
└── templates
    └── resources.yaml

And content

Chart.yaml
1
2
3
4
5
6
7
8
9
apiVersion: v2
name: github-app-jwt2token
version: 1.0.0

dependencies:
  - name: github-app-jwt2token-controller
    version: 1.1.2
    repository: https://technicaldomain.github.io/helm

templates\resources.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: githubapp.technicaldomain.xyz/v1
kind: ArgoCDRepo
metadata:
  name: example-argo-github-app
spec:
  privateKeySecret: technicaldomain-gha-argo-app
  argoCDRepositories:
    - repository: repo-1114444111
      namespace: argocd
---
apiVersion: githubapp.technicaldomain.xyz/v1
kind: DockerConfigJson
metadata:
  name: example-docker-config
spec:
  privateKeySecret: technicaldomain-gha-argo-app
  dockerConfigSecrets:
    - secret: ghcr
      namespace: gha-token-controller
      registry: ghcr.io
      username: x-access-token

A secret with the name technicaldomain-gha-argo-app should be created in advance using the same namespace where the controller is deployed.
Here is an example of the secret:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
apiVersion: v1
kind: Secret
metadata:
  name: github-app-private-key
  namespace: default
stringData:
  appId: "1111111"
  installationId: "2222222"
  privateKey: |
    -----BEGIN RSA PRIVATE KEY-----
    MIICWgIBAAKBgExrZNCJPDFzjkAzOwlJASNurUkmgzy66yx2tXf1xVZanMN8w5zq
    jQGct+vv3TAXEix5qHG+yMfY0tz2NVEYKcNKjcQnvbDpLtm9qjZgZ2UiDI92civR
    lIGt/0DzXp3Ooq2wxW3rUoIBQE5kBMFdvxJ/QC0zGrKjbQciABTHgj49AgMBAAEC
    gYBJtyauilMIGMHVaBXApS118mMxtvbNdDk60N/H8coDvLCPWiCPkymlrnk0HFMu
    +nJLeKdl4XVoYd01zEIuEbLmYpdafLW5GZ9QtUBno7wVh1js3ahbbwy1+Isj1Xk1
    Wu7nKDbkgYirwb0Ly+CmrieQ+X/Pnm8Z7eyfpPYSDhMQuQJBAJfjfbtAdAuSrS+A
    v9Ti0uqHVzcUeBTF1JWFYnvIRdnx/S4geLdVxmI6Lque5jJdkB2Vmb38fVZzi7Sh
    DKBqT08CQQCAzQgNrpPPhjdn115P9ZAOh5uo7lZFFG+7mc065cZaJyT+PgyuI5pI
    wjdM/Ne/mVSy9eI6GydG+JGWcTOo5BazAkAKBqQ4Bgsi8G2qIw+Gl+pgPMrPAfTj
    OiPMMt/LV+70cfrKXq5ZO7o6paiK/5QmYvKuYT+iwNXtLPdd1vukYyAVAkAaLA93
    4EKOx8IYaq3yZ36nRSz/LbcAAIAXyc/nKOueRBgDRY6EEB34rOZZ0YLxnvGUD9yx
    W/UmObozrLsHlZl7AkBxoq8qqsNkcQtz970QPoh9WSAIk0afOgcWKyKIyeN8Uu2i
    qraqR6mYwgJSPcc1RbOD7ykdMMq03g6upsk4Ws0P
    -----END RSA PRIVATE KEY-----
type: Opaque

Conclusion

Creating a Kubernetes controller is not a complicated process. The official documentation and examples provide a quick understanding of what you need to do and how to do it. Creating your controller and custom resources unlocks the unlimited power of Kubernetes and brings real flexibility in extending functionality.

Unfortunately, my controller is currently limited due to the absence of necessary functionality from GitHub’s side. However, I believe that it will be implemented someday, allowing me to enjoy all the features provided by my controller.