Don’t Write a (builtin) API, Write a CRD!

    Who am I?

    Solly Ross (@directxman12 /

    KubeBuilder Maintainer and Software Engineer on GKE

    My mission is to make writing Kubernetes extensions less arcane

      So… adding APIs to Kubernetes

      a (snarky) Oddessy

        How does one add a (built-in) API to Kubernetes?

        1. Make a folder in vendor/<group>/v<version>, and write your public types, with JSON and proto tags.

        2. Copy those types to pkg/apis/<group> to make your internal types, which are probably exactly the same as your external types, except without JSON tags.

        3. Make sure you’ve engraved all the magic runes added all the xyz-gen markers to your types

        4. Write defaulting code in defaults.go

        5. Generate automatic conversions between the internal type and the external type (hopefully conversion-gen doesn’t explode or fail to run because the file doesn’t exist)

        6. Write manual conversions (if your internal and external types don’t match)

        7. Write validation code in validation.go

        8. Start code generation (go-to-protobuf, client-gen, lister-gen, informer-gen) and go make some tea 🍵

        9. Create a registry & storage implementation in pkg/registry by copying somebody else’s registry cause it’s probably the same. You’ll also add your print columns and such here.

        10. Make sure you’ve got an install package in your group directories, and add your API version to the API server’s list, scheme, etc.

        11. Edit a pelethora of bookkeeping files, including (but not limited to): hack/lib/, hack/ 1, various linter files, and various linter exception files 2.

        12. Update the fuzzer if you have things like tagged unions.

        13. Start the non-code generators (generated-docs, generated-swagger-docs, openapi-spec) and go make another cup of tea 🍵

        14. Edit the cmd tests to make sure kubectl gives proper output.

        15. Edit kubectl to implement a new describe command for your type.

        16. Optional, but suggested: realize that you messed something up, make some changes, forget to run an update, have some test break subtly. Run ./hack/ in anger, go bake a cake 🍰, and come back to find that your laptop has overheated.

        1. not to be confused with hack/, which runs in a docker container.

        2. at any given point in time, it’s fairly likely that some generated file violates the project’s linter rules.

        So, what are the issues here?

          Enter CRDs!

          1. Make API folder anywhere

          2. Add external types with JSON tags

          3. Add markers to your fields for declarative defaulting, validation, print columns

          4. Run code generation if you really must 3

          5. Run controller-gen to generate your CRDs

          6. Add some tests to make sure your examples work and give the right output

          1. i.e. you want generated clients, informers, etc

          What does this bring us?

            But I don’t want to write OpenAPI by hand!

            Enter controller-gen!

              But what if I need …

                But what if I need …

                If we have problems with the existing validation, users will have them too! We should extend the format values for hard-to-validate common types.

                declarative validation

                // +kubebuilder:validation:Format="date-time"
                type DateTime string
                type Spec struct {
                    // +optional
                    // +kubebuilder:validation:Minimum=0
                    Replicas *int32 `json:"replicas"`
                    Next DateTime `json:"next"`
                    // +kubebuilder:validation:Pattern=`^\w+-\d{2,}$`
                    Prefix string `json:"prefix"`
                type Status struct {
                    Previous DateTime `json:"previous"`

                  But what if I need …

                  Immutability, Defaulting, & Apply

                  type Spec struct {
                      // +immutable
                      // coming soon™
                      DeepLink string `json:"deepLink"`
                      // +default={fixedReplicas: 33}
                      ScaleOptions *ScaleOptions  `json:"replicas"`
                      // +listType=map
                      // +listMapKey=key
                      ActuallyAMap []Item `json:"actuallyAMap"`
                  type ScaleOptions struct {
                      FixedReplicas *int32 `json:"fixedReplicas"`
                      Burst int32 `json:"burst"`
                  type Item struct {
                      Key string `json:"key"`
                      Value string `json:"value"`
                  1. at this point, @directxman12 looks around the room to SIG API Machinery for confirmation

                  But what if I need conversion ?

                  Conversion webhooks

                  Conversion webhooks

                      strategy: Webhook
                        caBundle: ¯\_(ツ)_/¯ 
                          namespace: kube-system
                          name: my-crd-webhook
                          path: /convert
                  type HubVersion struct { ... }
                  func (*HubVersion) Hub() {}
                  // +kubebuilder:storageversion
                  type OtherVersion struct { ... }
                  func (v *OtherVersion) ConvertTo(dst conversion.Hub) error {

                    But what if I need cross-field defaulting ?

                    Please don’t

                      But what if I need to embed a built-in API type ?

                      controller-gen will generate a basic validation schema for those types, but you won’t get full validation until pod create time Sometimes, this is desirable.

                      Otherwise, you’ll need to use a validating webhook.

                      We could improve the situation by any of:

                      1. cue concerned glaces from SIG API Machinery

                      But what if I need subresources ?

                      Scale and Status have built-in support in CRDs

                      Scale & Status subresources

                      // +kubebuilder:subresource:status
                      // +kubebuilder:subresource:scale:specpath=".spec.replicas",statuspath=".status.replicas"
                      type MyKind struct {}

                        No, I mean <insert bespoke subresource here>

                        Did you check with API review 🤨?


                          You’ll need a built-in API

                          If we get too many of these, we should eventually consider supporting these in CRDs as well.

                            But what if I need high read-write peformance ?

                            You’ll need a built-in API

                            We don’t have proto support yet for CRDs.

                            We’ll want to add proto support to CRDs eventually

                              But what if I need field selectors ?

                              Are you Pod, coming from the glorious future where everything is CRDs and we wear shiny eyemasks while we drive our hovercars?

                                No, but…

                                Use an informer with an indexer.

                                  But what if I need my types to be present for the APIServer to boot ?

                                  You’ll want a built-in API

                                  It’s technically possible to overcome this bootstrap problem in some cases, but you’ll need to be extra careful.

                                    But what if I need to actually install my types on cluster boot ?


                                    RuntimeClass and CSIDriver had issues since they had in-tree controllers depending on their CRDs, and we never quite solved them.

                                    Talk to SIG Cluster Lifecycle…

                                      So… have people actually done this?


                                          Any Questions?