flink-kube-operator/README.md

Flink Kube Operator

A Kubernetes operator for managing Apache Flink jobs via Custom Resource Definitions.

Installation

helm repo add lc-flink-operator https://git.logicamp.tech/Logicamp/flink-kube-operator/raw/branch/main/helm/
helm install flink-kube-operator lc-flink-operator/flink-kube-operator

FlinkJob CRD

Define Flink jobs as Kubernetes resources:

apiVersion: flink.logicamp.dev/v1alpha1
kind: FlinkJob
metadata:
  name: my-flink-job
spec:
  key: word-count
  name: "Word Count Example"
  entryClass: "com.example.WordCount"
  parallelism: 1
  jarRef: "registry.example.com/myorg/word-count:1.0.0"
  jarPullSecret: "registry-creds"
  args:
    kafka-host: kafka:9092
    kafka-group-id: my-group

Spec Fields

Field	Type	Required	Description
key	string	yes	Unique identifier for the job
name	string	yes	Human-readable job name
entryClass	string	yes	Flink job entry class
parallelism	int	yes	Job parallelism
jarRef	string	yes	OCI reference to the JAR artifact (e.g. registry.io/org/app:tag)
jarPullSecret	string	no	Name of a kubernetes.io/dockerconfigjson Secret for registry auth
args	map/list	no	Program arguments (map or list format)
savepointInterval	duration	no	Interval between automatic savepoints

Deprecated Fields (backward compat)

Field	Replaced by
jarUri	jarRef
jarURIBasicAuthUsername	jarPullSecret
jarURIBasicAuthPassword	jarPullSecret

If both jarRef and jarUri are set, jarRef takes precedence.

Registry Authentication

Kubernetes (pull secret)

Create a kubernetes.io/dockerconfigjson Secret and reference it via jarPullSecret:

kubectl create secret docker-registry registry-creds \
  --docker-server=registry.example.com \
  --docker-username=user \
  --docker-password=pass

spec:
  jarRef: "registry.example.com/myorg/my-job:1.0.0"
  jarPullSecret: "registry-creds"

Docker Compose / local dev (environment variables)

Set registry credentials as environment variables on the operator container:

# docker-compose.yaml
services:
  operator:
    image: flink-kube-operator:latest
    environment:
      REGISTRY_SERVER: "registry.example.com"
      REGISTRY_USERNAME: "user"
      REGISTRY_PASSWORD: "pass"

Credential Resolution Order

1. spec.jarPullSecret — reads the K8s Secret
2. REGISTRY_USERNAME + REGISTRY_PASSWORD env vars — fallback (optionally scoped by REGISTRY_SERVER)
3. Anonymous — public registries

Pushing JARs to an OCI Registry

Flink job JARs are stored as OCI artifacts with media type application/java-archive. Use one of the methods below to push your JAR.

Using crane (recommended)

crane is a tool for interacting with OCI registries.

# Install crane
go install github.com/google/go-containerregistry/cmd/crane@latest

# Log in to your registry
crane auth login registry.example.com -u user -p pass

# Push a JAR as an OCI artifact
crane append \
  --file=my-job.jar \
  --new_tag=registry.example.com/myorg/my-job:1.0.0 \
  --base=scratch

Using oras

oras is a tool for pushing OCI artifacts.

# Install oras
# https://oras.land/docs/installation

# Log in to your registry
oras login registry.example.com -u user -p pass

# Push a JAR with the correct media type
oras push registry.example.com/myorg/my-job:1.0.0 \
  --artifact-type application/java-archive \
  my-job.jar:application/java-archive

Using Docker (buildkit)

# Create a minimal Dockerfile
cat > Dockerfile.jar <<'EOF'
FROM scratch
COPY my-job.jar /my-job.jar
EOF

# Build and push
docker build -f Dockerfile.jar -t registry.example.com/myorg/my-job:1.0.0 .
docker push registry.example.com/myorg/my-job:1.0.0

GitHub Actions (CI/CD)

Use crane in your GitHub Actions workflow to push JARs as OCI artifacts. docker/build-push-action is not suitable — it builds container images, not arbitrary OCI artifacts.

name: Publish JAR

on:
  push:
    tags:
      - 'v*'

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}/my-job

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - uses: actions/checkout@v4

      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Set up crane
        uses: imjasonh/setup-crane@v0.4

      - name: Build JAR
        run: mvn package -DskipTests

      - name: Push JAR to OCI registry
        run: |
          crane append \
            --base=scratch \
            --new_tag=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }} \
            --new_layer=target/my-job.jar \
            --media_type=application/java-archive

For GitLab CI, Bitbucket Pipelines, or other CI systems, use the same crane append command after authenticating to your registry.

Versioning Strategy

• Use semantic version tags (:1.0.0, :1.1.0) for explicit version control
• Changing the tag in spec.jarRef triggers a job upgrade (pause → re-upload → resume from savepoint)
• The operator stores the SHA256 digest in status.runningJarDigest after each pull