flink-kube-operator/README.md

# Flink Kube Operator

A Kubernetes operator for managing Apache Flink jobs via Custom Resource Definitions.

## Installation

```bash
helm repo add lc-flink-operator https://git.logicamp.tech/Logicamp/flink-kube-operator/raw/branch/main/helm/
helm install flink-kube-operator lc-flink-operator/flink-kube-operator
```

## FlinkJob CRD

Define Flink jobs as Kubernetes resources:

```yaml
apiVersion: flink.logicamp.dev/v1alpha1
kind: FlinkJob
metadata:
  name: my-flink-job
spec:
  key: word-count
  name: "Word Count Example"
  entryClass: "com.example.WordCount"
  parallelism: 1
  jarRef: "registry.example.com/myorg/word-count:1.0.0"
  jarPullSecret: "registry-creds"
  args:
    kafka-host: kafka:9092
    kafka-group-id: my-group
```

### Spec Fields

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `key` | string | yes | Unique identifier for the job |
| `name` | string | yes | Human-readable job name |
| `entryClass` | string | yes | Flink job entry class |
| `parallelism` | int | yes | Job parallelism |
| `jarRef` | string | yes | OCI reference to the JAR artifact (e.g. `registry.io/org/app:tag`) |
| `jarPullSecret` | string | no | Name of a `kubernetes.io/dockerconfigjson` Secret for registry auth |
| `args` | map/list | no | Program arguments (map or list format) |
| `savepointInterval` | duration | no | Interval between automatic savepoints |

### Deprecated Fields (backward compat)

| Field | Replaced by |
|-------|-------------|
| `jarUri` | `jarRef` |
| `jarURIBasicAuthUsername` | `jarPullSecret` |
| `jarURIBasicAuthPassword` | `jarPullSecret` |

If both `jarRef` and `jarUri` are set, `jarRef` takes precedence.

## Registry Authentication

### Kubernetes (pull secret)

Create a `kubernetes.io/dockerconfigjson` Secret and reference it via `jarPullSecret`:

```bash
kubectl create secret docker-registry registry-creds \
  --docker-server=registry.example.com \
  --docker-username=user \
  --docker-password=pass
```

```yaml
spec:
  jarRef: "registry.example.com/myorg/my-job:1.0.0"
  jarPullSecret: "registry-creds"
```

### Docker Compose / local dev (environment variables)

Set registry credentials as environment variables on the operator container:

```yaml
# docker-compose.yaml
services:
  operator:
    image: flink-kube-operator:latest
    environment:
      REGISTRY_SERVER: "registry.example.com"
      REGISTRY_USERNAME: "user"
      REGISTRY_PASSWORD: "pass"
```

### Credential Resolution Order

1. `spec.jarPullSecret` — reads the K8s Secret
2. `REGISTRY_USERNAME` + `REGISTRY_PASSWORD` env vars — fallback (optionally scoped by `REGISTRY_SERVER`)
3. Anonymous — public registries

## Pushing JARs to an OCI Registry

Flink job JARs are stored as OCI artifacts with media type `application/java-archive`. Use one of the methods below to push your JAR.

### Using crane (recommended)

[crane](https://github.com/google/go-containerregistry/tree/main/cmd/crane) is a tool for interacting with OCI registries.

```bash
# Install crane
go install github.com/google/go-containerregistry/cmd/crane@latest

# Log in to your registry
crane auth login registry.example.com -u user -p pass

# Push a JAR as an OCI artifact
crane append \
  --file=my-job.jar \
  --new_tag=registry.example.com/myorg/my-job:1.0.0 \
  --base=scratch
```

### Using oras

[oras](https://oras.land/) is a tool for pushing OCI artifacts.

```bash
# Install oras
# https://oras.land/docs/installation

# Log in to your registry
oras login registry.example.com -u user -p pass

# Push a JAR with the correct media type
oras push registry.example.com/myorg/my-job:1.0.0 \
  --artifact-type application/java-archive \
  my-job.jar:application/java-archive
```

### Using Docker (buildkit)

```bash
# Create a minimal Dockerfile
cat > Dockerfile.jar <<'EOF'
FROM scratch
COPY my-job.jar /my-job.jar
EOF

# Build and push
docker build -f Dockerfile.jar -t registry.example.com/myorg/my-job:1.0.0 .
docker push registry.example.com/myorg/my-job:1.0.0
```

### GitHub Actions (CI/CD)

Use [crane](https://github.com/google/go-containerregistry/tree/main/cmd/crane) in your GitHub Actions workflow to push JARs as OCI artifacts. `docker/build-push-action` is not suitable — it builds container images, not arbitrary OCI artifacts.

```yaml
name: Publish JAR

on:
  push:
    tags:
      - 'v*'

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}/my-job

jobs:
  publish:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write

    steps:
      - uses: actions/checkout@v4

      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Set up crane
        uses: imjasonh/setup-crane@v0.4

      - name: Build JAR
        run: mvn package -DskipTests

      - name: Push JAR to OCI registry
        run: |
          crane append \
            --base=scratch \
            --new_tag=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }} \
            --new_layer=target/my-job.jar \
            --media_type=application/java-archive
```

For GitLab CI, Bitbucket Pipelines, or other CI systems, use the same `crane append` command after authenticating to your registry.

### Versioning Strategy

- Use semantic version tags (`:1.0.0`, `:1.1.0`) for explicit version control
- Changing the tag in `spec.jarRef` triggers a job upgrade (pause → re-upload → resume from savepoint)
- The operator stores the SHA256 digest in `status.runningJarDigest` after each pull