feat(jar): add OCI registry pull for JAR artifacts with backward compat
- Add jarRef and jarPullSecret fields to FlinkJob CRD (jarUri/basicAuth deprecated) - OCI pull via go-containerregistry with dual auth (K8s pull secret + env vars) - Media type validation on pulled layers - Atomic status patches with runningJarRef/runningJarDigest tracking - NeedsUpgrade/RunningRef/RunningRefPatchData domain helpers - README with usage guide, pushing JARs, and GitHub Actions CI/CD workflow - CONTEXT.md domain glossary
This commit is contained in:
201
README.md
201
README.md
@@ -1,6 +1,203 @@
|
||||
Installation:
|
||||
# Flink Kube Operator
|
||||
|
||||
A Kubernetes operator for managing Apache Flink jobs via Custom Resource Definitions.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
helm repo add lc-flink-operator https://git.logicamp.tech/Logicamp/flink-kube-operator/raw/branch/main/helm/
|
||||
helm install flink-kube-operator lc-flink-operator/flink-kube-operator
|
||||
```
|
||||
```
|
||||
|
||||
## FlinkJob CRD
|
||||
|
||||
Define Flink jobs as Kubernetes resources:
|
||||
|
||||
```yaml
|
||||
apiVersion: flink.logicamp.dev/v1alpha1
|
||||
kind: FlinkJob
|
||||
metadata:
|
||||
name: my-flink-job
|
||||
spec:
|
||||
key: word-count
|
||||
name: "Word Count Example"
|
||||
entryClass: "com.example.WordCount"
|
||||
parallelism: 1
|
||||
jarRef: "registry.example.com/myorg/word-count:1.0.0"
|
||||
jarPullSecret: "registry-creds"
|
||||
args:
|
||||
kafka-host: kafka:9092
|
||||
kafka-group-id: my-group
|
||||
```
|
||||
|
||||
### Spec Fields
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| `key` | string | yes | Unique identifier for the job |
|
||||
| `name` | string | yes | Human-readable job name |
|
||||
| `entryClass` | string | yes | Flink job entry class |
|
||||
| `parallelism` | int | yes | Job parallelism |
|
||||
| `jarRef` | string | yes | OCI reference to the JAR artifact (e.g. `registry.io/org/app:tag`) |
|
||||
| `jarPullSecret` | string | no | Name of a `kubernetes.io/dockerconfigjson` Secret for registry auth |
|
||||
| `args` | map/list | no | Program arguments (map or list format) |
|
||||
| `savepointInterval` | duration | no | Interval between automatic savepoints |
|
||||
|
||||
### Deprecated Fields (backward compat)
|
||||
|
||||
| Field | Replaced by |
|
||||
|-------|-------------|
|
||||
| `jarUri` | `jarRef` |
|
||||
| `jarURIBasicAuthUsername` | `jarPullSecret` |
|
||||
| `jarURIBasicAuthPassword` | `jarPullSecret` |
|
||||
|
||||
If both `jarRef` and `jarUri` are set, `jarRef` takes precedence.
|
||||
|
||||
## Registry Authentication
|
||||
|
||||
### Kubernetes (pull secret)
|
||||
|
||||
Create a `kubernetes.io/dockerconfigjson` Secret and reference it via `jarPullSecret`:
|
||||
|
||||
```bash
|
||||
kubectl create secret docker-registry registry-creds \
|
||||
--docker-server=registry.example.com \
|
||||
--docker-username=user \
|
||||
--docker-password=pass
|
||||
```
|
||||
|
||||
```yaml
|
||||
spec:
|
||||
jarRef: "registry.example.com/myorg/my-job:1.0.0"
|
||||
jarPullSecret: "registry-creds"
|
||||
```
|
||||
|
||||
### Docker Compose / local dev (environment variables)
|
||||
|
||||
Set registry credentials as environment variables on the operator container:
|
||||
|
||||
```yaml
|
||||
# docker-compose.yaml
|
||||
services:
|
||||
operator:
|
||||
image: flink-kube-operator:latest
|
||||
environment:
|
||||
REGISTRY_SERVER: "registry.example.com"
|
||||
REGISTRY_USERNAME: "user"
|
||||
REGISTRY_PASSWORD: "pass"
|
||||
```
|
||||
|
||||
### Credential Resolution Order
|
||||
|
||||
1. `spec.jarPullSecret` — reads the K8s Secret
|
||||
2. `REGISTRY_USERNAME` + `REGISTRY_PASSWORD` env vars — fallback (optionally scoped by `REGISTRY_SERVER`)
|
||||
3. Anonymous — public registries
|
||||
|
||||
## Pushing JARs to an OCI Registry
|
||||
|
||||
Flink job JARs are stored as OCI artifacts with media type `application/java-archive`. Use one of the methods below to push your JAR.
|
||||
|
||||
### Using crane (recommended)
|
||||
|
||||
[crane](https://github.com/google/go-containerregistry/tree/main/cmd/crane) is a tool for interacting with OCI registries.
|
||||
|
||||
```bash
|
||||
# Install crane
|
||||
go install github.com/google/go-containerregistry/cmd/crane@latest
|
||||
|
||||
# Log in to your registry
|
||||
crane auth login registry.example.com -u user -p pass
|
||||
|
||||
# Push a JAR as an OCI artifact
|
||||
crane append \
|
||||
--file=my-job.jar \
|
||||
--new_tag=registry.example.com/myorg/my-job:1.0.0 \
|
||||
--base=scratch
|
||||
```
|
||||
|
||||
### Using oras
|
||||
|
||||
[oras](https://oras.land/) is a tool for pushing OCI artifacts.
|
||||
|
||||
```bash
|
||||
# Install oras
|
||||
# https://oras.land/docs/installation
|
||||
|
||||
# Log in to your registry
|
||||
oras login registry.example.com -u user -p pass
|
||||
|
||||
# Push a JAR with the correct media type
|
||||
oras push registry.example.com/myorg/my-job:1.0.0 \
|
||||
--artifact-type application/java-archive \
|
||||
my-job.jar:application/java-archive
|
||||
```
|
||||
|
||||
### Using Docker (buildkit)
|
||||
|
||||
```bash
|
||||
# Create a minimal Dockerfile
|
||||
cat > Dockerfile.jar <<'EOF'
|
||||
FROM scratch
|
||||
COPY my-job.jar /my-job.jar
|
||||
EOF
|
||||
|
||||
# Build and push
|
||||
docker build -f Dockerfile.jar -t registry.example.com/myorg/my-job:1.0.0 .
|
||||
docker push registry.example.com/myorg/my-job:1.0.0
|
||||
```
|
||||
|
||||
### GitHub Actions (CI/CD)
|
||||
|
||||
Use [crane](https://github.com/google/go-containerregistry/tree/main/cmd/crane) in your GitHub Actions workflow to push JARs as OCI artifacts. `docker/build-push-action` is not suitable — it builds container images, not arbitrary OCI artifacts.
|
||||
|
||||
```yaml
|
||||
name: Publish JAR
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- 'v*'
|
||||
|
||||
env:
|
||||
REGISTRY: ghcr.io
|
||||
IMAGE_NAME: ${{ github.repository }}/my-job
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Log in to registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ${{ env.REGISTRY }}
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Set up crane
|
||||
uses: imjasonh/setup-crane@v0.4
|
||||
|
||||
- name: Build JAR
|
||||
run: mvn package -DskipTests
|
||||
|
||||
- name: Push JAR to OCI registry
|
||||
run: |
|
||||
crane append \
|
||||
--base=scratch \
|
||||
--new_tag=${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.ref_name }} \
|
||||
--new_layer=target/my-job.jar \
|
||||
--media_type=application/java-archive
|
||||
```
|
||||
|
||||
For GitLab CI, Bitbucket Pipelines, or other CI systems, use the same `crane append` command after authenticating to your registry.
|
||||
|
||||
### Versioning Strategy
|
||||
|
||||
- Use semantic version tags (`:1.0.0`, `:1.1.0`) for explicit version control
|
||||
- Changing the tag in `spec.jarRef` triggers a job upgrade (pause → re-upload → resume from savepoint)
|
||||
- The operator stores the SHA256 digest in `status.runningJarDigest` after each pull
|
||||
|
||||
Reference in New Issue
Block a user