An application to execute Flink SQL jobs.

1. Build application

   mvn package
   

2. Build an image

   minikube image build . -t flink-sql-runner:latest
   

3. Create a flink namespace:

   kubectl create namespace flink
   

4. Follow the steps in Deploying the operator section of the Flink Kubernetes Operator's Quick Start.

5. Install the required RBAC rules for Flink Job:

   kubectl create -f install/
   

6. Create a Flink job using the FlinkDeployment.yaml example. Supply your SQL statements to the job by replacing <SQL_STATEMENTS>. You can use Kubernetes secrets with Flink SQL Runner, to provide security credentials to Flink job for connecting to the source or the target systems. Secrets can be directly templated in the SQL statements with the following pattern:

   {{secret:<NAMESPACE>/<SECRET NAME>/<DATA KEY>}}
   

   If running with the local image built in step 2, ensure that the FlinkDeployment's image is updated.

7. Start a Flink job:

   kubectl create example/FlinkDeployment.yaml -n flink
   

   Update example/FlinkDeployment.yaml with your own SQL statements e.g. args: ["<SQL_STATEMENTS>"]. Note that semicolon ; is a special character used as a statement delimiter. If it's part of your SQL statements, make sure it is escaped by \\. For example, it might be used for properties.sasl.jaas.config value when using Kafka connector. In this case, it would look something like this:

   'properties.sasl.jaas.config' = 'org.apache.flink.kafka.shaded.org.apache.kafka.common.security.plain.PlainLoginModule required username=\"test-user\" password=\"{{secret:flink/test-user/user.password}}\"\\;'