7,943 questions
0
votes
0
answers
22
views
The library registration references a different set of library BLOBs than previous registrations for this job
Context : Flink Deployment Application cluster Mode
Usecase : Flink Job to read data from kafka , transform and produce to kafka
Deployment : On k8s cluster using k8s Flink operator
To start with ...
- 329
0
votes
0
answers
18
views
Could not specify connectTimeout for Influx Metrics reporter in config.yaml
InfluxDB reporter is defined as in Fink config.yaml.
metrics:
reporter:
influxdb:
factory:
class: org.apache.flink.metrics.influxdb.InfluxdbReporterFactory
schema: http
...
0
votes
0
answers
20
views
The DB Hhstory topic or its content is fully or partially missing
Flink CDC Oracle Task Fails with DebeziumException:
The db history topic is missing
I have set up Flink and used it to synchronize Oracle data to Doris. Now I’m testing multiple Oracle ...
- 1
0
votes
1
answer
17
views
How to Log State Access (get/put) in Flink SQL Join Operators with Operator Metadata?
I'm using Flink SQL to join data from multiple Kafka topics. Sometimes the resulting join is non-trivial to debug, and I want to log the state access involved in the join — specifically, I’d like to ...
- 347
1
vote
0
answers
29
views
Flink application fails when RocksDB Value State is cleaned by TTL
flink version 1.17.0.
I have two message streams A and B that consume from Kafka. The volume is very large. I need to store these two streams in rocksdb for association. It runs normally, but every ...
- 35
0
votes
0
answers
58
views
How do i pass Program Arguments From Apache Flink 2.0.0 Web Gui to my Job properly?
I need to submit a new Apache Flink Job from the Web GUI instanced in a docker container in Session Mode and i need to pass some arguments to the main function of my job written in Java.
I'm trying to ...
0
votes
0
answers
31
views
Failed authentication due to: SSL handshake
Failure : failed authentication due to: SSL handshake failed with strimzi kafka
Context : Flink App deployed on kubernetes cluster reading data from strimzi kafka and remote writing to cloud MSK.
...
- 329
0
votes
0
answers
23
views
PyFlink MiniCluster “Could not find a file system implementation for scheme ‘s3a’” when writing filesystem sink to S3/MinIO
I’m running PyFlink 1.19.1 inside a Jupyter notebook on a local MiniCluster and trying to write a filesystem sink to an S3-compatible MinIO endpoint. When I submit the same job (same JARs, same ...
- 33
0
votes
1
answer
39
views
Flink SQL integration with Iceberg Catalog backed by S3-Minio Storage [closed]
I am trying to setup a POC for Apache Flink integrated with Iceberg Catalog with S3-Minio being used as a backend storage for Iceberg (for storing table data).
I am using following docker-compose,
...
0
votes
0
answers
26
views
Flink CDC Pipeline - OutOfMemoryError: Direct buffer memory
I am trying to replicate a MySQL DB (~500GB size, incl. one large table w/ ~5 billion rows) into StarRocks using an Apache Flink CDC pipeline. This on a local cluster w/ everything in podman ...
- 173
0
votes
0
answers
31
views
Flink Kafka Source OOM
java.lang.RuntimeException: One or more fetchers have encountered exception
at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:225)
...
2
votes
0
answers
51
views
Flink FlatMap "Could not forward element to next operator"
I'm working on a Flink batch job that reads JSON strings from Kafka, converts them into Avro GenericRecords, and writes them to Parquet using AvroParquetWriters.
The JSON messages look like this:
{&...
- 41
0
votes
1
answer
60
views
Error in a flink job `No FileSystem for scheme "s3"`
I get this error No FileSystem for scheme "s3" when I run my test.
Container.ExecResult(exitCode=1, stdout=, stderr=WARNING: An illegal reflective access operation has occurred
WARNING: ...
- 1,145
0
votes
1
answer
30
views
Apache Flink: Postgres JAR not found
The issue
I am creating a CDC pipeline in Flink to replicate a Postgres database into a StarRocks database. The JobManager and the TaskManager are up and running. But no matter how I try to start the ...
- 173
2
votes
1
answer
44
views
Files stuck as .inprogress, not rolling into final Parquet files
I'm running a Flink streaming job using the Table API, which reads from Kafka and writes to S3 (for now I'm using a local path to simulate S3). The job uses a filesystem connector to write data in ...
- 41