Configuring the Jet Engine

Data Pipelines allow you to process data stored in one location and send the result to another, such as from a data lake to an analytics database or into a payment processing system. You can also use the same source and sink so the pipeline only processes data. You can configure Hazelcast's Jet engine using the Hazelcast Platform Operator.

Configuring the Jet Engine

The Jet engine is enabled by default and has the following configuration options. For more detailed information, see the API Reference.

Option	Default	Description
enabled	true	Set to false to disable the Jet engine.
resourceUploadEnabled	false	Set to true to enable the uploading of resources for Jet jobs.
instance/cooperativeThreadCount		The number of threads Jet creates in its cooperative multithreading pool.
instance/flowControlPeriodMS	100	The duration of the interval between flow-control packets.
instance/backupCount	1	The number of synchronous backups to configure on the IMap, which Jet needs internally to store job metadata and snapshots.
instance/scaleUpDelayMS	10000	The delay after which the auto-scaled jobs restart if a new member joins the cluster.
instance/losslessRestartEnabled	false	Whether the Lossless Cluster Restart feature is enabled. warning Lossless Restart requires Persistence to be enabled. For more info about Persistence, see Persistence, Backup, and Restore.
instance/maxProcessorAccumulatedRecords		The maximum number of records that can be accumulated by any single processor instance.
edgeDefaults/queueSize		Sets the capacity of processor-to-processor concurrent queues.
edgeDefaults/packetSizeLimit		Limits the size of the packet in bytes.
edgeDefaults/receiveWindowMultiplier		Sets the scaling factor used by the adaptive receive window sizing function.
bucketConfig		JAR files from an external bucket are placed accessible for the member when the following parameter values are supplied: secretName: Name of the Secret object which holds the credentials for your cloud provider. bucketURI: Full path for the external bucket. For example: gs://your-bucket/path/to/jars.
configMaps		List of names of ConfigMaps. Files in each ConfigMap will be downloaded.
remoteURLs		List of URLs from where the files will be downloaded.

Example configuration

yaml

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  version: '5.6.0-slim'
  licenseKeySecretName: hazelcast-license-key
  jet:
    enabled: true
    resourceUploadEnabled: true
    instance:
      cooperativeThreadCount: 4
      flowControlPeriodMillis: 100
      backupCount: 1
      scaleUpDelayMillis: 10000
      losslessRestartEnabled: false
      maxProcessorAccumulatedRecords: 1000000000
    edgeDefaults:
      queueSize: 1024
      packetSizeLimit: 16384
      receiveWindowMultiplier: 3

Configuration Properties

You can set configuration properties as Java system properties. See Setting Arbitrary JVM Arguments for more information.

Example configuration

yaml

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  version: '5.6.0-slim'
  licenseKeySecretName: hazelcast-license-key
  jet:
    enabled: true
  jvm:
    args:
    - "-Dhazelcast.jet.idle.cooperative.min.microseconds=50"