Package io.deephaven.kafka

Class KafkaTools.Produce

java.lang.Object
io.deephaven.kafka.KafkaTools.Produce
Enclosing class:
KafkaTools
public static class KafkaTools.Produce extends Object

  • Field Details

  • Constructor Details

    • Produce

      public Produce()

  • Method Details

    • ignoreSpec

      public static KafkaTools.Produce.KeyOrValueSpec ignoreSpec()
      Spec to explicitly ask one of the "consume" methods to ignore either key or value.

    • simpleSpec

      public static KafkaTools.Produce.KeyOrValueSpec simpleSpec(String columnName)
      A simple spec for sending one column as either key or value in a Kafka message.
      Parameters:
      columnName - The name of the column to include.
      Returns:
      A simple spec for the given input.

    • rawSpec

      public static KafkaTools.Produce.KeyOrValueSpec rawSpec(String columnName, Class<? extends org.apache.kafka.common.serialization.Serializer<?>> serializer)

    • jsonSpec

      public static KafkaTools.Produce.KeyOrValueSpec jsonSpec(String[] includeColumns, Predicate<String> excludeColumns, Map<String,String> columnToFieldMapping, String nestedObjectDelimiter, boolean outputNulls, String timestampFieldName)
      A JSON spec from a set of column names
      Parameters:
      includeColumns - An array with an entry for each column intended to be included in the JSON output. If null, include all columns except those specified in excludeColumns. If includeColumns is not null, excludeColumns should be null.
      excludeColumns - A set specifying column names to omit; can only be used when columnNames is null. In this case all table columns except for the ones in excludeColumns will be included.
      columnToFieldMapping - A map from column name to JSON field name to use for that column. Any column names implied by earlier arguments not included as a key in the map will be mapped to JSON fields of the same name. If null, map all columns to fields of the same name.
      nestedObjectDelimiter - if nested JSON fields are desired, the field separator that is used for the fieldNames parameter, or null for no nesting. For instance, if a particular column should be mapped to JSON field X nested inside field Y, the corresponding field name value for the column key in the columnToFieldMapping map can be the string "X__Y", in which case the value for nestedObjectDelimiter should be {code "_"}
      outputNulls - If false, omit fields with a null value.
      timestampFieldName - If not null, include a field of the given name with a publication timestamp.
      Returns:
      A JSON spec for the given inputs.

    • jsonSpec

      public static KafkaTools.Produce.KeyOrValueSpec jsonSpec(String[] includeColumns, Predicate<String> excludeColumns, Map<String,String> columnToFieldMapping)
      A JSON spec from a set of column names. Shorthand for jsonSpec(columNames, excludeColumns, columnToFieldMapping, null, false, null)
      Parameters:
      includeColumns - An array with an entry for each column intended to be included in the JSON output. If null, include all columns except those specified in excludeColumns. If includeColumns is not null, excludeColumns should be null.
      excludeColumns - A predicate specifying column names to omit; can only be used when columnNames is null. In this case all table columns except for the ones in excludeColumns will be included.
      columnToFieldMapping - A map from column name to JSON field name to use for that column. Any column names implied by earlier arguments not included as a key in the map will be mapped to JSON fields of the same name. If null, map all columns to fields of the same name.
      Returns:
      A JSON spec for the given inputs.

    • avroSpec

      public static KafkaTools.Produce.KeyOrValueSpec avroSpec(org.apache.avro.Schema schema, Map<String,String> fieldToColumnMapping, String timestampFieldName, Predicate<String> includeOnlyColumns, Predicate<String> excludeColumns)
      Avro spec to generate Avro messages from an Avro schema.
      Parameters:
      schema - An Avro schema. The message will implement this schema; all fields will be populated from some table column via explicit or implicit mapping.
      fieldToColumnMapping - A map from Avro schema field name to column name. Any field names not included as a key in the map will be mapped to columns with the same name (unless those columns are filtered out). If null, map all fields to columns of the same name (except for columns filtered out).
      timestampFieldName - If not null, include a field of the given name with a publication timestamp. The field with the given name should exist in the provided schema, and be of logical type timestamp micros.
      includeOnlyColumns - If not null, filter out any columns tested false in this predicate.
      excludeColumns - If not null, filter out any columns tested true in this predicate.
      Returns:
      A spec corresponding to the schema provided.

    • avroSpec

      public static KafkaTools.Produce.KeyOrValueSpec avroSpec(String schemaName, String schemaVersion, Map<String,String> fieldToColumnMapping, String timestampFieldName, Predicate<String> includeOnlyColumns, Predicate<String> excludeColumns, boolean publishSchema, String schemaNamespace, Properties columnProperties)
      Avro spec from fetching an Avro schema from a Confluent compatible Schema Server. The Properties used to initialize Kafka should contain the URL for the Schema Server to use under the "schema.registry.url" property.
      Parameters:
      schemaName - The registered name for the schema on Schema Server
      schemaVersion - The version to fetch. Pass the constant AVRO_LATEST_VERSION for latest
      fieldToColumnMapping - A map from Avro schema field name to column name. Any field names not included as a key in the map will be mapped to columns with the same name. If null, map all fields to columns of the same name.
      timestampFieldName - If not null, include a field of the given name with a publication timestamp. The field with the given name should exist in the provided schema, and be of logical type timestamp micros.
      includeOnlyColumns - If not null, filter out any columns tested false in this predicate.
      excludeColumns - If not null, filter out any columns tested true in this predicate.
      publishSchema - If true, instead of loading a schema already defined in schema server, define a new Avro schema based on the selected columns for this table and publish it to schema server. When publishing, if a schema version is provided and the version generated doesn't match, an exception results.
      schemaNamespace - When publishSchema is true, the namespace for the generated schema to be restered in schema server. When publishSchema is false, null should be passed.
      columnProperties - When publisSchema is true, a Properties object can be provided, specifying String properties implying particular Avro type mappings for them. In particular, column X of BigDecimal type should specify string properties "x.precision" and "x.scale".
      Returns:
      A spec corresponding to the schema provided.