Package io.deephaven.extensions.s3

Class S3Instructions

java.lang.Object
io.deephaven.extensions.s3.S3Instructions
All Implemented Interfaces:
LogOutputAppendable
@Immutable public abstract class S3Instructions extends Object implements LogOutputAppendable
This class provides instructions intended for reading from and writing to S3-compatible APIs. The default values documented in this class may change in the future. As such, callers may wish to explicitly set the values.

  • Constructor Details

    • S3Instructions

      public S3Instructions()

  • Method Details

    • builder

      public static S3Instructions.Builder builder()

    • regionName

      public abstract Optional<String> regionName()
      The region name to use when reading or writing to S3. If not provided, the region name is picked by the AWS SDK from 'aws.region' system property, "AWS_REGION" environment variable, the {user.home}/.aws/credentials or {user.home}/.aws/config files, or from EC2 metadata service, if running in EC2. If no region name is derived from the above chain or derived the region name derived is incorrect for the bucket accessed, the correct region name will be derived internally, at the cost of one additional request.

    • maxConcurrentRequests

      @Default public int maxConcurrentRequests()
      The maximum number of concurrent requests to make to S3, defaults to 256.

    • readAheadCount

      @Default public int readAheadCount()
      The number of fragments to send asynchronous read requests for while reading the current fragment. Defaults to 32, which means by default, we will fetch 32 fragments in advance when reading current fragment.

    • fragmentSize

      @Default public int fragmentSize()
      The maximum byte size of each fragment to read from S3 in bytes, defaults to 65536, must be larger than 8192. If there are fewer bytes remaining in the file, the fetched fragment can be smaller.

    • connectionTimeout

      @Default public Duration connectionTimeout()
      The amount of time to wait when initially establishing a connection before giving up and timing out, defaults to 2 seconds.

    • readTimeout

      @Default public Duration readTimeout()
      The amount of time to wait when reading a fragment before giving up and timing out, defaults to 2 seconds. The implementation may choose to internally retry the request multiple times, so long as the total time does not exceed this timeout.

    • credentials

      @Default public Credentials credentials()
      The credentials to use when reading or writing to S3. By default, uses Credentials.resolving().

    • writePartSize

      @Default public int writePartSize()
      The size of each part (in bytes) to upload when writing to S3, defaults to 10485760. The minimum allowed part size is 5242880. Setting a higher value may increase throughput, but may also increase memory usage. Note that the maximum number of parts allowed for a single file is 10,000. Therefore, for 10485760 part size, the maximum size of a single file that can be written is 10485760 * 10,000 bytes.

    • numConcurrentWriteParts

      @Default public int numConcurrentWriteParts()
      The maximum number of parts that can be uploaded concurrently when writing to S3 without blocking. Setting a higher value may increase throughput, but may also increase memory usage. Defaults to 64.

    • profileName

      public abstract Optional<String> profileName()
      The default profile name used for configuring the default region, credentials, etc., when reading or writing to S3. If not provided, the AWS SDK picks the profile name from the 'aws.profile' system property, the "AWS_PROFILE" environment variable, or defaults to "default".

      Setting a profile name assumes that the credentials are provided via this profile; if that is not the case, you must explicitly set credentials.

      See Also:
      • ClientOverrideConfiguration.Builder.defaultProfileName(String)

    • configFilePath

      public abstract Optional<Path> configFilePath()
      The path to the configuration file to use for configuring the default region, credentials, etc. when reading or writing to S3. If not provided, the AWS SDK picks the configuration file from the 'aws.configFile' system property, the "AWS_CONFIG_FILE" environment variable, or defaults to "{user.home}/.aws/config".

      Setting a configuration file path assumes that the credentials are provided via the configuration and credentials files; if that is not the case, you must explicitly set credentials.

      See Also:
      • ClientOverrideConfiguration.Builder.defaultProfileFile(ProfileFile)

    • credentialsFilePath

      public abstract Optional<Path> credentialsFilePath()
      The path to the credentials file to use for configuring the default region, credentials, etc. when reading or writing to S3. If not provided, the AWS SDK picks the credentials file from the 'aws.credentialsFile' system property, the "AWS_CREDENTIALS_FILE" environment variable, or defaults to "{user.home}/.aws/credentials".

      Setting a credentials file path assumes that the credentials are provided via the config and credentials files; if that is not the case, you must explicitly set credentials.

      See Also:
      • ClientOverrideConfiguration.Builder.defaultProfileFile(ProfileFile)

    • append

      public LogOutput append(LogOutput logOutput)
      Specified by:
      append in interface LogOutputAppendable

    • endpointOverride

      public abstract Optional<URI> endpointOverride()
      The endpoint to connect to. Callers connecting to AWS do not typically need to set this; it is most useful when connecting to non-AWS, S3-compatible APIs.
      See Also:

    • withEndpointOverride

      public abstract S3Instructions withEndpointOverride(URI endpointOverride)

    • withRegionName

      @VisibleForTesting public abstract S3Instructions withRegionName(Optional<String> regionName)