Skip to main content
Version: 0.8.0-incubating

Gravitino credential vending

Background​

Gravitino credential vending is used to generate temporary or static credentials for accessing data. With credential vending, Gravitino provides an unified way to control the access to diverse data sources in different platforms.

Capabilities​

  • Supports Gravitino Iceberg REST server.
  • Supports Gravitino server, only support Hadoop catalog.
  • Supports pluggable credentials with build-in credentials:
    • S3: S3TokenCredential, S3SecretKeyCredential
    • GCS: GCSTokenCredential
    • ADLS: ADLSTokenCredential, AzureAccountKeyCredential
    • OSS: OSSTokenCredential, OSSSecretKeyCredential
  • No support for Spark/Trino/Flink connector yet.

General configurations​

Gravitino server catalog propertiesGravitino Iceberg REST server configurationsDescriptionDefault valueRequiredSince Version
credential-provider-typegravitino.iceberg-rest.credential-provider-typeDeprecated, please use credential-providers instead.(none)Yes0.7.0-incubating
credential-providersgravitino.iceberg-rest.credential-providersThe credential provider types, separated by comma.(none)Yes0.8.0-incubating
credential-cache-expire-ratiogravitino.iceberg-rest.credential-cache-expire-ratioRatio of the credential's expiration time when Gravitino remove credential from the cache.0.15No0.8.0-incubating
credential-cache-max-sizegravitino.iceberg-rest.cache-max-sizeMax size for the credential cache.10000No0.8.0-incubating

Build-in credentials configurations​

S3 credentials​

S3 secret key credential​

A credential with static S3 access key id and secret access key.

Gravitino server catalog propertiesGravitino Iceberg REST server configurationsDescriptionDefault valueRequiredSince Version
credential-providersgravitino.iceberg-rest.credential-providerss3-secret-key for S3 secret key credential provider.(none)Yes0.8.0-incubating
s3-access-key-idgravitino.iceberg-rest.s3-access-key-idThe static access key ID used to access S3 data.(none)Yes0.6.0-incubating
s3-secret-access-keygravitino.iceberg-rest.s3-secret-access-keyThe static secret access key used to access S3 data.(none)Yes0.6.0-incubating

S3 token credential​

An S3 token is a token credential with scoped privileges, by leveraging STS Assume Role. To use an S3 token credential, you should create a role and grant it proper privileges.

Gravitino server catalog propertiesGravitino Iceberg REST server configurationsDescriptionDefault valueRequiredSince Version
credential-providersgravitino.iceberg-rest.credential-providerss3-token for S3 token credential provider.(none)Yes0.8.0-incubating
s3-access-key-idgravitino.iceberg-rest.s3-access-key-idThe static access key ID used to access S3 data.(none)Yes0.6.0-incubating
s3-secret-access-keygravitino.iceberg-rest.s3-secret-access-keyThe static secret access key used to access S3 data.(none)Yes0.6.0-incubating
s3-role-arngravitino.iceberg-rest.s3-role-arnThe ARN of the role to access the S3 data.(none)Yes0.7.0-incubating
s3-regiongravitino.iceberg-rest.s3-regionThe region of the S3 service, like us-west-2.(none)No0.6.0-incubating
s3-external-idgravitino.iceberg-rest.s3-external-idThe S3 external id to generate token.(none)No0.7.0-incubating
s3-token-expire-in-secsgravitino.iceberg-rest.s3-token-expire-in-secsThe S3 session token expire time in secs, it couldn't exceed the max session time of the assumed role.3600No0.7.0-incubating
s3-token-service-endpointgravitino.iceberg-rest.s3-token-service-endpointAn alternative endpoint of the S3 token service, This could be used with s3-compatible object storage service like MINIO that has a different STS endpoint.(none)No0.8.0-incubating

OSS credentials​

OSS secret key credential​

A credential with static OSS access key id and secret access key.

Gravitino server catalog propertiesGravitino Iceberg REST server configurationsDescriptionDefault valueRequiredSince Version
credential-providersgravitino.iceberg-rest.credential-providersoss-secret-key for OSS secret credential.(none)Yes0.8.0-incubating
oss-access-key-idgravitino.iceberg-rest.oss-access-key-idThe static access key ID used to access OSS data.(none)Yes0.7.0-incubating
oss-secret-access-keygravitino.iceberg-rest.oss-secret-access-keyThe static secret access key used to access OSS data.(none)Yes0.7.0-incubating

OSS token credential​

An OSS token is a token credential with scoped privileges, by leveraging STS Assume Role. To use an OSS token credential, you should create a role and grant it proper privileges.

Gravitino server catalog propertiesGravitino Iceberg REST server configurationsDescriptionDefault valueRequiredSince Version
credential-providersgravitino.iceberg-rest.credential-providersoss-token for s3 token credential.(none)Yes0.8.0-incubating
oss-access-key-idgravitino.iceberg-rest.oss-access-key-idThe static access key ID used to access OSS data.(none)Yes0.7.0-incubating
oss-secret-access-keygravitino.iceberg-rest.oss-secret-access-keyThe static secret access key used to access OSS data.(none)Yes0.7.0-incubating
oss-role-arngravitino.iceberg-rest.oss-role-arnThe ARN of the role to access the OSS data.(none)Yes0.8.0-incubating
oss-regiongravitino.iceberg-rest.oss-regionThe region of the OSS service, like oss-cn-hangzhou, only used when credential-providers is oss-token.(none)No0.8.0-incubating
oss-external-idgravitino.iceberg-rest.oss-external-idThe OSS external id to generate token.(none)No0.8.0-incubating
oss-token-expire-in-secsgravitino.iceberg-rest.oss-token-expire-in-secsThe OSS security token expire time in secs.3600No0.8.0-incubating

ADLS credentials​

Azure account key credential​

A credential with static Azure storage account name and key.

Gravitino server catalog propertiesGravitino Iceberg REST server configurationsDescriptionDefault valueRequiredSince Version
credential-providersgravitino.iceberg-rest.credential-providersazure-account-key for Azure account key credential.(none)Yes0.8.0-incubating
azure-storage-account-namegravitino.iceberg-rest.azure-storage-account-nameThe static storage account name used to access ADLS data.(none)Yes0.8.0-incubating
azure-storage-account-keygravitino.iceberg-rest.azure-storage-account-keyThe static storage account key used to access ADLS data.(none)Yes0.8.0-incubating

ADLS token credential​

An ADLS token is a token credential with scoped privileges, by leveraging Azure User Delegation Sas. To use an ADLS token credential, you should create a Microsoft Entra ID service principal and grant it proper privileges.

Gravitino server catalog propertiesGravitino Iceberg REST server configurationsDescriptionDefault valueRequiredSince Version
credential-providersgravitino.iceberg-rest.credential-providersadls-token for ADLS token credential.(none)Yes0.8.0-incubating
azure-storage-account-namegravitino.iceberg-rest.azure-storage-account-nameThe static storage account name used to access ADLS data.(none)Yes0.8.0-incubating
azure-storage-account-keygravitino.iceberg-rest.azure-storage-account-keyThe static storage account key used to access ADLS data.(none)Yes0.8.0-incubating
azure-tenant-idgravitino.iceberg-rest.azure-tenant-idAzure Active Directory (AAD) tenant ID.(none)Yes0.8.0-incubating
azure-client-idgravitino.iceberg-rest.azure-client-idAzure Active Directory (AAD) client ID used for authentication.(none)Yes0.8.0-incubating
azure-client-secretgravitino.iceberg-rest.azure-client-secretAzure Active Directory (AAD) client secret used for authentication.(none)Yes0.8.0-incubating
adls-token-expire-in-secsgravitino.iceberg-rest.adls-token-expire-in-secsThe ADLS SAS token expire time in secs.3600No0.8.0-incubating

GCS credentials​

GCS token credential​

An GCS token is a token credential with scoped privileges, by leveraging GCS Credential Access Boundaries. To use an GCS token credential, you should create an GCS service account and grant it proper privileges.

Gravitino server catalog propertiesGravitino Iceberg REST server configurationsDescriptionDefault valueRequiredSince Version
credential-providersgravitino.iceberg-rest.credential-providersgcs-token for GCS token credential.(none)Yes0.8.0-incubating
gcs-credential-file-pathgravitino.iceberg-rest.gcs-credential-file-pathDeprecated, please use gcs-service-account-file instead.GCS Application default credential.No0.7.0-incubating
gcs-service-account-filegravitino.iceberg-rest.gcs-service-account-fileThe location of GCS credential file.GCS Application default credential.No0.8.0-incubating
note

For Gravitino Iceberg REST server, please ensure that the credential file can be accessed by the server. For example, if the server is running on a GCE machine, or you can set the environment variable as export GOOGLE_APPLICATION_CREDENTIALS=/xx/application_default_credentials.json, even when the gcs-service-account-file has already been configured.

Custom credentials​

Gravitino supports custom credentials, you can implement the org.apache.gravitino.credential.CredentialProvider interface to support custom credentials, and place the corresponding jar to the classpath of Iceberg catalog server or Hadoop catalog.

Deployment​

Besides setting credentials related configuration, please download Gravitino cloud bundle jar and place it in the classpath of Iceberg REST server or Hadoop catalog.

For Hadoop catalog, please use Gravitino cloud bundle jar with Hadoop and cloud packages:

For Iceberg REST catalog server, please use Gravitino cloud bundle jar without Hadoop and cloud packages:

note

For OSS, Iceberg doesn't provide Iceberg Aliyun bundle jar which contains OSS packages, you could provide the OSS jar by yourself or use Gravitino Aliyun bundle jar with Hadoop and cloud packages, please refer to OSS configuration for more details.

The classpath of the server:

  • Iceberg REST server: the classpath differs in different deploy mode, please refer to Server management part.
  • Hadoop catalog: catalogs/hadoop/libs/

Usage example​

Credential vending for Iceberg REST server​

Suppose the Iceberg table data is stored in S3, follow the steps below:

  1. Download the Gravitino AWS bundle jar without hadoop packages, and place it to the classpath of Iceberg REST server.

  2. Add s3 token credential configurations.

gravitino.iceberg-rest.warehouse = s3://{bucket_name}/{warehouse_path}
gravitino.iceberg-rest.io-impl= org.apache.iceberg.aws.s3.S3FileIO
gravitino.iceberg-rest.credential-providers = s3-token
gravitino.iceberg-rest.s3-access-key-id = xxx
gravitino.iceberg-rest.s3-secret-access-key = xxx
gravitino.iceberg-rest.s3-region = {region_name}
gravitino.iceberg-rest.s3-role-arn = {role_arn}
  1. Exploring the Iceberg table with Spark client with credential vending enabled.
./bin/spark-sql -v \
--packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.3.1 \
--conf spark.jars={path}/iceberg-aws-bundle-1.5.2.jar \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.rest=org.apache.iceberg.spark.SparkCatalog  \
--conf spark.sql.catalog.rest.type=rest  \
--conf spark.sql.catalog.rest.uri=http://127.0.0.1:9001/iceberg/ \
--conf spark.sql.catalog.rest.header.X-Iceberg-Access-Delegation=vended-credentials