Skip to main content
Version: 1.3.1

Fileset Catalog with S3

Introduction​

This page shows how to store fileset data in Amazon S3 while Gravitino manages the metadata, and how to read and write that data through the Gravitino Virtual File System (GVFS).

Everything on this page is specific to Amazon S3. The fileset model itself, the properties shared by every storage backend, and the way properties are inherited from catalog to schema to fileset are described in Fileset Catalog.

The examples run in order and use the same names throughout: metalake metalake, catalog s3_catalog, schema s3_schema, fileset example_fileset, and http://localhost:8090 as the server URL. Replace them with your own values.

Prerequisites​

  1. Download the gravitino-aws-bundle-${gravitino-version}.jar file.
  2. Place it in the fileset catalog classpath at ${GRAVITINO_HOME}/catalogs/fileset/libs/.
  3. Start the Gravitino server:
${GRAVITINO_HOME}/bin/gravitino-server.sh start

The catalog automatically loads the Amazon S3 filesystem provider once the bundle jar is on the classpath. The deprecated filesystem-providers and default-filesystem-provider catalog properties do not need to be set.

The bundle uses JSSE for TLS and does not include the optional WildFly OpenSSL provider. If you explicitly set Hadoop's fs.s3a.ssl.channel.mode to OpenSSL, install a compatible provider separately on the catalog or client classpath. The default mode does not require it.

Amazon S3 Properties​

These properties are needed in addition to the shared catalog properties. The same values are also needed by the GVFS clients, so they are listed together here — note that the Python client spells them with underscores while the catalog and the Java client use hyphens.

Catalog and Java clientPython clientDescriptionRequired
s3-endpoints3_endpointEndpoint of the S3 service. S3-compatible storage such as MinIO always needs it.Yes, except for the Python client against AWS S3
s3-access-key-ids3_access_key_idAccess key of the S3 service.Yes
s3-secret-access-keys3_secret_access_keySecret key of the S3 service.Yes
credential-providers(n/a)The credential provider types, separated by comma. Possible values are s3-token, s3-secret-key, aws-irsa. Setting it enables credential vending, so clients no longer need the credentials above. See credential vending for the extra properties each provider takes.No
note
  • The location must start with s3a://, not s3://. The hadoop-aws library does not support the s3:// scheme.
  • For MinIO and other S3-compatible services, set s3-endpoint to that service. If it requires path-style access, add gravitino.bypass.fs.s3a.path.style.access=true to ${GRAVITINO_HOME}/catalogs/fileset/conf/fileset.conf for server-side operations. Also set s3-path-style-access=true on a GVFS Java client, or spark.hadoop.s3-path-style-access=true in Spark. For the Python GVFS client, pass config_kwargs={"s3": {"addressing_style": "path"}} to GravitinoVirtualFileSystem; for pandas, add the same config_kwargs entry at the top level of storage_options.

Schema and fileset properties are documented on the shared page: see schema properties and fileset properties.

A fileset catalog stores its data under location, which for Amazon S3 looks like s3a://bucket/root.

Create the Catalog, Schema, and Fileset​

Step 1: Create the catalog​

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "s3_catalog",
  "type": "FILESET",
  "comment": "A fileset catalog backed by Amazon S3",
  "properties": {
    "location": "s3a://bucket/root",
    "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com",
    "s3-access-key-id": "access_key",
    "s3-secret-access-key": "secret_key"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs

Step 2: Create the schema​

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "s3_schema",
  "comment": "A schema in the Amazon S3 fileset catalog",
  "properties": {
    "location": "s3a://bucket/root/schema"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs/s3_catalog/schemas

Step 3: Create the fileset​

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "example_fileset",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "storageLocation": "s3a://bucket/root/schema/example_fileset",
  "properties": {
    "k1": "v1"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs/s3_catalog/schemas/s3_schema/filesets

The fileset is now addressable as gvfs://fileset/s3_catalog/s3_schema/example_fileset from any GVFS client.

Access the Fileset​

Java client jars​

Every Java or Hadoop-based client needs gravitino-filesystem-hadoop3-runtime, which is published on Maven Central, plus the Amazon S3 filesystem implementation. Only the latter differs by environment:

EnvironmentJar providing the Amazon S3 filesystem
No Hadoop installedgravitino-aws-bundle, a fat jar bundling the Amazon S3 filesystem implementation and the AWS SDK
Hadoop already presenthadoop-aws-${hadoop-version}.jar and aws-java-sdk-bundle-1.12.262.jar, shipped with Hadoop under ${HADOOP_HOME}/share/hadoop/tools/lib

The artifacts in full:

  • gravitino-aws-bundle-${gravitino-version}.jar: a "fat" jar that includes the gravitino-aws functionality together with every dependency it needs, such as hadoop-aws and the AWS SDK. Use it when the environment has no pre-existing Hadoop setup.
  • gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar: a "fat" jar that bundles the Gravitino virtual filesystem client and already includes the gravitino-aws functionality. Java and Hadoop-based clients require it to access Gravitino filesets.
  • hadoop-aws-${hadoop-version}.jar and aws-java-sdk-bundle-1.12.262.jar: the standard Hadoop dependencies for Amazon S3 access, shipped with Hadoop under ${HADOOP_HOME}/share/hadoop/tools/lib. Supply them yourself when running inside an existing Hadoop environment.
  • gravitino-aws-${gravitino-version}.jar: a "thin" jar carrying only the AWS integration code. It is already contained in both jars above, so it is not needed as a direct dependency unless you prefer to manage all Hadoop and AWS dependencies yourself.
<!-- No Hadoop environment -->
<dependency>
  <groupId>org.apache.gravitino</groupId>
  <artifactId>gravitino-aws-bundle</artifactId>
  <version>${GRAVITINO_VERSION}</version>
</dependency>
<dependency>
  <groupId>org.apache.gravitino</groupId>
  <artifactId>gravitino-filesystem-hadoop3-runtime</artifactId>
  <version>${GRAVITINO_VERSION}</version>
</dependency>
<!-- Existing Hadoop environment -->
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-common</artifactId>
  <version>${HADOOP_VERSION}</version>
</dependency>
<dependency>
  <groupId>org.apache.hadoop</groupId>
  <artifactId>hadoop-aws</artifactId>
  <version>${HADOOP_VERSION}</version>
</dependency>
<dependency>
  <groupId>org.apache.gravitino</groupId>
  <artifactId>gravitino-filesystem-hadoop3-runtime</artifactId>
  <version>${GRAVITINO_VERSION}</version>
</dependency>
note

The thin gravitino-aws jar is not needed. Its functionality is already included in both gravitino-aws-bundle and gravitino-filesystem-hadoop3-runtime.

GVFS Java client​

On top of the base GVFS configuration, set the Amazon S3 properties from the table above.

Configuration conf = new Configuration();
conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs");
conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem");
conf.set("fs.gravitino.server.uri", "http://localhost:8090");
conf.set("fs.gravitino.client.metalake", "metalake");
conf.set("s3-endpoint", "http://s3.ap-northeast-1.amazonaws.com");
conf.set("s3-access-key-id", "access_key");
conf.set("s3-secret-access-key", "secret_key");

Path filesetPath = new Path("gvfs://fileset/s3_catalog/s3_schema/example_fileset/new_dir");
FileSystem fs = filesetPath.getFileSystem(conf);
fs.mkdirs(filesetPath);

Apache Spark​

The example below uses PySpark 3.5.0 in an environment that already has Hadoop 3.3.4.

pip install pyspark==3.5.0
pip install apache-gravitino==${GRAVITINO_VERSION}
import os
from pyspark.sql import SparkSession

# On JDK 17, also add:
#   --conf "spark.driver.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
#   --conf "spark.executor.extraJavaOptions=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
os.environ["PYSPARK_SUBMIT_ARGS"] = (
    "--jars /path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar,"
    "/path/to/hadoop-aws-3.3.4.jar,"
    "/path/to/aws-java-sdk-bundle-1.12.262.jar "
    "--master local[1] pyspark-shell"
)

spark = (SparkSession.builder
    .appName("s3_fileset")
    .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs")
    .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem")
    .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090")
    .config("spark.hadoop.fs.gravitino.client.metalake", "metalake")
    .config("spark.hadoop.s3-endpoint", "http://s3.ap-northeast-1.amazonaws.com")
    .config("spark.hadoop.s3-access-key-id", "access_key")
    .config("spark.hadoop.s3-secret-access-key", "secret_key")
    .config("spark.driver.memory", "2g")
    .config("spark.driver.port", "2048")
    .getOrCreate())

data = [("Alice", 25), ("Bob", 30), ("Cathy", 45)]
spark_df = spark.createDataFrame(data, schema=["Name", "Age"])
gvfs_path = "gvfs://fileset/s3_catalog/s3_schema/example_fileset/people"

spark_df.coalesce(1).write.mode("overwrite").option("header", "true").csv(gvfs_path)

If Spark runs without a Hadoop environment, only the jar list changes:

os.environ["PYSPARK_SUBMIT_ARGS"] = (
    "--jars /path/to/gravitino-aws-bundle-${gravitino-version}.jar,"
    "/path/to/gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar "
    "--master local[1] pyspark-shell"
)
note

Some Spark versions need a Hadoop environment in the driver and do not pick up filesystem implementations passed with --jars. If that happens, add the jars to the Spark classpath directly.

Hadoop fs command​

  1. Add the following to ${HADOOP_HOME}/etc/hadoop/core-site.xml:
<property>
  <name>fs.AbstractFileSystem.gvfs.impl</name>
  <value>org.apache.gravitino.filesystem.hadoop.Gvfs</value>
</property>
<property>
  <name>fs.gvfs.impl</name>
  <value>org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem</value>
</property>
<property>
  <name>fs.gravitino.server.uri</name>
  <value>http://localhost:8090</value>
</property>
<property>
  <name>fs.gravitino.client.metalake</name>
  <value>metalake</value>
</property>
<property>
  <name>s3-endpoint</name>
  <value>http://s3.ap-northeast-1.amazonaws.com</value>
</property>
<property>
  <name>s3-access-key-id</name>
  <value>access_key</value>
</property>
<property>
  <name>s3-secret-access-key</name>
  <value>secret_key</value>
</property>

  1. Add these jars to the Hadoop classpath:

    • gravitino-filesystem-hadoop3-runtime-${gravitino-version}.jar, from Maven Central.
    • hadoop-aws-${hadoop-version}.jar and aws-java-sdk-bundle-1.12.262.jar, shipped with Hadoop under ${HADOOP_HOME}/share/hadoop/tools/lib.

  2. Access the fileset:

${HADOOP_HOME}/bin/hadoop fs -ls gvfs://fileset/s3_catalog/s3_schema/example_fileset
${HADOOP_HOME}/bin/hadoop fs -put /path/to/local/file gvfs://fileset/s3_catalog/s3_schema/example_fileset

GVFS Python client​

pip install apache-gravitino==${GRAVITINO_VERSION}

On top of the base GVFS configuration, pass the Amazon S3 properties in options, spelled with underscores.

from gravitino import gvfs

options = {
    "cache_size": 20,
    "cache_expired_time": 3600,
    "auth_type": "simple",
    "s3_endpoint": "http://s3.ap-northeast-1.amazonaws.com",
    "s3_access_key_id": "access_key",
    "s3_secret_access_key": "secret_key",
}

fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090",
                                     metalake_name="metalake",
                                     options=options)
fs.ls("gvfs://fileset/s3_catalog/s3_schema/example_fileset/")

pandas​

pandas reaches the same paths through storage_options. Use the fs instance from the preceding GVFS example to discover the generated Spark part file.

import pandas as pd

storage_options = {
    "server_uri": "http://localhost:8090",
    "metalake_name": "metalake",
    "options": {
        "s3_endpoint": "http://s3.ap-northeast-1.amazonaws.com",
        "s3_access_key_id": "access_key",
        "s3_secret_access_key": "secret_key",
    }
}

csv_path = next(
    f"gvfs://{path}"
    for path in fs.ls(
        "gvfs://fileset/s3_catalog/s3_schema/example_fileset/people",
        detail=False,
    )
    if (
        path.rsplit("/", 1)[-1].startswith("part-")
        and path.endswith(".csv")
    )
)
ds = pd.read_csv(csv_path, storage_options=storage_options)
ds.head()

For further use cases, see Gravitino Virtual File System.

Credential Vending​

With credential vending the catalog holds the Amazon S3 credentials and the Gravitino server hands out a credential per request, so clients never hold cloud keys of their own. See Credential Vending for the general mechanism and S3 credentials for the properties each provider takes.

The supported providers are s3-token, which vends a short-lived STS token; s3-secret-key, which vends the static access key configured on the catalog; and aws-irsa, which vends credentials from an IAM role for service accounts and currently reads the web identity token from a file. The example below uses s3-token.

Configure the catalog, schema, and fileset​

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "s3_catalog_with_vending",
  "type": "FILESET",
  "comment": "A fileset catalog backed by Amazon S3 with credential vending",
  "properties": {
    "location": "s3a://bucket/root",
    "s3-endpoint": "http://s3.ap-northeast-1.amazonaws.com",
    "s3-access-key-id": "access_key",
    "s3-secret-access-key": "secret_key",
    "credential-providers": "s3-token",
    "s3-region": "ap-northeast-1",
    "s3-role-arn": "arn:aws:iam::123456789012:role/gravitino-fileset"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs

Create the schema and fileset in the credential-vending catalog:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "s3_schema",
  "comment": "A schema in the Amazon S3 credential-vending catalog",
  "properties": {
    "location": "s3a://bucket/root/schema"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs/s3_catalog_with_vending/schemas

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "example_fileset",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "storageLocation": "s3a://bucket/root/schema/example_fileset",
  "properties": {}
}' http://localhost:8090/api/metalakes/metalake/catalogs/s3_catalog_with_vending/schemas/s3_schema/filesets

The s3-token provider needs two more catalog properties.

Property NameDescription
s3-regionRegion of the bucket, for example ap-northeast-1
s3-role-arnARN of the role that grants access to the data

Access without local credentials​

Enable vending on the client and drop the credential properties.

Configuration conf = new Configuration();
conf.setBoolean("fs.gravitino.enableCredentialVending", true);
conf.set("fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs");
conf.set("fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem");
conf.set("fs.gravitino.server.uri", "http://localhost:8090");
conf.set("fs.gravitino.client.metalake", "metalake");
// No need to set s3-access-key-id or s3-secret-access-key

Path filesetPath = new Path(
    "gvfs://fileset/s3_catalog_with_vending/s3_schema/example_fileset/new_dir");
FileSystem fs = filesetPath.getFileSystem(conf);
fs.mkdirs(filesetPath);
spark = (SparkSession.builder
    .appName("s3_fileset")
    .config("spark.hadoop.fs.gravitino.enableCredentialVending", "true")
    .config("spark.hadoop.fs.AbstractFileSystem.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.Gvfs")
    .config("spark.hadoop.fs.gvfs.impl", "org.apache.gravitino.filesystem.hadoop.GravitinoVirtualFileSystem")
    .config("spark.hadoop.fs.gravitino.server.uri", "http://localhost:8090")
    .config("spark.hadoop.fs.gravitino.client.metalake", "metalake")
    # No need to set s3-access-key-id or s3-secret-access-key
    .getOrCreate())
options = {
    "auth_type": "simple",
    "enable_credential_vending": True,
    # No need to set s3-access-key-id or s3-secret-access-key
}
fs = gvfs.GravitinoVirtualFileSystem(server_uri="http://localhost:8090",
                                     metalake_name="metalake",
                                     options=options)