Manage Fileset Metadata

Introduction

This page introduces how to manage fileset metadata in Apache Gravitino. Filesets are a collection of files and directories. Users can leverage filesets to manage non-tabular data like training datasets and other raw data.

Typically, a fileset is mapped to a directory on a file system like HDFS, S3, ADLS, GCS, etc. With the fileset managed by Gravitino, the non-tabular data can be managed as assets together with tabular data in Gravitino in a unified way. The following operations will use HDFS as an example, for other HCFS like S3, OSS, GCS, etc., refer to the corresponding operations fileset-with-s3, fileset-with-oss, fileset-with-gcs and fileset-with-adls.

After a fileset is created, users can easily access, manage the files/directories through the fileset's identifier, without needing to know the physical path of the managed dataset. Also, with unified access control mechanism, filesets can be managed via the same role based access control mechanism without needing to set access controls across different storage systems.

To use filesets, make sure that:

• Gravitino server has started, and the host and port is http://localhost:8090.
• A metalake has been created and enabled

Catalog Operations

Create a Catalog

tip

For a fileset catalog, you must specify the catalog type as FILESET when creating the catalog.

Create a catalog by sending a POST request to the /api/metalakes/{metalake_name}/catalogs endpoint or use the Gravitino Java client. The following is an example of creating a catalog:

Shell

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "catalog",
  "type": "FILESET",
  "comment": "comment",
  "properties": {
    "location": "file:///tmp/root"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs

Java

GravitinoClient gravitinoClient = GravitinoClient
    .builder("http://localhost:8090")
    .withMetalake("metalake")
    .build();

Map<String, String> properties = ImmutableMap.<String, String>builder()
    .put("location", "file:///tmp/root")
    // Property "location" is optional. If specified, a managed fileset without
    // a storage location will be stored under this location.
    .build();

Catalog catalog = gravitinoClient.createCatalog("catalog",
    Type.FILESET,
    "This is a fileset catalog",
    properties);
// ...

Python

gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake")
catalog = gravitino_client.create_catalog(name="catalog",
                                          catalog_type=Catalog.Type.FILESET,
                                          provider=None, 
                                          comment="This is a fileset catalog",
                                          properties={"location": "/tmp/test1"})

Gravitino supports the following catalog providers:

Catalog provider	Catalog property
fileset or None	Fileset catalog property

Load a Catalog

Refer to Load a catalog in relational catalog for more details. For a fileset catalog, the load operation is the same.

Alter a Catalog

Refer to Alter a catalog in relational catalog for more details. For a fileset catalog, the alter operation is the same.

Drop a Catalog

Refer to Drop a catalog in relational catalog for more details. For a fileset catalog, the drop operation is the same.

note

• Gravitino doesn't support dropping a catalog with schemas and filesets under it. You have to drop all the schemas and filesets under the catalog before dropping the catalog.
• The value of location property in a catalog should be a directory on a file system (HDFS) or path prefix on a cloud storage (S3, GCS, etc.).** The same goes for schema and fileset.

List All Catalogs in a Metalake

Refer to List all catalogs in a metalake in relational catalog for more details. For a fileset catalog, the list operation is the same.

List All Catalog Information in a Metalake

Refer to List all catalog information in a metalake in relational catalog for more details. For a fileset catalog, the list operation is the same.

Schema Operations

Schema is a virtual namespace in a fileset catalog, which is used to organize the fileset. It is similar to the concept of schema in relational catalog.

tip

Users should create a metalake and a catalog before creating a schema.

Create a Schema

Create a schema by sending a POST request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas endpoint or use the Gravitino Java client. The following is an example of creating a schema:

Shell

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "schema",
  "comment": "comment",
  "properties": {
    "location": "file:///tmp/root/schema"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas

Java

GravitinoClient gravitinoClient = GravitinoClient
    .builder("http://localhost:8090")
    .withMetalake("metalake")
    .build();

// Assuming you have just created a Fileset catalog named `catalog`
Catalog catalog = gravitinoClient.loadCatalog("catalog");

SupportsSchemas supportsSchemas = catalog.asSchemas();

Map<String, String> schemaProperties = ImmutableMap.<String, String>builder()
    // Property "location" is optional, if specified all the managed fileset without
    // specifying storage location will be stored under this location.
    .put("location", "file:///tmp/root/schema")
    .build();
Schema schema = supportsSchemas.createSchema("schema",
    "This is a schema",
    schemaProperties
);
// ...

Python

gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake")

catalog: Catalog = gravitino_client.load_catalog(name="catalog")
catalog.as_schemas().create_schema(name="schema", 
                                   comment="This is a schema",
                                   properties={"location": "/tmp/root/schema"})

Gravitino supports the following schema property:

Catalog provider	Schema property
fileset	Fileset schema property

Load a Schema

Refer to Load a schema in relational catalog for more details. For a fileset catalog, the schema load operation is the same.

Alter a Schema

Refer to Alter a schema in relational catalog for more details. For a fileset catalog, the schema alter operation is the same.

Drop a Schema

Refer to Drop a schema in relational catalog for more details. For a fileset catalog, the schema drop operation is the same.

Note that the drop operation will delete all the fileset metadata under this schema if cascade set to true. Besides, for MANAGED fileset, this drop operation will also remove all the files/directories of this fileset; for EXTERNAL fileset, this drop operation will only delete the metadata of this fileset.

List All Schemas Under a Catalog

Refer to List all schemas under a catalog in relational catalog for more details. For a fileset catalog, the schema list operation is the same.

Fileset Operations

tip

• Users should create a metalake, a catalog, and a schema before creating a fileset.
• Gravitino only supports managing Hadoop Compatible File System (HCFS) locations.

Create a Fileset

Create a fileset by sending a POST request to the /api/metalakes/{metalake_name} /catalogs/{catalog_name}/schemas/{schema_name}/filesets endpoint or use the Gravitino Java client. The following is an example of creating a fileset with single storage location:

Shell

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "example_fileset",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "storageLocation": "file:///tmp/root/schema/example_fileset",
  "properties": {
    "k1": "v1"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/filesets

Java

GravitinoClient gravitinoClient = GravitinoClient
    .builder("http://localhost:8090")
    .withMetalake("metalake")
    .build();

Catalog catalog = gravitinoClient.loadCatalog("catalog");
FilesetCatalog filesetCatalog = catalog.asFilesetCatalog();

Map<String, String> propertiesMap = ImmutableMap.<String, String>builder()
        .put("k1", "v1")
        .build();

filesetCatalog.createFileset(
  NameIdentifier.of("schema", "example_fileset"),
  "This is an example fileset",
  Fileset.Type.MANAGED,
  "file:///tmp/root/schema/example_fileset",
  propertiesMap,
);

Python

gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake")

catalog: Catalog = gravitino_client.load_catalog(name="catalog")
catalog.as_fileset_catalog().create_fileset(ident=NameIdentifier.of("schema", "example_fileset"),
                                            type=Fileset.Type.MANAGED,
                                            comment="This is an example fileset",
                                            storage_location="/tmp/root/schema/example_fileset",
                                            properties={"k1": "v1"})

Gravitino supports two types of filesets:

• MANAGED: The storage location of the fileset is managed by Gravitino when specified as MANAGED, the physical location of the fileset will be deleted when this fileset is dropped.
• EXTERNAL: The storage location of the fileset is not managed by Gravitino, when specified as EXTERNAL, the files of the fileset will not be deleted when the fileset is dropped.

note

During fileset creation or deletion, Gravitino automatically creates or removes the corresponding filesystem directories for the fileset locations. This behavior is skipped in either of these cases:

1. When the catalog property disable-filesystem-ops is set to true
2. When the location contains placeholders

StorageLocation

The storageLocation is the physical location of the fileset. Users can specify this location when creating a fileset, or follow the rules of the catalog/schema location if not specified.

For a MANAGED fileset, the storage location is determined in the following priority order:

1. If the user specifies storageLocation during fileset creation:

   • This location is used, with any placeholders replaced by the corresponding fileset property values.

2. If the user doesn't specify storageLocation:

   • If schema property location is specified:

     • Use <schema location>/<fileset name> if schema location has no placeholders
     • Use <schema location> with placeholders replaced by fileset property values

   • Otherwise, if catalog property location is specified:

     • Use <catalog location>/<schema name>/<fileset name> if catalog location has no placeholders
     • Use <catalog location> with placeholders replaced by fileset property values

   • If neither schema nor catalog location is specified:

     • The user must provide storageLocation during fileset creation

For an EXTERNAL fileset, the user must always specify storageLocation during fileset creation. If the provided location contains placeholders, they will be replaced by the corresponding fileset property values.

Placeholder

The storageLocation in each level can contain placeholders, format as {{name}}, which will be replaced by the corresponding fileset property value when the fileset object is created. The placeholder property in the fileset object is formed as placeholder-{{name}}. For example, if the storageLocation is file:///tmp/{{schema}}-{{fileset}}-{{verion}}, and the fileset object named "catalog1.schema1.fileset1" contains the properties placeholder-version=v1, the actual storageLocation will be file:///tmp/schema1-fileset1-v1.

The following is an example of creating a fileset with placeholders in the storageLocation:

Shell

# create a catalog first
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "test_catalog",
  "type": "FILESET",
  "comment": "comment",
  "properties": {
    "location": "file:///{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs

# create a schema under the catalog
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "test_schema",
  "comment": "comment",
  "properties": {}
}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas

# create a fileset by placeholders
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "example_fileset",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "properties": {
    "placeholder-project": "test_project",
    "placeholder-user": "test_user"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas/test_schema/filesets

# the actual storage location of the fileset will be: file:///test_catalog/test_schema/workspace_test_project/test_user

Java

GravitinoClient gravitinoClient = GravitinoClient
    .builder("http://localhost:8090")
    .withMetalake("metalake")
    .build();
// create a catalog first
Catalog catalog = gravitinoClient.createCatalog(
    "test_catalog",
    Type.FILESET,
    "comment",
    ImmutableMap.of("location", "file:///{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}"));
FilesetCatalog filesetCatalog = catalog.asFilesetCatalog();

// create a schema under the catalog
filesetCatalog.createSchema("test_schema", "comment", null);

// create a fileset by placeholders
filesetCatalog.createFileset(
  NameIdentifier.of("test_schema", "example_fileset"),
  "This is an example fileset",
  Fileset.Type.MANAGED,
  null,
  ImmutableMap.of("placeholder-project", "test_project", "placeholder-user", "test_user")
);

// the actual storage location of the fileset will be: file:///test_catalog/test_schema/workspace_test_project/test_user

Python

gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake")

# create a catalog first
catalog: Catalog = gravitino_client.create_catalog(name="test_catalog",
                                                   catalog_type=Catalog.Type.FILESET,
                                                   provider=None,
                                                   comment="comment",
                                                   properties={"location": "file:///{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}"})

# create a schema under the catalog
catalog.as_schemas().create_schema(name="test_schema", comment="comment", properties={})

# create a fileset by placeholders
catalog.as_fileset_catalog().create_fileset(ident=NameIdentifier.of("test_schema", "example_fileset"),
                                            type=Fileset.Type.MANAGED,
                                            comment="This is an example fileset",
                                            storage_location=None,
                                            properties={"placeholder-project": "test_project", "placeholder-user": "test_user"})

# the actual storage location of the fileset will be: file:///test_catalog/test_schema/workspace_test_project/test_user

StorageLocations

Create a fileset with multiple storage locations. The storageLocations is a map of location name to storage location. The generation rules of each location follow the generation rules of a single location. The following is an example of creating a fileset with multiple storage locations:

Shell

# create a catalog first
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "test_catalog",
  "type": "FILESET",
  "comment": "comment",
  "properties": {
    "location-l1": "file:///{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}",
    "location-l2": "hdfs:///{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs

# create a schema under the catalog
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "test_schema",
  "comment": "comment",
  "properties": {
    "location-l3": "s3a://myBucket/{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas

# create a fileset by placeholders
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "example_fileset",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "storageLocations": {
    "l4": "gs://myBucket/{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}"
  },
  "properties": {
    "placeholder-project": "test_project",
    "placeholder-user": "test_user",
    "default-location-name": "l1"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas/test_schema/filesets

# the fileset will be created with 4 storage locations:
{
  "name": "example_fileset",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "storageLocation": null,
  "storageLocations": {
    "l1": "file:///test_catalog/test_schema/workspace_test_project/test_user",
    "l2": "hdfs:///test_catalog/test_schema/workspace_test_project/test_user",
    "l3": "s3a://myBucket/test_catalog/test_schema/workspace_test_project/test_user",
    "l4": "gs://myBucket/test_catalog/test_schema/workspace_test_project/test_user"
  },
  "properties": {
    "placeholder-project": "test_project",
    "placeholder-user": "test_user",
    "default-location-name": "l1"
  }
}

Java

GravitinoClient gravitinoClient = GravitinoClient
    .builder("http://localhost:8090")
    .withMetalake("metalake")
    .build();
// create a catalog first
Catalog catalog = gravitinoClient.createCatalog(
    "test_catalog",
    Type.FILESET,
    "comment",
    ImmutableMap.of(
        "location-l1", "file:///{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}",
        "location-l2", "hdfs:///{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}"));
FilesetCatalog filesetCatalog = catalog.asFilesetCatalog();

// create a schema under the catalog
filesetCatalog.createSchema(
    "test_schema",
    "comment",
    ImmutableMap.of("location-l3", "s3a://myBucket/{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}"));

// create a fileset by placeholders
filesetCatalog.createMultipleLocationFileset(
  NameIdentifier.of("test_schema", "example_fileset"),
  "This is an example fileset",
  Fileset.Type.MANAGED,
  ImmutableMap.of("l4", "gs://myBucket/{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}"),
  ImmutableMap.of(
      "placeholder-project", "test_project", 
      "placeholder-user", "test_user",
      "default-location-name", "l1")
);

// the fileset will be created with 4 storage locations:
{
  "name": "example_fileset",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "storageLocation": null,
  "storageLocations": {
    "l1": "file:///test_catalog/test_schema/workspace_test_project/test_user",
    "l2": "hdfs:///test_catalog/test_schema/workspace_test_project/test_user",
    "l3": "s3a://myBucket/test_catalog/test_schema/workspace_test_project/test_user",
    "l4": "gs://myBucket/test_catalog/test_schema/workspace_test_project/test_user"
  },
  "properties": {
    "placeholder-project": "test_project",
    "placeholder-user": "test_user",
    "default-location-name": "l1"
  }
}

Python

gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake")

# create a catalog first
catalog: Catalog = gravitino_client.create_catalog(
   name="test_catalog",
   catalog_type=Catalog.Type.FILESET,
   provider=None,
   comment="comment",
   properties={
      "location-l1": "file:///{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}",
      "location-l2": "hdfs:///{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}",
    }
)

# create a schema under the catalog
catalog.as_schemas().create_schema(
   name="test_schema",
   comment="comment",
   properties={
      "location-l3": "s3a://myBucket/{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}",
   }
)

# create a fileset by placeholders
catalog.as_fileset_catalog().create_multiple_location_fileset(
    ident=NameIdentifier.of("test_schema", "example_fileset"),
    type=Fileset.Type.MANAGED,
    comment="This is an example fileset",
    storage_locations={
        "l4": "gs://myBucket/{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}",
    },
    roperties={
       "placeholder-project": "test_project",
       "placeholder-user": "test_user",
       "default-location-name": "l1",
    }
)

# the fileset will be created with 4 storage locations:
{
  "name": "example_fileset",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "storageLocation": null,
  "storageLocations": {
    "l1": "file:///test_catalog/test_schema/workspace_test_project/test_user",
    "l2": "hdfs:///test_catalog/test_schema/workspace_test_project/test_user",
    "l3": "s3a://myBucket/test_catalog/test_schema/workspace_test_project/test_user",
    "l4": "gs://myBucket/test_catalog/test_schema/workspace_test_project/test_user"
  },
  "properties": {
    "placeholder-project": "test_project",
    "placeholder-user": "test_user",
    "default-location-name": "l1"
  }
}

Alter a Fileset

Modify a fileset by sending a PUT request to the /api/metalakes/{metalake_name} /catalogs/{catalog_name}/schemas/{schema_name}/filesets/{fileset_name} endpoint or just use the Gravitino Java client. The following is an example of modifying a fileset:

Shell

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "updates": [
    {
      "@type": "removeProperty",
      "property": "key2"
    }, {
      "@type": "setProperty",
      "property": "key3",
      "value": "value3"
    }
  ]
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/filesets/fileset

Java

// ...
// Assuming you have just created a Fileset catalog named `catalog`
Catalog catalog = gravitinoClient.loadCatalog("catalog");

FilesetCatalog filesetCatalog = catalog.asFilesetCatalog();

Fileset f = filesetCatalog.alterFileset(NameIdentifier.of("schema", "fileset"),
    FilesetChange.rename("fileset_renamed"), FilesetChange.updateComment("xxx"));
// ...

Python

gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake")

catalog: Catalog = gravitino_client.load_catalog(name="catalog")
changes = (
   FilesetChange.remove_property("fileset_properties_key1"),
   FilesetChange.set_property("fileset_properties_key2", "fileset_properties_new_value"),
)
fileset_new = catalog.as_fileset_catalog().alter_fileset(NameIdentifier.of("schema", "fileset"), 
                                                         *changes)

Gravitino supports the following changes to a fileset:

Supported modification	JSON	Java
Rename a fileset	{"@type":"rename","newName":"fileset_renamed"}	FilesetChange.rename("fileset_renamed")
Update a comment	{"@type":"updateComment","newComment":"new_comment"}	FilesetChange.updateComment("new_comment")
Set a fileset property	{"@type":"setProperty","property":"key1","value":"value1"}	FilesetChange.setProperty("key1", "value1")
Remove a fileset property	{"@type":"removeProperty","property":"key1"}	FilesetChange.removeProperty("key1")
Remove comment (deprecated)	{"@type":"removeComment"}	FilesetChange.removeComment()

Drop a Fileset

Remove a fileset by sending a DELETE request to the /api/metalakes/{metalake_name} /catalogs/{catalog_name}/schemas/{schema_name}/filesets/{fileset_name} endpoint or by using the Gravitino Java client. The following is an example of dropping a fileset:

Shell

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/filesets/fileset

Java

// ...
// Assuming you have just created a Fileset catalog named `catalog`
Catalog catalog = gravitinoClient.loadCatalog("catalog");

FilesetCatalog filesetCatalog = catalog.asFilesetCatalog();

// Drop a fileset
filesetCatalog.dropFileset(NameIdentifier.of("schema", "fileset"));
// ...

Python

gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake")

catalog: Catalog = gravitino_client.load_catalog(name="catalog")
catalog.as_fileset_catalog().drop_fileset(ident=NameIdentifier.of("schema", "fileset"))

For a MANAGED fileset, the physical location of the fileset will be deleted when this fileset is dropped. For EXTERNAL fileset, only the metadata of the fileset will be removed.

List Filesets

List all filesets in a schema by sending a GET request to the /api/metalakes/ {metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/filesets endpoint or by using the Gravitino Java client. The following is an example of listing all the filesets in a schema:

Shell

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/filesets

Java

// ...
Catalog catalog = gravitinoClient.loadCatalog("catalog");

FilesetCatalog filesetCatalog = catalog.asFilesetCatalog();
NameIdentifier[] identifiers =
    filesetCatalog.listFilesets(Namespace.of("schema"));
// ...

Python

gravitino_client: GravitinoClient = GravitinoClient(uri="http://localhost:8090", metalake_name="metalake")

catalog: Catalog = gravitino_client.load_catalog(name="catalog")
fileset_list: List[NameIdentifier] = catalog.as_fileset_catalog().list_filesets(namespace=Namespace.of("schema")))

Manage Filesets Across Multiple Clusters

In general, the locations of all schemas and filesets under a fileset catalog belong to a single Hadoop cluster if they are HDFS location.

The example for creating a fileset is as follows:

# create fileset catalog
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "fileset_catalog",
  "type": "FILESET",
  "comment": "This is a fileset catalog",
  "provider": "fileset",
  "properties": {
    "location": "hdfs://172.17.0.2:9000/fileset_catalog"
  }
}' http://localhost:8090/api/metalakes/test/catalogs

# create a fileset schema under the catalog with inherited properties
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "test_schema",
  "comment": "This is a schema",
  "properties": {
  }
}' http://localhost:8090/api/metalakes/test/catalogs/fileset_catalog/schemas

# create a fileset under the schema with inherited properties
curl -X POST -H "Accept: application/vnd.gravitino.v1+json"
-H "Content-Type: application/json" -d '{
  "name": "fs1",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "properties": {
  }
}' http://localhost:8090/api/metalakes/test/catalogs/fileset_catalog/schemas/test_schema/filesets

Within a fileset catalog, schemas and filesets can automatically inherit configuration properties from their parent catalog. For example, the location property can be inherited — a schema can inherit the catalog’s location as its base path, and a fileset can in turn inherit the schema’s location as its base path.

The property inheritance priority is as follows: catalog < schema < fileset.

If a fileset needs to use a different storage path, it can specify its own location configuration to override the inherited one.

The fileset catalog also supports multiple clusters. Each schema and fileset under a catalog can independently specify their own cluster locations and connection configurations.

For example, a complex catalog structure might look like this:

catalog1 -> hdfs://cluster1/catalog1
    schema1 -> hdfs://cluster1/catalog1/schema1
        fileset1 -> hdfs://cluster1/catalog1/schema1/fileset1
        fileset2 -> hdfs://cluster1/catalog1/schema1/fileset2
    schema2 -> hdfs://cluster2/tmp/schema2
        fileset3 -> hdfs://cluster2/tmp/schema2/fsd
        fileset4 -> hdfs://cluster3/customers

In this example, the default location of catalog1 is hdfs://cluster1/catalog1. schema1 and its filesets are stored in the same cluster as defined by the catalog (cluster1). However, schema2 and its filesets (fileset3, fileset4) are located in different clusters (cluster2 and cluster3, respectively).

The example for creating Filesets with different clusters as follows:

# create fileset catalog
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "fileset_catalog",
  "type": "FILESET",
  "comment": "This is a fileset catalog",
  "provider": "fileset",
  "properties": {
    "location": "hdfs://172.17.0.2:9000/fileset_catalog"
  }
}' http://localhost:8090/api/metalakes/test/catalogs

# create a fileset schema under the catalog with inherited properties
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "test_schema",
  "comment": "This is a schema",
  "properties": {
  }
}' http://localhost:8090/api/metalakes/test/catalogs/fileset_catalog/schemas


# create fileset fs1 in the default HDFS cluster
curl -X POST -H "Accept: application/vnd.gravitino.v1+json"
-H "Content-Type: application/json" -d '{
  "name": "fs1",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "properties": {
  }
}' http://localhost:8090/api/metalakes/test/catalogs/fileset_catalog/schemas/test_schema/filesets

# create fileset fs2 in another HDFS cluster
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "fs2",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "storageLocation": "hdfs://172.17.0.3:9000/tmp/fs2",
  "properties": {
    "config.resources": "/etc/conf/cluster1/core-site.xml,/etc/conf/cluster1/hdfs-site.xml"
  }
}' http://localhost:8090/api/metalakes/test/catalogs/fileset_catalog/schemas/test_schema/filesets

Alternatively, you can pass configurations for multiple clusters using the fs.path.config.{cluster_name}.* format when creating the fileset catalog. All schemas and filesets under the catalog can inherit configuration based on the cluster name, and the configuration can be overridden at the schema or fileset level if needed. The system can automatically retrieve the appropriate configuration from fs.path.config configuration based on the fileset’s storage location and override the corresponding fileset properties by key.

# create fileset catalog with multiple cluster configurations, and specify HDFS and S3 clusters configurations by logical names
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "fileset_catalog",
  "type": "FILESET",
  "comment": "This is a fileset catalog",
  "provider": "fileset",
  "properties": {
    "location": "hdfs://172.17.0.2:9000/fileset_catalog",
    "fs.path.config.hadoop1"="hdfs://172.17.0.2:9000",
    "fs.path.config.hadoop1.config.resources"="/etc/conf/cluster1/core-site.xml,/etc/conf/cluster1/hdfs-site.xml",
    "fs.path.config.aws1"="s3://mybuctket",
    "fs.path.config.aws1.access.key"="your_access_key",
    "fs.path.config.aws1.secret.key"="your_secret_key"
  }
}' http://localhost:8090/api/metalakes/test/catalogs

# create fileset fs1 in the default HDFS cluster, it will inherit the hadoop1 cluster configuration from the catalog
curl -X POST -H "Accept: application/vnd.gravitino.v1+json"
-H "Content-Type: application/json" -d '{
  "name": "fs1",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "storageLocation": "hdfs://172.17.0.2:9000/datawarehouse/fs1"
}' http://localhost:8090/api/metalakes/test/catalogs/fileset_catalog/schemas/test_schema/filesets

# create fileset fs2 in another s3 cluster, it will inherit the aws1 cluster configuration from the catalog
curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "fs2",
  "comment": "This is an example fileset",
  "type": "MANAGED",
  "storageLocation": "s3://mybuctket/datawarehouse/fs2"
}' http://localhost:8090/api/metalakes/test/catalogs/fileset_catalog/schemas/test_schema/filesets