Manage Catalogs and Schemas

Introduction

This page covers the Gravitino API for catalogs and schemas. For what a catalog and a schema are, the catalog types, what Gravitino stores, permissions, and how to work with them in the UI, see Catalogs and Schemas.

Connection properties differ by provider and are documented on each catalog type's own page.

Catalog Operations

Create a Catalog

A catalog needs a name, a type, and for most types a provider. Properties carry the connection details. The example below registers a Hive metastore; other providers take different properties.

REST

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "name": "sales",
  "type": "RELATIONAL",
  "provider": "hive",
  "comment": "Sales estate",
  "properties": {"metastore.uris": "thrift://localhost:9083"}
}' http://localhost:8090/api/metalakes/example/catalogs

Java

GravitinoClient client = GravitinoClient
    .builder("http://localhost:8090")
    .withMetalake("example")
    .build();

Catalog catalog = client.createCatalog(
    "sales",
    Catalog.Type.RELATIONAL,
    "hive",
    "Sales estate",
    ImmutableMap.of("metastore.uris", "thrift://localhost:9083"));

Python

catalog = client.create_catalog(
    name="sales",
    catalog_type=Catalog.Type.RELATIONAL,
    provider="hive",
    comment="Sales estate",
    properties={"metastore.uris": "thrift://localhost:9083"})

Fileset and model catalogs are managed by Gravitino rather than federated, so they take no provider.

REST

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "name": "landing",
  "type": "FILESET",
  "comment": "Landing zone",
  "properties": {"location": "s3a://example-bucket/landing"}
}' http://localhost:8090/api/metalakes/example/catalogs

Load a Catalog

REST

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
  http://localhost:8090/api/metalakes/example/catalogs/sales

Java

Catalog catalog = client.loadCatalog("sales");

Python

catalog = client.load_catalog("sales")

Alter a Catalog

Changes are applied as a list in one request.

Change	JSON	Java	Python
Rename	{"@type":"rename","newName":"sales_v2"}	CatalogChange.rename("sales_v2")	CatalogChange.rename("sales_v2")
Update the comment	{"@type":"updateComment","newComment":"new_comment"}	CatalogChange.updateComment("new_comment")	CatalogChange.update_comment("new_comment")
Set a property	{"@type":"setProperty","property":"key1","value":"value1"}	CatalogChange.setProperty("key1", "value1")	CatalogChange.set_property("key1", "value1")
Remove a property	{"@type":"removeProperty","property":"key1"}	CatalogChange.removeProperty("key1")	CatalogChange.remove_property("key1")

REST

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "updates": [
    {"@type": "updateComment", "newComment": "Sales estate, production"}
  ]
}' http://localhost:8090/api/metalakes/example/catalogs/sales

Java

Catalog catalog = client.alterCatalog(
    "sales", CatalogChange.updateComment("Sales estate, production"));

Python

catalog = client.alter_catalog(
    "sales", CatalogChange.update_comment("Sales estate, production"))

Enable or Disable a Catalog

A catalog that is not in use can only be listed, loaded, enabled, or dropped.

REST

curl -X PATCH -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{"inUse": false}' \
  http://localhost:8090/api/metalakes/example/catalogs/sales

Java

client.disableCatalog("sales");
client.enableCatalog("sales");

Python

client.disable_catalog("sales")
client.enable_catalog("sales")

Drop a Catalog

Without force, the catalog must have no schemas and must not be in use. With force, Gravitino removes the registration and everything it holds about the contents.

REST

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
  "http://localhost:8090/api/metalakes/example/catalogs/sales?force=false"

Java

boolean dropped = client.dropCatalog("sales", false);

Python

dropped = client.drop_catalog("sales", force=False)

List Catalogs

REST

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
  http://localhost:8090/api/metalakes/example/catalogs

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
  "http://localhost:8090/api/metalakes/example/catalogs?details=true"

Java

String[] catalogNames = client.listCatalogs();
Catalog[] catalogs = client.listCatalogsInfo();

Python

catalog_names = client.list_catalogs()
catalogs = client.list_catalogs_info()

Schema Operations

Schema operations are the same for every catalog type. Creating a schema through Gravitino creates it in the source system too, where the source supports that.

Create a Schema

REST

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "name": "public",
  "comment": "Shared datasets",
  "properties": {}
}' http://localhost:8090/api/metalakes/example/catalogs/sales/schemas

Java

Catalog catalog = client.loadCatalog("sales");
Schema schema = catalog.asSchemas().createSchema(
    "public", "Shared datasets", Collections.emptyMap());

Python

catalog = client.load_catalog("sales")
schema = catalog.as_schemas().create_schema(
    schema_name="public", comment="Shared datasets", properties={})

Load a Schema

REST

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
  http://localhost:8090/api/metalakes/example/catalogs/sales/schemas/public

Java

Schema schema = catalog.asSchemas().loadSchema("public");

Python

schema = catalog.as_schemas().load_schema("public")

Alter a Schema

A schema takes property changes only. It cannot be renamed, and its comment cannot be changed.

Change	JSON	Java	Python
Set a property	{"@type":"setProperty","property":"key1","value":"value1"}	SchemaChange.setProperty("key1", "value1")	SchemaChange.set_property("key1", "value1")
Remove a property	{"@type":"removeProperty","property":"key1"}	SchemaChange.removeProperty("key1")	SchemaChange.remove_property("key1")

REST

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" -d '{
  "updates": [
    {"@type": "setProperty", "property": "owner", "value": "sales-eng"}
  ]
}' http://localhost:8090/api/metalakes/example/catalogs/sales/schemas/public

Java

Schema schema = catalog.asSchemas().alterSchema(
    "public", SchemaChange.setProperty("owner", "sales-eng"));

Python

schema = catalog.as_schemas().alter_schema(
    "public", SchemaChange.set_property("owner", "sales-eng"))

Drop a Schema

Without cascade, the schema must be empty. With cascade, everything inside it goes as well.

REST

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
  "http://localhost:8090/api/metalakes/example/catalogs/sales/schemas/public?cascade=false"

Java

boolean dropped = catalog.asSchemas().dropSchema("public", false);

Python

dropped = catalog.as_schemas().drop_schema("public", cascade=False)

List Schemas

REST

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
  http://localhost:8090/api/metalakes/example/catalogs/sales/schemas

Java

String[] schemaNames = catalog.asSchemas().listSchemas();

Python

schema_names = catalog.as_schemas().list_schemas()