Skip to main content
Version: 0.7.0-incubating

Manage relational metadata using Apache Gravitino

This page introduces how to manage relational metadata by Apache Gravitino, relational metadata refers to relational catalog, schema, table and partitions. Through Gravitino, you can create, edit, and delete relational metadata via unified REST APIs or Java client.

In this document, Gravitino uses Apache Hive catalog as an example to show how to manage relational metadata by Gravitino. Other relational catalogs are similar to Hive catalog, but they may have some differences, especially in catalog property, table property, and column type. For more details, please refer to the related doc.

Assuming:

Catalog operations

Create a catalog

tip

The code below is an example of creating a Hive catalog. For other relational catalogs, the code is similar, but the catalog type, provider, and properties may be different. For more details, please refer to the related doc.

For relational catalog, you must specify the catalog type as RELATIONAL when creating a catalog.

You can create a catalog by sending a POST request to the /api/metalakes/{metalake_name}/catalogs endpoint or just use the Gravitino Java client. The following is an example of creating a catalog:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"name": "catalog",
"type": "RELATIONAL",
"comment": "comment",
"provider": "hive",
"properties": {
"metastore.uris": "thrift://localhost:9083"
}
}' http://localhost:8090/api/metalakes/metalake/catalogs

Currently, Gravitino supports the following catalog providers:

Catalog providerCatalog property
hiveHive catalog property
lakehouse-icebergIceberg catalog property
lakehouse-paimonPaimon catalog property
lakehouse-hudiHudi catalog property
jdbc-mysqlMySQL catalog property
jdbc-postgresqlPostgreSQL catalog property
jdbc-dorisDoris catalog property
jdbc-oceanbaseOceanBase catalog property

Load a catalog

You can load a catalog by sending a GET request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name} endpoint or just use the Gravitino Java client. The following is an example of loading a catalog:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" http://localhost:8090/api/metalakes/metalake/catalogs/catalog

Alter a catalog

You can modify a catalog by sending a PUT request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name} endpoint or just use the Gravitino Java client. The following is an example of altering a catalog:

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"updates": [
{
"@type": "rename",
"newName": "alter_catalog"
},
{
"@type": "setProperty",
"property": "key3",
"value": "value3"
}
]
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog

Currently, Gravitino supports the following changes to a catalog:

Supported modificationJSONJava
Rename catalog{"@type":"rename","newName":"catalog_renamed"}CatalogChange.rename("catalog_renamed")
Update comment{"@type":"updateComment","newComment":"new_comment"}CatalogChange.updateComment("new_comment")
Set a property{"@type":"setProperty","property":"key1","value":"value1"}CatalogChange.setProperty("key1", "value1")
Remove a property{"@type":"removeProperty","property":"key1"}CatalogChange.removeProperty("key1")
warning

Most catalog-altering operations are generally safe. However, if you want to change the catalog's URI, you should proceed with caution. Changing the URI may point to a different cluster, rendering the metadata stored in Gravitino unusable. For instance, if the old URI and the new URI point to different clusters that both have a database named db1, changing the URI might cause the old metadata, such as audit information, to be used when accessing db1, which is undesirable.

Therefore, do not change the catalog's URI unless you fully understand the consequences of such a modification.

Enable a catalog

Catalog has a reserved property - in-use, which indicates whether the catalog is available for use. By default, the in-use property is set to true. To enable a disabled catalog, you can send a PATCH request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name} endpoint or use the Gravitino Java client.

The following is an example of enabling a catalog:

curl -X PATCH -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{"inUse": true}' \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog
info

This operation does nothing if the catalog is already enabled.

Disable a catalog

Once a catalog is disabled:

  • Users can only list, load, drop, or enable it.
  • Any other operation on the catalog or its sub-entities will result in an error.

To disable a catalog, you can send a PATCH request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name} endpoint or use the Gravitino Java client.

The following is an example of disabling a catalog:

curl -X PATCH -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{"inUse": false}' \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog
info

This operation does nothing if the catalog is already disabled.

Drop a catalog

Deleting a catalog by "force" is not a default behavior, so please make sure:

  • There are no schemas under the catalog. Otherwise, you will get an error.
  • The catalog is disabled. Otherwise, you will get an error.

Deleting a catalog by "force" will:

  • Delete all sub-entities (schemas, tables, etc.) under the catalog.
  • Delete the catalog itself even if it is enabled.
  • Not delete the external resources (such as database, table, etc.) associated with sub-entities unless they are managed (such as managed fileset).

You can remove a catalog by sending a DELETE request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name} endpoint or just use the Gravitino Java client. The following is an example of dropping a catalog:

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog?force=false

List all catalogs in a metalake

You can list all catalogs under a metalake by sending a GET request to the /api/metalakes/{metalake_name}/catalogs endpoint or just use the Gravitino Java client. The following is an example of listing all the catalogs in a metalake:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs

List all catalogs' information in a metalake

You can list all catalogs' information under a metalake by sending a GET request to the /api/metalakes/{metalake_name}/catalogs?details=true endpoint or just use the Gravitino Java client. The following is an example of listing all the catalogs' information in a metalake:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs?details=true

Schema operations

tip

Users should create a metalake and a catalog, then ensure that the metalake and catalog are enabled before operating schemas.

Create a schema

You can create a schema by sending a POST request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas endpoint or just use the Gravitino Java client. The following is an example of creating a schema:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"name": "schema",
"comment": "comment",
"properties": {
"key1": "value1"
}
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas

Currently, Gravitino supports the following schema property:

Catalog providerSchema property
hiveHive schema property
lakehouse-icebergIceberg scheme property
lakehouse-paimonPaimon scheme property
lakehouse-hudiHudi scheme property
jdbc-mysqlMySQL schema property
jdbc-postgresqlPostgreSQL schema property
jdbc-dorisDoris schema property
jdbc-oceanbaseOceanBase schema property

Load a schema

You can create a schema by sending a GET request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name} endpoint or just use the Gravitino Java client. The following is an example of loading a schema:

curl -X GET \-H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema

Alter a schema

You can change a schema by sending a PUT request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name} endpoint or just use the Gravitino Java client. The following is an example of modifying a schema:

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"updates": [
{
"@type": "removeProperty",
"property": "key2"
}, {
"@type": "setProperty",
"property": "key3",
"value": "value3"
}
]
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema

Currently, Gravitino supports the following changes to a schema:

Supported modificationJSONJava
Set a property{"@type":"setProperty","property":"key1","value":"value1"}SchemaChange.setProperty("key1", "value1")
Remove a property{"@type":"removeProperty","property":"key1"}SchemaChange.removeProperty("key1")

Drop a schema

You can remove a schema by sending a DELETE request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name} endpoint or just use the Gravitino Java client. The following is an example of dropping a schema:

// cascade can be true or false
curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema?cascade=true

If cascade is true, Gravitino will drop all tables under the schema. Otherwise, Gravitino will throw an exception if there are tables under the schema. Some catalogs may not support cascading deletion of a schema, please refer to the related doc for more details.

List all schemas under a catalog

You can list all schemas under a catalog by sending a GET request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas endpoint or just use the Gravitino Java client. The following is an example of listing all the schemas in a catalog:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas

Table operations

tip

Users should create a metalake, a catalog and a schema, then ensure that the metalake and catalog are enabled before before operating tables.

Create a table

You can create a table by sending a POST request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/tables endpoint or just use the Gravitino Java client. The following is an example of creating a table:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"name": "example_table",
"comment": "This is an example table",
"columns": [
{
"name": "id",
"type": "integer",
"comment": "id column comment",
"nullable": false,
"autoIncrement": true,
"defaultValue": {
"type": "literal",
"dataType": "integer",
"value": "-1"
}
},
{
"name": "name",
"type": "varchar(500)",
"comment": "name column comment",
"nullable": true,
"autoIncrement": false,
"defaultValue": {
"type": "literal",
"dataType": "null",
"value": "null"
}
},
{
"name": "StartingDate",
"type": "timestamp",
"comment": "StartingDate column comment",
"nullable": false,
"autoIncrement": false,
"defaultValue": {
"type": "function",
"funcName": "current_timestamp",
"funcArgs": []
}
},
{
"name": "info",
"type": {
"type": "struct",
"fields": [
{
"name": "position",
"type": "string",
"nullable": true,
"comment": "position field comment"
},
{
"name": "contact",
"type": {
"type": "list",
"elementType": "integer",
"containsNull": false
},
"nullable": true,
"comment": "contact field comment"
},
{
"name": "rating",
"type": {
"type": "map",
"keyType": "string",
"valueType": "integer",
"valueContainsNull": false
},
"nullable": true,
"comment": "rating field comment"
}
]
},
"comment": "info column comment",
"nullable": true
},
{
"name": "dt",
"type": "date",
"comment": "dt column comment",
"nullable": true
}
],
"partitioning": [
{
"strategy": "identity",
"fieldName": [ "dt" ]
}
],
"distribution": {
"strategy": "hash",
"number": 32,
"funcArgs": [
{
"type": "field",
"fieldName": [ "id" ]
}
]
},
"sortOrders": [
{
"sortTerm": {
"type": "field",
"fieldName": [ "age" ]
},
"direction": "asc",
"nullOrdering": "nulls_first"
}
],
"indexes": [
{
"indexType": "primary_key",
"name": "PRIMARY",
"fieldNames": [["id"]]
}
],
"properties": {
"format": "ORC"
}
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/tables
caution

The provided example demonstrates table creation but isn't directly executable in Gravitino, since not all catalogs fully support these capabilities.

In order to create a table, you need to provide the following information:

  • Table column name and type
  • Table column default value (optional)
  • Table column auto-increment (optional)
  • Table property (optional)

Apache Gravitino table column type

The following types that Gravitino supports:

TypeJavaJSONDescription
BooleanTypes.BooleanType.get()booleanBoolean type
ByteTypes.ByteType.get()byteByte type, indicates a numerical value of 1 byte
Unsigned ByteTypes.ByteType.unsigned()byte unsignedUnsigned Byte type, indicates a unsigned numerical value of 1 byte
ShortTypes.ShortType.get()shortShort type, indicates a numerical value of 2 bytes
Unsigned ShortTypes.ShortType.unsigned()short unsignedUnsigned Short type, indicates a unsigned numerical value of 2 bytes
IntegerTypes.IntegerType.get()integerInteger type, indicates a numerical value of 4 bytes
Unsigned IntegerTypes.IntegerType.unsigned()integer unsignedUnsigned Integer type, indicates a unsigned numerical value of 4 bytes
LongTypes.LongType.get()longLong type, indicates a numerical value of 8 bytes
Unsigned LongTypes.LongType.unsigned()long unsignedUnsigned Long type, indicates a unsigned numerical value of 8 bytes
FloatTypes.FloatType.get()floatFloat type, indicates a single-precision floating point number
DoubleTypes.DoubleType.get()doubleDouble type, indicates a double-precision floating point number
Decimal(precision, scale)Types.DecimalType.of(precision, scale)decimal(p, s)Decimal type, indicates a fixed-precision decimal number with the constraint that the precision must be in range [1, 38] and the scala must be in range [0, precision]
StringTypes.StringType.get()stringString type
FixedChar(length)Types.FixedCharType.of(length)char(l)Char type, indicates a fixed-length string
VarChar(length)Types.VarCharType.of(length)varchar(l)Varchar type, indicates a variable-length string, the length is the maximum length of the string
TimestampTypes.TimestampType.withoutTimeZone()timestampTimestamp type, indicates a timestamp without timezone
TimestampWithTimezoneTypes.TimestampType.withTimeZone()timestamp_tzTimestamp with timezone type, indicates a timestamp with timezone
DateTypes.DateType.get()dateDate type
TimeTypes.TimeType.withoutTimeZone()timeTime type
IntervalToYearMonthTypes.IntervalYearType.get()interval_yearInterval type, indicates an interval of year and month
IntervalToDayTimeTypes.IntervalDayType.get()interval_dayInterval type, indicates an interval of day and time
Fixed(length)Types.FixedType.of(length)fixed(l)Fixed type, indicates a fixed-length binary array
BinaryTypes.BinaryType.get()binaryBinary type, indicates a arbitrary-length binary array
ListTypes.ListType.of(elementType, elementNullable){"type": "list", "containsNull": JSON Boolean, "elementType": type JSON}List type, indicate a list of elements with the same type
MapTypes.MapType.of(keyType, valueType){"type": "map", "keyType": type JSON, "valueType": type JSON, "valueContainsNull": JSON Boolean}Map type, indicate a map of key-value pairs
StructTypes.StructType.of([Types.StructType.Field.of(name, type, nullable)]){"type": "struct", "fields": [JSON StructField, {"name": string, "type": type JSON, "nullable": JSON Boolean, "comment": string}]}Struct type, indicate a struct of fields
UnionTypes.UnionType.of([type1, type2, ...]){"type": "union", "types": [type JSON, ...]}Union type, indicates a union of types
UUIDTypes.UUIDType.get()uuidUUID type, indicates a universally unique identifier

The related java doc is here.

External type

External type is a special type of column type, when you need to use a data type that is not in the Gravitino type system, and you explicitly know its string representation in an external catalog (usually used in JDBC catalogs), then you can use the ExternalType to represent the type. Similarly, if the original type is unsolvable, it will be represented by ExternalType. The following shows the data structure of an external type in JSON and Java, enabling easy retrieval of its string value.

{
"type": "external",
"catalogString": "user-defined"
}
Unparsed type

Unparsed type is a special type of column type, it used to address compatibility issues in type serialization and deserialization between the server and client. For instance, if a new column type is introduced on the Gravitino server that the client does not recognize, it will be treated as an unparsed type on the client side. The following shows the data structure of an unparsed type in JSON and Java, enabling easy retrieval of its value.

{
"type": "unparsed",
"unparsedType": "unknown-type"
}

Table column default value

When defining a table column, you can specify a literal or an expression as the default value. The default value typically applies to new rows that are inserted into the table by the underlying catalog.

The following is a table of the column default value that Gravitino supports for different catalogs:

Catalog providerSupported default value
hive
lakehouse-iceberg
lakehouse-paimon
lakehouse-hudi
jdbc-mysql
jdbc-postgresql
jdbc-doris
jdbc-oceanbase

Table column auto-increment

Auto-increment provides a convenient way to ensure that each row in a table has a unique identifier without the need for manually managing identifier allocation. The following table shows the column auto-increment that Gravitino supports for different catalogs:

Catalog providerSupported auto-increment
hive
lakehouse-iceberg
lakehouse-paimon
lakehouse-hudi
jdbc-mysql✔(limitations)
jdbc-postgresql
jdbc-doris
jdbc-oceanbase✔(limitations)

Table property and type mapping

The following is the table property that Gravitino supports:

Catalog providerTable propertyType mapping
hiveHive table propertyHive type mapping
lakehouse-icebergIceberg table propertyIceberg type mapping
lakehouse-paimonPaimon table propertyPaimon type mapping
lakehouse-hudiHudi table propertyHudi type mapping
jdbc-mysqlMySQL table propertyMySQL type mapping
jdbc-postgresqlPostgreSQL table propertyPostgreSQL type mapping
jdbc-dorisDoris table propertyDoris type mapping
jdbc-oceanbaseOceanBase table propertyOceanBase type mapping

Table partitioning, distribution, sort ordering and indexes

In addition to the basic settings, Gravitino supports the following features:

FeatureDescriptionJava doc
Table partitioningEqual to PARTITION BY in Apache Hive, It is a partitioning strategy that is used to split a table into parts based on partition keys. Some table engine may not support this featurePartition
Table distributionEqual to CLUSTERED BY in Apache Hive, distribution a.k.a (Clustering) is a technique to split the data into more manageable files/parts, (By specifying the number of buckets to create). The value of the distribution column will be hashed by a user-defined number into buckets.Distribution
Table sort orderingEqual to SORTED BY in Apache Hive, sort ordering is a method to sort the data in specific ways such as by a column or a function, and then store table data. it will highly improve the query performance under certain scenarios.SortOrder
Table indexesEqual to KEY/INDEX in MySQL , unique key enforces uniqueness of values in one or more columns within a table. It ensures that no two rows have identical values in specified columns, thereby facilitating data integrity and enabling efficient data retrieval and manipulation operations.Index

For more information, please see the related document on partitioning, bucketing, sorting, and indexes.

note

The code above is an example of creating a Hive table. For other catalogs, the code is similar, but the supported column type, and table properties may be different. For more details, please refer to the related doc.

Load a table

You can load a table by sending a GET request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/tables/{table_name} endpoint or just use the Gravitino Java client. The following is an example of loading a table:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/tables/table
note
  • When Gravitino loads a table from a catalog with various data types, if Gravitino is unable to parse the data type, it will use an External Type to preserve the original data type, ensuring that the table can be loaded successfully.
  • When Gravitino loads a table from a catalog that supports default value, if Gravitino is unable to parse the default value, it will use an Unparsed Expression to preserve the original default value, ensuring that the table can be loaded successfully.

Alter a table

You can modify a table by sending a PUT request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/tables/{table_name} endpoint or just use the Gravitino Java client. The following is an example of modifying a table:

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
"updates": [
{
"@type": "removeProperty",
"property": "key2"
}, {
"@type": "setProperty",
"property": "key3",
"value": "value3"
}
]
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/tables/table

Currently, Gravitino supports the following changes to a table:

Supported modificationJSONJava
Rename table{"@type":"rename","newName":"table_renamed"}TableChange.rename("table_renamed")
Update comment{"@type":"updateComment","newComment":"new_comment"}TableChange.updateComment("new_comment")
Set a table property{"@type":"setProperty","property":"key1","value":"value1"}TableChange.setProperty("key1", "value1")
Remove a table property{"@type":"removeProperty","property":"key1"}TableChange.removeProperty("key1")
Add a column{"@type":"addColumn","fieldName":["position"],"type":"varchar(20)","comment":"Position of user","position":"FIRST","nullable": true, "autoIncrement": false, "defaultValue" : {"type": "literal", "dataType": "varchar(20)", "value": "Default Position"}}TableChange.addColumn(...)
Delete a column{"@type":"deleteColumn","fieldName": ["name"], "ifExists": true}TableChange.deleteColumn(...)
Rename a column{"@type":"renameColumn","oldFieldName":["name_old"], "newFieldName":"name_new"}TableChange.renameColumn(...)
Update the column comment{"@type":"updateColumnComment", "fieldName": ["name"], "newComment": "new comment"}TableChange.updateColumnCommment(...)
Update the type of a column{"@type":"updateColumnType","fieldName": ["name"], "newType":"varchar(100)"}TableChange.updateColumnType(...)
Update the nullability of a column{"@type":"updateColumnNullability","fieldName": ["name"],"nullable":true}TableChange.updateColumnNullability(...)
Update the position of a column{"@type":"updateColumnPosition","fieldName": ["name"], "newPosition":"default"}TableChange.updateColumnPosition(...)
Update the default value of a column{"@type":"updateColumnDefaultValue","fieldName": ["name"], "newDefaultValue":{"type":"literal","dataType":"varchar(100)","value":"new default value}}TableChange.updateColumnDefaultValue(...)

Drop a table

You can remove a table by sending a DELETE request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/tables/{table_name} endpoint or just use the Gravitino Java client. The following is an example of dropping a table:

## Purge can be true or false, if purge is true, Gravitino will remove the data from the table.

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/tables/table?purge=true

There are two ways to remove a table: dropTable and purgeTable:

  • dropTable removes both the metadata and the directory associated with the table from the file system if the table is not an external table. In case of an external table, only the associated metadata is removed.
  • purgeTable completely removes both the metadata and the directory associated with the table and skipping trash, if the table is an external table or the catalogs don't support purge table, UnsupportedOperationException is thrown.

Hive catalog and lakehouse-iceberg catalog supports purgeTable while jdbc-mysql, jdbc-postgresql and lakehouse-paimon catalog doesn't support.

List all tables under a schema

You can list all tables in a schema by sending a GET request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/tables endpoint or just use the Gravitino Java client. The following is an example of listing all the tables in a schema:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/tables