Skip to main content
Version: 0.6.1-incubating

Manage relational metadata using Apache Gravitino

This page introduces how to manage relational metadata by Apache Gravitino, relational metadata refers to relational catalog, schema, table and partitions. Through Gravitino, you can create, edit, and delete relational metadata via unified REST APIs or Java client.

In this document, Gravitino uses Apache Hive catalog as an example to show how to manage relational metadata by Gravitino. Other relational catalogs are similar to Hive catalog, but they may have some differences, especially in catalog property, table property, and column type. For more details, please refer to the related doc.

Assuming:

Catalog operations​

Create a catalog​

tip

The code below is an example of creating a Hive catalog. For other relational catalogs, the code is similar, but the catalog type, provider, and properties may be different. For more details, please refer to the related doc.

For relational catalog, you must specify the catalog type as RELATIONAL when creating a catalog.

You can create a catalog by sending a POST request to the /api/metalakes/{metalake_name}/catalogs endpoint or just use the Gravitino Java client. The following is an example of creating a catalog:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "catalog",
  "type": "RELATIONAL",
  "comment": "comment",
  "provider": "hive",
  "properties": {
    "metastore.uris": "thrift://localhost:9083"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs

Currently, Gravitino supports the following catalog providers:

Catalog providerCatalog property
hiveHive catalog property
lakehouse-icebergIceberg catalog property
lakehouse-paimonPaimon catalog property
jdbc-mysqlMySQL catalog property
jdbc-postgresqlPostgreSQL catalog property
jdbc-dorisDoris catalog property

Load a catalog​

You can load a catalog by sending a GET request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name} endpoint or just use the Gravitino Java client. The following is an example of loading a catalog:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" http://localhost:8090/api/metalakes/metalake/catalogs/catalog

Alter a catalog​

You can modify a catalog by sending a PUT request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name} endpoint or just use the Gravitino Java client. The following is an example of altering a catalog:

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "updates": [
    {
      "@type": "rename",
      "newName": "alter_catalog"
    },
    {
      "@type": "setProperty",
      "property": "key3",
      "value": "value3"
    }
  ]
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog

Currently, Gravitino supports the following changes to a catalog:

Supported modificationJSONJava
Rename catalog{"@type":"rename","newName":"catalog_renamed"}CatalogChange.rename("catalog_renamed")
Update comment{"@type":"updateComment","newComment":"new_comment"}CatalogChange.updateComment("new_comment")
Set a property{"@type":"setProperty","property":"key1","value":"value1"}CatalogChange.setProperty("key1", "value1")
Remove a property{"@type":"removeProperty","property":"key1"}CatalogChange.removeProperty("key1")
warning

Most catalog-altering operations are generally safe. However, if you want to change the catalog's URI, you should proceed with caution. Changing the URI may point to a different cluster, rendering the metadata stored in Gravitino unusable. For instance, if the old URI and the new URI point to different clusters that both have a database named db1, changing the URI might cause the old metadata, such as audit information, to be used when accessing db1, which is undesirable.

Therefore, do not change the catalog's URI unless you fully understand the consequences of such a modification.

Drop a catalog​

You can remove a catalog by sending a DELETE request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name} endpoint or just use the Gravitino Java client. The following is an example of dropping a catalog:

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog
note

Dropping a catalog only removes metadata about the catalog, schemas, and tables under the catalog in Gravitino, It doesn't remove the real data (table and schema) in Apache Hive.

List all catalogs in a metalake​

You can list all catalogs under a metalake by sending a GET request to the /api/metalakes/{metalake_name}/catalogs endpoint or just use the Gravitino Java client. The following is an example of listing all the catalogs in a metalake:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs

List all catalogs' information in a metalake​

You can list all catalogs' information under a metalake by sending a GET request to the /api/metalakes/{metalake_name}/catalogs?details=true endpoint or just use the Gravitino Java client. The following is an example of listing all the catalogs' information in a metalake:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs?details=true

Schema operations​

tip

Users should create a metalake and a catalog before creating a schema.

Create a schema​

You can create a schema by sending a POST request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas endpoint or just use the Gravitino Java client. The following is an example of creating a schema:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "schema",
  "comment": "comment",
  "properties": {
    "key1": "value1"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas

Currently, Gravitino supports the following schema property:

Catalog providerSchema property
hiveHive schema property
lakehouse-icebergIceberg scheme property
lakehouse-paimonPaimon scheme property
jdbc-mysqlMySQL schema property
jdbc-postgresqlPostgreSQL schema property
jdbc-dorisDoris schema property

Load a schema​

You can create a schema by sending a GET request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name} endpoint or just use the Gravitino Java client. The following is an example of loading a schema:

curl -X GET \-H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema

Alter a schema​

You can change a schema by sending a PUT request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name} endpoint or just use the Gravitino Java client. The following is an example of modifying a schema:

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "updates": [
    {
      "@type": "removeProperty",
      "property": "key2"
    }, {
      "@type": "setProperty",
      "property": "key3",
      "value": "value3"
    }
  ]
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema

Currently, Gravitino supports the following changes to a schema:

Supported modificationJSONJava
Set a property{"@type":"setProperty","property":"key1","value":"value1"}SchemaChange.setProperty("key1", "value1")
Remove a property{"@type":"removeProperty","property":"key1"}SchemaChange.removeProperty("key1")

Drop a schema​

You can remove a schema by sending a DELETE request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name} endpoint or just use the Gravitino Java client. The following is an example of dropping a schema:

// cascade can be true or false
curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema?cascade=true

If cascade is true, Gravitino will drop all tables under the schema. Otherwise, Gravitino will throw an exception if there are tables under the schema. Some catalogs may not support cascading deletion of a schema, please refer to the related doc for more details.

List all schemas under a catalog​

You can list all schemas under a catalog by sending a GET request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas endpoint or just use the Gravitino Java client. The following is an example of listing all the schemas in a catalog:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas

Table operations​

tip

Users should create a metalake, a catalog and a schema before creating a table.

Create a table​

You can create a table by sending a POST request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/tables endpoint or just use the Gravitino Java client. The following is an example of creating a table:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "example_table",
  "comment": "This is an example table",
  "columns": [
    {
      "name": "id",
      "type": "integer",
      "comment": "id column comment",
      "nullable": false,
      "autoIncrement": true,
      "defaultValue": {
        "type": "literal",
        "dataType": "integer",
        "value": "-1"
      }
    },
    {
      "name": "name",
      "type": "varchar(500)",
      "comment": "name column comment",
      "nullable": true,
      "autoIncrement": false,
      "defaultValue": {
        "type": "literal",
        "dataType": "null",
        "value": "null"
      }
    },
    {
      "name": "StartingDate",
      "type": "timestamp",
      "comment": "StartingDate column comment",
      "nullable": false,
      "autoIncrement": false,
      "defaultValue": {
        "type": "function",
        "funcName": "current_timestamp",
        "funcArgs": []
      }
    },
    {
      "name": "info",
      "type": {
        "type": "struct",
        "fields": [
          {
            "name": "position",
            "type": "string",
            "nullable": true,
            "comment": "position field comment"
          },
          {
            "name": "contact",
            "type": {
              "type": "list",
              "elementType": "integer",
              "containsNull": false
            },
            "nullable": true,
            "comment": "contact field comment"
          },
          {
            "name": "rating",
            "type": {
              "type": "map",
              "keyType": "string",
              "valueType": "integer",
              "valueContainsNull": false
            },
            "nullable": true,
            "comment": "rating field comment"
          }
        ]
      },
      "comment": "info column comment",
      "nullable": true
    },
    {
      "name": "dt",
      "type": "date",
      "comment": "dt column comment",
      "nullable": true
    }
  ],
  "partitioning": [
    {
      "strategy": "identity",
      "fieldName": [ "dt" ]
    }
  ],
  "distribution": {
    "strategy": "hash",
    "number": 32,
    "funcArgs": [
      {
        "type": "field",
        "fieldName": [ "id" ]
      }
    ]
  },
  "sortOrders": [
    {
      "sortTerm": {
        "type": "field",
        "fieldName": [ "age" ]
      },
      "direction": "asc",
      "nullOrdering": "nulls_first"
    }
  ],
  "indexes": [
    {
      "indexType": "primary_key",
      "name": "PRIMARY",
      "fieldNames": [["id"]]
    }
  ],
  "properties": {
    "format": "ORC"
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/tables
caution

The provided example demonstrates table creation but isn't directly executable in Gravitino, since not all catalogs fully support these capabilities.

In order to create a table, you need to provide the following information:

  • Table column name and type
  • Table column default value (optional)
  • Table column auto-increment (optional)
  • Table property (optional)

Apache Gravitino table column type​

The following types that Gravitino supports:

TypeJavaJSONDescription
BooleanTypes.BooleanType.get()booleanBoolean type
ByteTypes.ByteType.get()byteByte type, indicates a numerical value of 1 byte
Unsigned ByteTypes.ByteType.unsigned()byte unsignedUnsigned Byte type, indicates a unsigned numerical value of 1 byte
ShortTypes.ShortType.get()shortShort type, indicates a numerical value of 2 bytes
Unsigned ShortTypes.ShortType.unsigned()short unsignedUnsigned Short type, indicates a unsigned numerical value of 2 bytes
IntegerTypes.IntegerType.get()integerInteger type, indicates a numerical value of 4 bytes
Unsigned IntegerTypes.IntegerType.unsigned()integer unsignedUnsigned Integer type, indicates a unsigned numerical value of 4 bytes
LongTypes.LongType.get()longLong type, indicates a numerical value of 8 bytes
Unsigned LongTypes.LongType.unsigned()long unsignedUnsigned Long type, indicates a unsigned numerical value of 8 bytes
FloatTypes.FloatType.get()floatFloat type, indicates a single-precision floating point number
DoubleTypes.DoubleType.get()doubleDouble type, indicates a double-precision floating point number
Decimal(precision, scale)Types.DecimalType.of(precision, scale)decimal(p, s)Decimal type, indicates a fixed-precision decimal number with the constraint that the precision must be in range [1, 38] and the scala must be in range [0, precision]
StringTypes.StringType.get()stringString type
FixedChar(length)Types.FixedCharType.of(length)char(l)Char type, indicates a fixed-length string
VarChar(length)Types.VarCharType.of(length)varchar(l)Varchar type, indicates a variable-length string, the length is the maximum length of the string
TimestampTypes.TimestampType.withoutTimeZone()timestampTimestamp type, indicates a timestamp without timezone
TimestampWithTimezoneTypes.TimestampType.withTimeZone()timestamp_tzTimestamp with timezone type, indicates a timestamp with timezone
DateTypes.DateType.get()dateDate type
TimeTypes.TimeType.withoutTimeZone()timeTime type
IntervalToYearMonthTypes.IntervalYearType.get()interval_yearInterval type, indicates an interval of year and month
IntervalToDayTimeTypes.IntervalDayType.get()interval_dayInterval type, indicates an interval of day and time
Fixed(length)Types.FixedType.of(length)fixed(l)Fixed type, indicates a fixed-length binary array
BinaryTypes.BinaryType.get()binaryBinary type, indicates a arbitrary-length binary array
ListTypes.ListType.of(elementType, elementNullable){"type": "list", "containsNull": JSON Boolean, "elementType": type JSON}List type, indicate a list of elements with the same type
MapTypes.MapType.of(keyType, valueType){"type": "map", "keyType": type JSON, "valueType": type JSON, "valueContainsNull": JSON Boolean}Map type, indicate a map of key-value pairs
StructTypes.StructType.of([Types.StructType.Field.of(name, type, nullable)]){"type": "struct", "fields": [JSON StructField, {"name": string, "type": type JSON, "nullable": JSON Boolean, "comment": string}]}Struct type, indicate a struct of fields
UnionTypes.UnionType.of([type1, type2, ...]){"type": "union", "types": [type JSON, ...]}Union type, indicates a union of types
UUIDTypes.UUIDType.get()uuidUUID type, indicates a universally unique identifier

The related java doc is here.

External type​

External type is a special type of column type, when you need to use a data type that is not in the Gravitino type system, and you explicitly know its string representation in an external catalog (usually used in JDBC catalogs), then you can use the ExternalType to represent the type. Similarly, if the original type is unsolvable, it will be represented by ExternalType. The following shows the data structure of an external type in JSON and Java, enabling easy retrieval of its string value.

{
  "type": "external",
  "catalogString": "user-defined"
}
Unparsed type​

Unparsed type is a special type of column type, it used to address compatibility issues in type serialization and deserialization between the server and client. For instance, if a new column type is introduced on the Gravitino server that the client does not recognize, it will be treated as an unparsed type on the client side. The following shows the data structure of an unparsed type in JSON and Java, enabling easy retrieval of its value.

{
  "type": "unparsed",
  "unparsedType": "unknown-type"
}

Table column default value​

When defining a table column, you can specify a literal or an expression as the default value. The default value typically applies to new rows that are inserted into the table by the underlying catalog.

The following is a table of the column default value that Gravitino supports for different catalogs:

Catalog providerSupported default value
hive✘
lakehouse-iceberg✘
lakehouse-paimon✘
jdbc-mysql✔
jdbc-postgresql✔

Table column auto-increment​

Auto-increment provides a convenient way to ensure that each row in a table has a unique identifier without the need for manually managing identifier allocation. The following table shows the column auto-increment that Gravitino supports for different catalogs:

Catalog providerSupported auto-increment
hive✘
lakehouse-iceberg✘
lakehouse-paimon✘
jdbc-mysql✔(limitations)
jdbc-postgresql✔

Table property and type mapping​

The following is the table property that Gravitino supports:

Catalog providerTable propertyType mapping
hiveHive table propertyHive type mapping
lakehouse-icebergIceberg table propertyIceberg type mapping
lakehouse-paimonPaimon table propertyPaimon type mapping
jdbc-mysqlMySQL table propertyMySQL type mapping
jdbc-postgresqlPostgreSQL table propertyPostgreSQL type mapping
dorisDoris table propertyDoris type mapping

Table partitioning, distribution, sort ordering and indexes​

In addition to the basic settings, Gravitino supports the following features:

FeatureDescriptionJava doc
Table partitioningEqual to PARTITION BY in Apache Hive, It is a partitioning strategy that is used to split a table into parts based on partition keys. Some table engine may not support this featurePartition
Table distributionEqual to CLUSTERED BY in Apache Hive, distribution a.k.a (Clustering) is a technique to split the data into more manageable files/parts, (By specifying the number of buckets to create). The value of the distribution column will be hashed by a user-defined number into buckets.Distribution
Table sort orderingEqual to SORTED BY in Apache Hive, sort ordering is a method to sort the data in specific ways such as by a column or a function, and then store table data. it will highly improve the query performance under certain scenarios.SortOrder
Table indexesEqual to KEY/INDEX in MySQL , unique key enforces uniqueness of values in one or more columns within a table. It ensures that no two rows have identical values in specified columns, thereby facilitating data integrity and enabling efficient data retrieval and manipulation operations.Index

For more information, please see the related document on partitioning, bucketing, sorting, and indexes.

note

The code above is an example of creating a Hive table. For other catalogs, the code is similar, but the supported column type, and table properties may be different. For more details, please refer to the related doc.

Load a table​

You can load a table by sending a GET request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/tables/{table_name} endpoint or just use the Gravitino Java client. The following is an example of loading a table:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json"  \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/tables/table
note
  • When Gravitino loads a table from a catalog with various data types, if Gravitino is unable to parse the data type, it will use an External Type to preserve the original data type, ensuring that the table can be loaded successfully.
  • When Gravitino loads a table from a catalog that supports default value, if Gravitino is unable to parse the default value, it will use an Unparsed Expression to preserve the original default value, ensuring that the table can be loaded successfully.

Alter a table​

You can modify a table by sending a PUT request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/tables/{table_name} endpoint or just use the Gravitino Java client. The following is an example of modifying a table:

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "updates": [
    {
      "@type": "removeProperty",
      "property": "key2"
    }, {
      "@type": "setProperty",
      "property": "key3",
      "value": "value3"
    }
  ]  
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/tables/table

Currently, Gravitino supports the following changes to a table:

Supported modificationJSONJava
Rename table{"@type":"rename","newName":"table_renamed"}TableChange.rename("table_renamed")
Update comment{"@type":"updateComment","newComment":"new_comment"}TableChange.updateComment("new_comment")
Set a table property{"@type":"setProperty","property":"key1","value":"value1"}TableChange.setProperty("key1", "value1")
Remove a table property{"@type":"removeProperty","property":"key1"}TableChange.removeProperty("key1")
Add a column{"@type":"addColumn","fieldName":["position"],"type":"varchar(20)","comment":"Position of user","position":"FIRST","nullable": true, "autoIncrement": false, "defaultValue" : {"type": "literal", "dataType": "varchar(20)", "value": "Default Position"}}TableChange.addColumn(...)
Delete a column{"@type":"deleteColumn","fieldName": ["name"], "ifExists": true}TableChange.deleteColumn(...)
Rename a column{"@type":"renameColumn","oldFieldName":["name_old"], "newFieldName":"name_new"}TableChange.renameColumn(...)
Update the column comment{"@type":"updateColumnComment", "fieldName": ["name"], "newComment": "new comment"}TableChange.updateColumnCommment(...)
Update the type of a column{"@type":"updateColumnType","fieldName": ["name"], "newType":"varchar(100)"}TableChange.updateColumnType(...)
Update the nullability of a column{"@type":"updateColumnNullability","fieldName": ["name"],"nullable":true}TableChange.updateColumnNullability(...)
Update the position of a column{"@type":"updateColumnPosition","fieldName": ["name"], "newPosition":"default"}TableChange.updateColumnPosition(...)
Update the default value of a column{"@type":"updateColumnDefaultValue","fieldName": ["name"], "newDefaultValue":{"type":"literal","dataType":"varchar(100)","value":"new default value}}TableChange.updateColumnDefaultValue(...)

Drop a table​

You can remove a table by sending a DELETE request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/tables/{table_name} endpoint or just use the Gravitino Java client. The following is an example of dropping a table:

## Purge can be true or false, if purge is true, Gravitino will remove the data from the table.

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/tables/table?purge=true

There are two ways to remove a table: dropTable and purgeTable:

  • dropTable removes both the metadata and the directory associated with the table from the file system if the table is not an external table. In case of an external table, only the associated metadata is removed.
  • purgeTable completely removes both the metadata and the directory associated with the table and skipping trash, if the table is an external table or the catalogs don't support purge table, UnsupportedOperationException is thrown.

Hive catalog and lakehouse-iceberg catalog supports purgeTable while jdbc-mysql, jdbc-postgresql and lakehouse-paimon catalog doesn't support.

List all tables under a schema​

You can list all tables in a schema by sending a GET request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/tables endpoint or just use the Gravitino Java client. The following is an example of listing all the tables in a schema:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/tables