Skip to main content
Version: 0.9.0-incubating

Manage model metadata using Gravitino

This page introduces how to manage model metadata in Apache Gravitino. Gravitino model catalog is a kind of model registry, which provides the ability to manage machine learning models' versioned metadata. It follows the typical Gravitino 3-level namespace (catalog, schema, and model) and supports managing the versions for each model.

Currently, it supports model and model version registering, listing, loading, and deleting.

To use the model catalog, please make sure that:

Catalog operations​

Create a catalog​

info

For a model catalog, you must specify the catalog type as MODEL when creating the catalog. Please also be aware that the provider is not required for a model catalog.

You can create a catalog by sending a POST request to the /api/metalakes/{metalake_name}/catalogs endpoint or just use the Gravitino Java/Python client. The following is an example of creating a catalog:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "model_catalog",
  "type": "MODEL",
  "comment": "This is a model catalog",
  "properties": {
    "k1": "v1"
  }
}' http://localhost:8090/api/metalakes/example/catalogs

Load a catalog​

Refer to Load a catalog in relational catalog for more details. For a model catalog, the load operation is the same.

Alter a catalog​

Refer to Alter a catalog in relational catalog for more details. For a model catalog, the alter operation is the same.

Drop a catalog​

Refer to Drop a catalog in relational catalog for more details. For a model catalog, the drop operation is the same.

List all catalogs in a metalake​

Please refer to List all catalogs in a metalake in relational catalog for more details. For a model catalog, the list operation is the same.

List all catalogs' information in a metalake​

Please refer to List all catalogs' information in a metalake in relational catalog for more details. For a model catalog, the list operation is the same.

Schema operations​

Schema is a virtual namespace in a model catalog, which is used to organize the models. It is similar to the concept of schema in the relational catalog.

tip

Users should create a metalake and a catalog before creating a schema.

Create a schema​

You can create a schema by sending a POST request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas endpoint or just use the Gravitino Java/Python client. The following is an example of creating a schema:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "model_schema",
  "comment": "This is a model schema",
  "properties": {
    "k1": "v1"
  }
}' http://localhost:8090/api/metalakes/example/catalogs/model_catalog/schemas

Load a schema​

Please refer to Load a schema in relational catalog for more details. For a model catalog, the schema load operation is the same.

Alter a schema​

Please refer to Alter a schema in relational catalog for more details. For a model catalog, the schema alter operation is the same.

Drop a schema​

Please refer to Drop a schema in relational catalog for more details. For a model catalog, the schema drop operation is the same.

Note that the drop operation will delete all the model metadata under this schema if cascade set to true.

List all schemas under a catalog​

Please refer to List all schemas under a catalog in relational catalog for more details. For a model catalog, the schema list operation is the same.

Model operations​

tip
  • Users should create a metalake, a catalog, and a schema before creating a model.

Register a model​

You can register a model by sending a POST request to the /api/metalakes/{metalake_name} /catalogs/{catalog_name}/schemas/{schema_name}/models endpoint or just use the Gravitino Java/Python client. The following is an example of creating a model:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "example_model",
  "comment": "This is an example model",
  "properties": {
    "k1": "v1"
  }
}' http://localhost:8090/api/metalakes/example/catalogs/model_catalog/schemas/model_schema/models

Get a model​

You can get a model by sending a GET request to the /api/metalakes/{metalake_name} /catalogs/{catalog_name}/schemas/{schema_name}/models/{model_name} endpoint or by using the Gravitino Java/Python client. The following is an example of getting a model:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/example/catalogs/model_catalog/schemas/model_schema/models/example_model

Alter a model​

You can modify a model's metadata (e.g. rename, update comment, or modify properties) by sending a PUT request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/ {schema_name}/models/{model_name} endpoint or using the Gravitino Java/Python client. The following is an example of modifying a model:

cat <<EOF >model.json
{
  "updates": [
    {
      "@type": "updateComment",
      "newComment": "Updated model comment"
    },
    {
      "@type": "rename",
      "newName": "new_name"
    },
    {
      "@type": "setProperty",
      "property": "k2",
      "value": "v2"
    },
    {
      "@type": "removeProperty",
      "property": "k1"
    }
  ]
}
EOF
 
curl -X PUT \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -d '@model.json' \
  http://localhost:8090/api/metalakes/mymetalake/catalogs/mycatalog/schemas/myschema/models/mymodel

Supported modifications​

The following operations are supported for altering a model:

OperationJSON ExampleJava MethodPython Method
Rename model{"@type":"rename","newName":"new_name"}ModelChange.rename("new_name")ModelChange.rename("new_name")
Update comment{"@type":"updateComment","newComment":"new comment"}ModelChange.updateComment("new comment")ModelChange.update_comment("new comment")
Set property{"@type":"setProperty","property":"key","value":"value"}ModelChange.setProperty("key", "value")ModelChange.set_property("key", "value")
Remove property{"@type":"removeProperty","property":"key"}ModelChange.removeProperty("key")ModelChange.remove_property("key")
note
  • Multiple modifications can be applied in a single request.
  • If the target model does not exist, a 404 Not Found error will be returned.

Delete a model​

You can delete a model by sending a DELETE request to the /api/metalakes/{metalake_name} /catalogs/{catalog_name}/schemas/{schema_name}/models/{model_name} endpoint or by using the Gravitino Java/Python client. The following is an example of deleting a model:

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/example/catalogs/model_catalog/schemas/model_schema/models/example_model

Note that the delete operation will delete all the model versions under this model.

List models​

You can list all the models in a schema by sending a GET request to the /api/metalakes/ {metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/models endpoint or by using the Gravitino Java/Python client. The following is an example of listing all the models in a schema:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/example/catalogs/model_catalog/schemas/model_schema/models

ModelVersion operations​

tip
  • Users should create a metalake, a catalog, a schema, and a model before link a model version to the model.

You can link a ModelVersion by sending a POST request to the /api/metalakes/{metalake_name} /catalogs/{catalog_name}/schemas/{schema_name}/models/{model_name}/versions endpoint or by using the Gravitino Java/Python client. The following is an example of linking a ModelVersion:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "uri": "path/to/model",
  "aliases": ["alias1", "alias2"],
  "comment": "This is version 0",
  "properties": {
    "k1": "v1"
  }
}' http://localhost:8090/api/metalakes/example/catalogs/model_catalog/schemas/model_schema/models/example_model/versions

The comment and properties of ModelVersion can be different from the model.

Get a ModelVersion​

You can get a ModelVersion by sending a GET request to the /api/metalakes/{metalake_name} /catalogs/{catalog_name}/schemas/{schema_name}/models/{model_name}/versions/{version_number} endpoint or by using the Gravitino Java/Python client. The following is an example of getting a ModelVersion:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/example/catalogs/model_catalog/schemas/model_schema/models/example_model/versions/0

Get a ModelVersion by alias​

You can also get a ModelVersion by sending a GET request to the /api/metalakes/{metalake_name} /catalogs/{catalog_name}/schemas/{schema_name}/models/{model_name}/aliases/{alias} endpoint or by using the Gravitino Java/Python client. The following is an example of getting a ModelVersion by alias:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/example/catalogs/model_catalog/schemas/model_schema/models/example_model/aliases/alias1

Alter a ModelVersion​

You can modify a modelVersion's metadata (e.g. update uri, update comment, or modify properties) by sending a PUT request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name} /schemas/{schema_name} /models/{model_name}/versions/{version_number} endpoint or using the Gravitino Java/Python client. The following is an example of modifying a model version:

cat <<EOF >model.json
{
  "updates": [
    {
      "@type": "updateComment",
      "newComment": "Updated comment of model version"
    },
    {
      "@type": "updateUri",
      "newUri": "new_uri"
    },
    {
      "@type": "setProperty",
      "property": "key",
      "value": "value"
    },
    {
      "@type": "removeProperty",
      "property": "key"
    }
  ]
}
EOF
 
curl -X PUT \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -d '@model.json' \
  http://localhost:8090/api/metalakes/mymetalake/catalogs/mycatalog/schemas/myschema/models/mymodel/versions/0

Supported modifications​

OperationJSON ExampleJava MethodPython Method
Update uri{"@type":"updateUri","newName":"new_uri"}ModelVersionChange.updateUri("new_uri")ModelVersionChange.update_uri("new_uri")
Update comment{"@type":"updateComment","newComment":"new_comment"}ModelVersionChange.updateComment("new_comment")ModelVersionChange.update_comment("new_comment")
Set property{"@type":"setProperty","property":"key","value":"value"}ModelVersionChange.setProperty("key", "value")ModelVersionChange.set_property("key", "value")
Remove property{"@type":"removeProperty","property":"key"}ModelVersionChange.removeProperty("key")ModelVersionChange.remove_property("key")
note
  • Multiple modifications can be applied in a single request.
  • If the target model does not exist, a 404 Not Found error will be returned.
  • If the target model version does not exist, a 404 Not Found error will be returned.

Alter a ModelVersion by alias​

You can also modify a modelVersion's metadata (e.g. update uri, update comment, or modify properties) by sending a PUT request to the /api/metalakes/{metalake_name}/catalogs/ {catalog_name}/schemas/{schema_name}/models/{model_name}/aliases/{alias} endpoint or using the Gravitino Java/Python client. The following is an example of modifying a model version:

cat <<EOF >model.json
{
  "updates": [
    {
      "@type": "updateComment",
      "newComment": "Updated comment of model version"
    },
    {
      "@type": "updateUri",
      "newUri": "new_uri"
    },
    {
      "@type": "setProperty",
      "property": "key",
      "value": "value"
    },
    {
      "@type": "removeProperty",
      "property": "key"
    }
  ]
}
EOF
 
curl -X PUT \
  -H "Accept: application/vnd.gravitino.v1+json" \
  -H "Content-Type: application/json" \
  -d '@model.json' \
  http://localhost:8090/api/metalakes/mymetalake/catalogs/mycatalog/schemas/myschema/models/mymodel/aliases/myalias

Supported modifications​

OperationJSON ExampleJava MethodPython Method
Update uri{"@type":"updateUri","newName":"new_uri"}ModelVersionChange.updateUri("new_uri")ModelVersionChange.update_uri("new_uri")
Update comment{"@type":"updateComment","newComment":"new_comment"}ModelVersionChange.updateComment("new_comment")ModelVersionChange.update_comment("new_comment")
Set property{"@type":"setProperty","property":"key","value":"value"}ModelVersionChange.setProperty("key", "value")ModelVersionChange.set_property("key", "value")
Remove property{"@type":"removeProperty","property":"key"}ModelVersionChange.removeProperty("key")ModelVersionChange.remove_property("key")
note
  • Multiple modifications can be applied in a single request.
  • If the target model does not exist, a 404 Not Found error will be returned.
  • If the target model version does not exist, a 404 Not Found error will be returned.

Delete a ModelVersion​

You can delete a ModelVersion by sending a DELETE request to the /api/metalakes/{metalake_name} /catalogs/{catalog_name}/schemas/{schema_name}/models/{model_name}/versions/{version_number} endpoint or by using the Gravitino Java/Python client. The following is an example of deleting a ModelVersion:

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/example/catalogs/model_catalog/schemas/model_schema/models/example_model/versions/0

Delete a ModelVersion by alias​

You can also delete a ModelVersion by sending a DELETE request to the /api/metalakes/ {metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/models/{model_name}/aliases/{alias} endpoint or by using the Gravitino Java/Python client. The following is an example of deleting a ModelVersion by alias:

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/example/catalogs/model_catalog/schemas/model_schema/models/example_model/aliases/alias1

List ModelVersions​

You can list all the ModelVersions in a model by sending a GET request to the /api/metalakes/ {metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/models/{model_name}/versions endpoint or by using the Gravitino Java/Python client. The following is an example of listing all the ModelVersions in a model:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/example/catalogs/model_catalog/schemas/model_schema/models/example_model/versions