Skip to main content
Version: 1.3.0

Manage Messaging Metadata

Introduction​

This page introduces how to manage messaging metadata using Apache Gravitino. Messaging metadata refers to the topic metadata of the messaging system such as Apache Kafka, Apache Pulsar, Apache RocketMQ, etc. Through Gravitino, you can create, update, delete, and list topics via unified RESTful APIs or Java client.

To use the messaging catalog, make sure that:

Catalog Operations​

Create a Catalog​

tip

For a messaging catalog, you must specify the type as messaging when creating a catalog.

Create a catalog by sending a POST request to the /api/metalakes/{metalake_name}/catalogs endpoint or use the Gravitino Java client. The following is an example of creating a messaging catalog:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "catalog",
  "type": "MESSAGING",
  "comment": "comment",
  "provider": "kafka",
  "properties": {
    "bootstrap.servers": "localhost:9092",
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs

Gravitino supports the following catalog providers:

Catalog providerCatalog property
kafkaKafka catalog property

Load a Catalog​

Refer to Load a catalog in relational catalog for more details. For a messaging catalog, the load operation is the same.

Alter a Catalog​

Refer to Alter a catalog in relational catalog for more details. For a messaging catalog, the alter operation is the same.

Drop a Catalog​

Refer to Drop a catalog in relational catalog for more details. For a messaging catalog, the drop operation is the same.

List All Catalogs in a Metalake​

Refer to List all catalogs in a metalake in relational catalog for more details. For a messaging catalog, the list operation is the same.

List All Catalog Information in a Metalake​

Refer to List all catalog information in a metalake in relational catalog for more details. For a messaging catalog, the list operation is the same.

Schema Operations​

Schema is a logical grouping of topics in a messaging catalog, if the messaging system does not support topics grouping, schema operations are not supported but a "default" schema will be automatically created to include all topics

note

Gravitino supports only the Kafka catalog. Since Kafka does not support topic grouping, only list and load operations are supported for schema.

Create a Schema​

Refer to Create a schema in relational catalog for more details. For a messaging catalog, the create operation is the same.

Load a Schema​

Refer to Load a schema in relational catalog for more details. For a messaging catalog, the load operation is the same.

Alter a Schema​

Refer to Alter a schema in relational catalog for more details. For a messaging catalog, the alter operation is the same.

Drop a Schema​

Refer to Drop a schema in relational catalog for more details. For a messaging catalog, the drop operation is the same.

List All Schemas Under a Catalog​

Refer to List all schemas under a catalog in relational catalog for more details. For a messaging catalog, the list operation is the same.

Topic Operations​

tip

Users should create a metalake, a catalog and a schema, then ensure that the metalake and catalog are enabled before operating topics.

Create a Topic​

Create a topic by sending a POST request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/topics endpoint or use the Gravitino Java client. The following is an example of creating a topic:

curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "name": "example_topic",
  "comment": "This is an example topic",
  "properties": {
    "partition-count": "3",
    "replication-factor": 1
  }
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/default/topics

Alter a Topic​

Modify a topic by sending a PUT request to the /api/metalakes/{metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/topics/{topic_name} endpoint or use the Gravitino Java client. The following is an example of altering a topic:

curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" -d '{
  "updates": [
    {
      "@type": "removeProperty",
      "property": "key2"
    }, {
      "@type": "setProperty",
      "property": "key3",
      "value": "value3"
    }
  ]
}' http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/default/topics/topic

Gravitino supports the following changes to a topic:

Supported modificationJSONJava
Update a comment{"@type":"updateComment","newComment":"new_comment"}TopicChange.updateComment("new_comment")
Set a topic property{"@type":"setProperty","property":"key1","value":"value1"}TopicChange.setProperty("key1", "value1")
Remove a topic property{"@type":"removeProperty","property":"key1"}TopicChange.removeProperty("key1")

Drop a Topic​

Remove a topic by sending a DELETE request to the /api/metalakes/{metalake_name} /catalogs/{catalog_name}/schemas/{schema_name}/topics/{topic_name} endpoint or by using the Gravitino Java client. The following is an example of dropping a topic:

curl -X DELETE -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/default/topics/topic

List All Topics Under a Schema​

List all topics in a schema by sending a GET request to the /api/metalakes/ {metalake_name}/catalogs/{catalog_name}/schemas/{schema_name}/topics endpoint or by using the Gravitino Java client. The following is an example of listing all the topics in a schema:

curl -X GET -H "Accept: application/vnd.gravitino.v1+json" \
-H "Content-Type: application/json" \
http://localhost:8090/api/metalakes/metalake/catalogs/catalog/schemas/schema/topics