Skip to main content

11 posts tagged with "model"

View All Tags

Apache Gravitino 1.3.1

· 14 min read
Bharath Krishna
Bharath Krishna
committer

We are glad to announce the release of Apache Gravitino 1.3.1! This is a patch release that focuses on stability, correctness, and security hardening. It carries a large number of fixes across the core server, authorization, Iceberg REST, the Trino, Spark and Flink connectors, and the ClickHouse, Doris, Glue, Hive, JDBC and Lance catalogs. It also completes tag support for views and functions across the Java and Python clients, adds view APIs to the Python client, and introduces extension SPIs that let downstream projects plug in their own catalog adapters.

Although 1.3.1 is a patch release, the Trino and Spark connectors contain several behavior changes that require action before upgrading from 1.3.0. Please read the section below first.

Upgrade Notes / Behavior Changes​

Each of the changes below leaves a cluster looking healthy immediately after the upgrade and fails later, so review them before upgrading rather than after.

Trino connector​

  • Non-REST lakehouse-iceberg catalogs are now routed through the Gravitino Iceberg REST service (IRC) by default. #12554 A metalake with catalog-backend=jdbc or hive Iceberg catalogs changes behavior on upgrade: the connector now builds an internal Iceberg REST catalog against Gravitino's IRC instead of translating catalog-backend into Trino's jdbc/hive_metastore catalog type. Set gravitino.iceberg.rest-routing-enabled=false to keep the previous behavior.

  • If the IRC has authentication enabled, it now requires its own credential, configured separately on every Trino node. #12554 This credential is not derived from Gravitino catalog properties and does not travel with the generated CREATE CATALOG statement. It must be set locally in catalog.properties (or through Trino's ${ENV:...} secret substitution) on the coordinator and every worker before upgrading:

    gravitino.iceberg.rest-catalog.security=OAUTH2
    gravitino.iceberg.rest-catalog.oauth2.credential=${ENV:IRC_OAUTH_CREDENTIAL}
    gravitino.iceberg.rest-catalog.oauth2.server-uri=http://your-idp/token
    gravitino.iceberg.rest-catalog.oauth2.scope=email

    Missing this on any node does not fail CREATE CATALOG — the catalog registers and lists normally. It surfaces only as an authentication error the first time a query on that node reaches the IRC.

  • The internal JDBC connection credential (trino.jdbc.password) is no longer propagated into the catalogs the connector creates. #12542 It no longer reaches connector logs or the generated catalog properties file.

Spark connector​

  • Non-REST lakehouse-iceberg catalogs are now routed through the IRC by default, matching Trino. #12709 Set spark.sql.gravitino.iceberg.rest-routing-enabled=false to keep the previous behavior.

  • When no IRC endpoint can be discovered, the connector now fails with an actionable error instead of silently falling back to Hive/JDBC translation. #12709 Set spark.sql.gravitino.iceberg.rest-routing-enabled=false to restore the old fallback.

  • If the Gravitino client uses OAuth2, the connector reuses that configuration for the IRC connection by default (spark.sql.gravitino.iceberg.reuseOAuth2=true). #12709 Set reuseOAuth2=false to configure the IRC's credential separately.

  • An unconstrained PostgreSQL NUMERIC column (no precision or scale) is now read as a string instead of Decimal(38, 18). #13040 Queries performing arithmetic, aggregation, or comparison on such a column need an explicit cast after upgrading.

Improvements​

Core & Server​

  • Locking: Lock the entity node instead of the schema when creating tables, topics, views and models. #13206
  • Service discovery: Support an advertised URI for Iceberg REST service discovery. #13098
  • Connection testing: Allow catalog users to test existing catalog connections with stored configuration. #13085
  • Health reporting: Report unhealthy status after observed out-of-memory errors. #13066
  • Job status: Use entityStore.update() for job status transitions. #12669
  • Ownership: Support group owners in batchGetOwner. #12532
  • Policies: Support FUNCTION and VIEW metadata object policies. #12501
  • Catalogs: Support connection tests with proposed changes. #12794
  • Catalogs: Support testing existing catalog connections. #12552
  • Indexes: Add a properties field to TableChange.AddIndex for custom index parameters. #11946
  • Security: Harden JDBC unsafe-parameter detection against connectionProperties smuggling. #12024

Authorization​

  • Performance: Optimize list authorization. #11775
  • Performance: Improve the JCasbin permission checks. #12037
  • Diagnostics: Show the full NameIdentifier in denial messages. #12670
  • Refactor: Reuse the shared user and group authorization expressions. #12973
  • Refactor: Reuse the shared table authorization expressions. #12875

Catalogs & Connectors​

  • Trino connector: Support Starburst SPI compatibility. #12526
  • Trino connector: Add a CatalogConnectorAdapterProvider SPI for external adapters. #13211
  • Trino connector: Make gravitino.metalake optional. #13301
  • Trino connector: Defer Iceberg REST passthrough metadata initialization. #13194
  • Trino connector: Report catalog registration status through system tables. #12546
  • Trino connector: Route lakehouse-iceberg catalogs through the Iceberg REST server. #12554
  • Trino connector: Support TLS and session role for the internal JDBC connection. #12542
  • Trino connector: Extend session user forwarding to OAuth2, with per-user credential vending for IRC-backed Iceberg. #12045
  • Trino connector: Log via io.airlift.log.Logger. #12634
  • Spark connector: Add a SparkCatalogExtension SPI for external catalogs. #13208
  • Spark connector: Route lakehouse-iceberg catalogs through the Iceberg REST server. #12709
  • Flink connector: Recognize catalog types from external BaseCatalogFactory jars. #13213
  • Flink connector: Allow catalogs to customize Flink type conversion. #12961
  • JDBC catalog: Fail fast with a clear error when the JDBC driver is missing. #13192
  • ClickHouse: Support setting the data skipping index type. #11912
  • Doris: Upgrade the type system for Doris 3.0+/4.0.x compatibility. #11590

Clients​

  • Java client: Support tags for views and functions. #11902
  • Python client: Add view API definitions. #12039
  • Python client: Implement client-side view models. #12089
  • Python client: Add view create/drop operations. #12116
  • Python client: Add view query/alter operations. #12158
  • Python client: Support tags for views and functions. #12181
  • Tags: Support tags for views and functions. #11844

MCP Server​

  • Authentication: Fetch and refresh OAuth client-credentials tokens. #12530
  • Security: Allow the MCP server image to run as a non-root user. #12448

Web UI​

  • Session: Expose UI session timeout settings through server configuration. #12588
  • Login: Support built-in IdP basic login. #11681

Docker & Build​

  • Connector images: Add UBI-based Docker images for the Trino, Flink and Spark connectors. #13265
  • Connector images: Bundle the connector image README into the Trino, Flink and Spark images. #13461
  • Kerberos Hive: Update the kerberos-hive image to 0.1.7. #12730
  • Build: Optimize build test execution. #12583

Documentation​

  • Restructure the fileset catalog pages around a runnable path. #12434
  • Update the security and access control pages. #12391
  • Document the health and readiness endpoints. #12402
  • Update the server configuration and relational backend storage pages. #12291
  • Update the Trino Iceberg REST engine page. #12242
  • Update the Doris catalog documentation for 3.0.x/4.0.x compatibility. #11995
  • Document the Trino connector authType=basic forwardUser limitation. #13472

Bug Fixes​

Core & Server​

  • Keep column ids stable and pass the new schema on cross-schema table rename. #13309
  • Keep column tags consistent across column drop and rename. #13304
  • Clean up relations, owner and privileges when deleting a tag or policy. #13311
  • Validate name and comment length before persisting entities. #13312
  • Close catalogs when force-dropping a metalake. #13296
  • Clean up missing schemas on explicit cascading drops. #13278
  • Batch table column metadata inserts. #13273
  • Keep plugin privileges when the external drop returns false. #13174
  • Track local jobs only by their owning executor in multi-node deployments. #13146
  • Reject Spark jobs at submission when Spark is not available in the local job executor. #13131
  • Add a PostgreSQL group-role batch upsert. #13035
  • Restore event dispatcher ordering. #13007
  • Preserve errors thrown by PrincipalUtils.doAs. #12975
  • Release the ClassLoader of a dropped catalog. #12986
  • Defer catalog wrapper cleanup with an operation lease. #12403
  • Prevent silent authorization updates on closed catalogs. #12405
  • Fix built-in policy supportedObjectTypes validation and error message. #12685
  • Fix fileset schema location resolution when schema properties omit location. #12654
  • Cascade cache invalidation to hierarchical schema descendants. #12416
  • Make the drop-event log throttling atomic. #10169
  • Make PostgreSQL table version soft delete idempotent. #12238
  • Refresh the built-in job template when content drifts even if the version is unchanged. #11976
  • Sort column ordinal positions while fetching from the entity store. #11980
  • Do not wrap plain ClassNotFoundException in IsolatedClassLoader. #12014
  • Tolerate concurrent directory creation instead of failing spuriously. #11939
  • Inject table hook dependencies. #12201

Server & REST API​

  • Avoid exposing missing metalakes in authorization errors. #13360
  • Avoid exposing service admins publicly. #12777
  • Check the object before listing roles. #13326
  • Suppress verbose JCasbin authorization logs. #13270
  • Return 400 for invalid metadata object types. #12727
  • Return accurate HTTP statuses for unsupported operations. #12879
  • Return JSON errors for the whole pre-resource-method WebApplicationException family. #12783
  • Move malformed path-parameter handling to a Jersey ExceptionMapper. #12783
  • Reject null request bodies in the remaining REST operations. #12834
  • Reject null request bodies before validate() in create/register/add operations. #12788
  • Handle null request bodies in createView. #12769
  • Cover root-mounted servlets with the request-context, audit, and custom filter chain. #12760
  • Report dotted metadata names clearly. #12977
  • Validate the statistic name length. #12887
  • Return visible service admins as a list. #11836
  • Add @ResponseMetered to JobOperations#cancelJob. #13041

Authorization & Audit​

  • Invalidate function grants after drop. #12871
  • Authorize generic view operations. #12729
  • Allow catalog owners to grant schema privileges. #12269
  • Allow getting the owner of a disabled metalake. #12848
  • Evaluate list denies by access path. #12837
  • Align group load authorization. #12112
  • Add the missing supportsScheme to JdbcCredentialProvider. #12642
  • Reduce owner lookups in user and group lists. #13261
  • Handle a missing parent catalog during metadata id resolution. #12214
  • Stop a lost JCasbin role policy load from denying forever. #12230
  • Clear only role policies on cache eviction. #12169
  • Capture and redact request query parameters in audit log entries. #12872
  • Include role names in role assignment logs. #12885
  • Suppress internal operation audit events. #12892
  • Validate and authorize lineage events. #12840

Iceberg​

  • Discard builder changes when filtering snapshots by refs. #13452
  • Set metadata_location when loading a table with snapshots=refs. #13290
  • Map PostgreSQL JDBC auth failures to ConnectionFailedException. #13033
  • Forward the access-delegation header on federated loadTable. #12949
  • Share the managed memory catalog in auxiliary mode. #12851
  • Inject the GCS FileIO token from gcs-service-account-file. #9418
  • Support Azure service principal authentication for ADLS FileIO. #12958
  • Fix the JDBC CommunicationsException after idle timeout for the MySQL backend. #12589
  • Relocate Jackson in the Iceberg Aliyun bundle. #12357

Catalogs​

  • Hive: Support NOT NULL and DEFAULT column constraints on the Hive 3 metastore. #13350
  • Hive: Skip the HMS stats update for property/comment-only alterTable. #13020
  • Hive: Close the Log4j LoggerContext in HiveClientClassLoader#close() to prevent a Metaspace leak. #12246
  • Glue: Fail fast and give actionable errors on missing AWS credentials. #13012
  • Glue: Report rejected AWS credentials clearly. #13369
  • Glue: Include the AWS error detail in Glue failure messages. #12990
  • Glue: Do not expose Glue VIRTUAL_VIEW objects as tables. #13010
  • Glue: Derive the table location from the Glue database LocationUri. #12996
  • Glue: Ignore blank AWS static credentials. #12270
  • AWS: Support MinIO role ARNs in IRSA credential vending. #11883
  • ClickHouse: Escape single quotes in database and table name SQL to prevent injection. #11839
  • ClickHouse: Fix type conversion for Decimal, DateTime64, LowCardinality, and IPv4/IPv6. #11879
  • ClickHouse: Fix GraphiteMergeTree classification and add missing type mappings. #11910
  • ClickHouse: Distinguish MATERIALIZED/ALIAS from DEFAULT via default_kind. #11881
  • ClickHouse: Preserve the SETTINGS clause on table round-trip. #11880
  • ClickHouse: Preserve the MergeTree engine parameters through a create-load round-trip. #12273
  • ClickHouse: Preserve the composite primary key on table load. #12748
  • ClickHouse: Propagate a clustered table rename. #12761
  • ClickHouse: Read the sort key from system.tables. #11972
  • ClickHouse: Support user-defined GRANULARITY and validate the shard key type. #11802
  • ClickHouse: Reject varchar columns without length support.
  • Doris: Fix partition parsing for the Doris 3.0+ format. #11590
  • Doris: Fix INDEX syntax and support AUTO_INCREMENT for Doris 3.0+. #11590
  • Doris: Validate ADD INDEX fields. #12736
  • Doris: Support the replication allocation property. #11829
  • Doris: Handle missing-table errors from DROP TABLE. #13343
  • Doris: Preserve table comments when loading tables.
  • JDBC: Resolve PostgreSQL database names before accepting catalogs. #12938
  • JDBC: Quote comments in generated DDL. #12007
  • PostgreSQL: Report unconstrained NUMERIC as an external type. #13040
  • PostgreSQL: Quote the table name in ALTER statements for mixed-case tables. #13024
  • MySQL: Preserve the full type declaration for lossy MySQL column types. #13032
  • Catalogs: Return jdbc-user in plaintext for Iceberg and Paimon. #13151
  • Catalogs: Preserve upstream error messages when wrapping exceptions. #12998
  • Catalogs: Inspect thread targets without resolving subclass fields. #13075

Lance​

  • Authenticate the Lance REST service to the Gravitino server. #12255
  • Reject empty storage option keys. #13189
  • Prevent pagination index overflow. #13185
  • Reject nonempty Arrow input before table creation. #12988
  • Hydrate an empty schema before table alteration. #12407
  • Validate the identifier in ListNamespaces at schema level. #12468
  • Retry the repair-on-load metadata update on an optimistic-lock conflict. #11891
  • Add an external table guard to deregisterTable. #11893

Connectors​

  • Trino: Fail fast when Iceberg REST routing has no usable authentication. #13357
  • Trino: Enable Iceberg REST per-user sessions only under OAuth2. #13068
  • Trino: Reuse service metadata when the user token is missing. #13096
  • Trino: Build a complete function specification for SQL UDFs. #13139
  • Trino: Keep the engine types when applying projections. #12518
  • Flink: Use the catalog database instead of the schema for PostgreSQL JDBC table scans. #12929
  • Flink: Skip a no-op table alter to avoid "updates must not be empty". #12734
  • Flink: Exclude vulnerable log4j 2.x from the hive-common compile classpath. #11899

Maintenance Jobs​

  • Authenticate builtin-iceberg-update-stats against secured servers. #13137
  • Fail Iceberg jobs when the Spark runtime is missing. #13133

MCP Server​

  • Unify the statistics metadata full-name parameter. #13122
  • Return tag details when listing metadata tags. #13120
  • Preserve the authorization scheme for static credentials. #12447
  • Invoke the isort/black console scripts and pin their versions. #12660

Clients & Web UI​

  • Python client: Serialize the property field in RemoveCatalogPropertyRequest. #12324
  • OpenAPI: Restore the ExternalType variant to DataType. #11985
  • OpenAPI: Repair three codegen-breaking defects invisible to lint. #11985
  • OpenAPI: Drop a stray OpenAPI 3.1 null token from the statistics value. #11985
  • Common: Reject out-of-range numeric statistic values. #12598
  • Common: Validate statistic entries in PartitionStatisticsUpdateDTO. #12657
  • Web UI: Gate the metalake fetch until the OAuth token is ready. #12200
  • Web UI: Fix retaining the previous sub-path when switching filesets. #12289

Licensing & Packaging​

  • Correct legal documents in the Maven artifacts. #12755
  • Declare undeclared bundled components in LICENSE.bin and LICENSE.trino. #13451
  • Correct bundled component entries in the LICENSE and NOTICE files. #13253
  • Add the bundled Google auth and HTTP Client jars to the LICENSE files. #13100
  • Exclude the optional WildFly OpenSSL implementation. #12755
  • Upgrade the bundled PostgreSQL JDBC driver. #11795
  • Bind-mount packages/ in kerberos-hive so the archives stop shipping in the image. #12730
  • Prevent an empty classpath entry in common.sh. #12444
  • Polish log and error message quality across the Java modules. #10387

Acknowledgements​

Thanks to everyone who contributed to the 1.3.1 work — code, reviews, tests, issue triage, design, and feedback.

1fanwang, agnes-xinyi-lu, AzazelSensei, beyondhj, bharos, binhuiliu, danhuawang, diqiu50, FANNG1, Fayupable, forLp811, geyanggang, hqbhoho, hutiefang76, jarredhj0214, jerryshao, jiangxt2, JoegenUSTC, justinmclean, kevinw66, lasdf1234, LauraXia123, liangyouze, LuciferYang, markhoerth, mchades, MehulBatra, mrsibe, nevzheng, Octavi00, paultanay, qianleijava, roryqi, ss666, sujeito-operator, Valerde, vanphuoc3012, whua3, xxubai, yuqi1129, zhang-arvin

Apache, Apache Flink, Apache Hive, Apache Hudi, Apache Iceberg, Apache Ranger, Apache Spark, Apache Paimon and Apache Gravitino are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

Apache Gravitino 1.3.0 Release Notes

· 7 min read
Minghuang Li
Minghuang Li
PMC Member

We are glad to announce the release of Apache Gravitino 1.3.0! This release focuses on logical view management, hierarchical namespaces, AWS Glue catalog and engine connector support, built-in identity provider support, and Iceberg REST Catalog improvements.

It also includes important behavior changes for credential vending, Docker image layout, and Iceberg REST Catalog upgrades, plus broad improvements across authorization, connectors, clients, Web UI, observability, and deployment.

For the complete list of commits and pull requests included in this release, see the full changelog.

Highlights​

AWS Glue Catalog and Engine Connector Support​

Added AWS Glue Catalog support and Trino/Spark connector adapters, allowing Hive and Iceberg metadata in Glue to be governed and queried through Gravitino.

Unified View Management​

Added a unified view definition model and view support across Hive, Iceberg, and Apache Paimon, covering APIs, persistence, Java client support, and Web UI management.

Core Cache and Authorization Consistency​

Improved multi-node cache correctness with version-validated authorization caches, entity-change-log tracking, and global cache invalidation.

Enterprise-Grade Iceberg REST Catalog​

Added federated Iceberg REST Catalog support, nested multi-level namespaces, vended-credential refresh, freshness-aware table loading, and asynchronous cleanup.

Upgrade Notes / Behavior Changes​

Please review these changes before upgrading; they may require deployment configuration updates or may affect existing behavior.

  • Installation path moved to /opt/gravitino: The Gravitino Docker image home/install path changed from /root/gravitino to /opt/gravitino, and the published Docker images now match the current Dockerfile and Helm defaults. When upgrading a Helm deployment, update env.GRAVITINO_HOME and path references such as extraVolumeMounts, for example log mounts, from /root/gravitino to /opt/gravitino. (#11272, #11312, #11569, #11671)
  • Iceberg REST Catalog upgrades require a separate database backup: If the IRC service is backed by its own PostgreSQL database, back it up separately before upgrading. The Helm chart does not perform automatic schema migration; back up the database and apply the upgrade SQL scripts manually. (#11093, #11120)
  • Sensitive catalog properties are hidden from catalog load responses by default: Starting in 1.3.0, credentials such as jdbc-user, jdbc-password, cloud access keys, and similar sensitive catalog properties are excluded from GET /api/metalakes/{metalake}/catalogs/{catalog} responses. Clients and connectors should retrieve them through the credential vending API. For short-term compatibility during migration, gravitino.catalog.credential.backfillToProperties=true can restore the old behavior, but it exposes credentials in catalog properties and should be disabled after clients are upgraded. (#11264, #11554, #11669, #11692, #11741, #11745)
  • The web-v2 UI is now the default UI. (#11335)
  • Iceberg REST JDBC catalog now defaults to strict mode, so operations against non-existent namespaces return a 404 rather than silently succeeding. (#11285)
  • Iceberg table metadata cache is enabled by default, with an increased default capacity. (#11133)
  • The default Iceberg JDBC schema version is now v1. (#10851)
  • Iceberg REST config endpoint no longer includes the prefix, per the Iceberg REST spec. (#10640)
  • Hadoop upgraded from 2.10.2 to 3.3.6, and the legacy hadoop2 dependency line was removed. (#10788)

New Features​

  • Logical View Management: Gravitino now manages logical views as first-class, versioned entities across supported catalogs, with REST APIs, relational persistence, Java client support, Web UI management, and connector support for Hive, Iceberg, Paimon, Spark, and Flink.
  • Hierarchical Namespaces: Added multi-level nested namespaces in both the core REST server and the Iceberg REST Catalog, enabling deeper catalog structures and more flexible organization for complex business domains.
  • AWS Glue catalog and engine connectors: Added a new AWS Glue catalog with schema/table CRUD, native Iceberg table support through the Glue SDK and Iceberg SDK, integration tests, and Trino/Spark connector adapters.
  • Built-in identity provider and local authentication: Added a built-in IdP model, password hashing, user/group/relation storage, REST APIs, and Basic authentication for deployments that do not require an external IdP.
  • Authorization expansion: Added function authorization, group-aware ownership, group-inherited roles, scoped delegated privilege management with MANAGE_GRANTS, and stronger cache invalidation for authorization state.
  • Iceberg REST Catalog enhancements: Added asynchronous hard-deletion cleanup, vended-credential refresh for S3/GCS/OSS/ADLS, registerTable credential refresh, ETag-based freshness-aware table loading, federation handling improvements, and REST backend support.
  • Python Client Enhancements: Added authorization management, metadata-object statistics operations, and relational catalog support to the Python client.
  • Trino Connector Enhancements: Added CREATE TABLE AS SELECT, UDF adaptation, session-credential forwarding, Iceberg snapshot maintenance procedures, and multi-version integration test coverage.
  • Flink Connector Enhancements: Added view support for Iceberg and Paimon catalogs and support for Flink 1.19 and 1.20.
  • Operational Health and Audit Logs: Added health-check endpoints for Gravitino and IRC, plus a JSON formatter for audit logs.
  • New Hologres JDBC catalog: Added schema and table operations, a frontend, and integration tests for governing Alibaba Cloud Hologres.

Selected Improvements​

  • Core server: Cache consistency was strengthened with entity-change-log polling, CatalogManager cache invalidation, retention cleanup, fuller v2 audit operation mapping, and safer schema/table update validation.
  • Authorization: Performance and correctness improved through JCasbin cache work, per-request group-owner caching, batch owner lookup, and credential-vending support across Iceberg, Paimon, JDBC, and MySQL connectors.
  • Iceberg & Iceberg REST Catalog: Server-side Iceberg 1.11.0, backend HTTP timeout configs, federation refactoring, and client io-impl inference from table location.
  • Catalog & connectors: Glue complex types, Spark Glue support, Hive view support, Paimon view CRUD, Lance dependency upgrades, and reduced Hive Metastore catalog package size.
  • Web UI: The web-v2 UI became the default and added support for Glue, hierarchical schemas, and relational catalog views.
  • Table Maintenance Service: Refined the existing optimizer/table-maintenance workflow with documentation, build, dependency, module-structure, and test/CI cleanup.
  • Dependency & build: Hadoop 3.3.6, Log4j 2.25.4, PostgreSQL JDBC 42.7.11, H2 2.2.224, Hive runtime classpath cleanup, branch-1.3 cherry-pick automation, CI acceleration, JaCoCo PR reporting, Helm chart unit tests, and Helm OCI publishing.

Notable Bug Fixes​

  • Catalogs and connectors: Fixed Glue catalog edge cases, Trino catalog rollback/drop behavior, Trino OAuth2 and remote IRC table handling, Spark PostgreSQL timestamp handling, Flink/Paimon schema and partition handling, Hive metastore hostname/ClassLoader/Kerberos failures, ClickHouse issues, JDBC datetime filtering, Delta table restart metadata, timestamp time-zone consistency, and mixed-case table-name behavior.
  • Iceberg REST Catalog: Fixed hierarchical namespace drop handling, unsupported hierarchical-schema returns, authorization skip-check errors, connection pool shutdowns, remote IRC create view/table failures, staged create failures with authorization or credential vending, and AWS credential loading for Trino remote IRC.
  • Authorization and authentication: Fixed multi-admin IdP initialization, stale role bindings after privilege revocation, slow listCatalogs under authorization, table creation with authorization enabled, view authorization/cache behavior, owner-based SELECT checks, first-attempt new-user authorization failures, long-running query reauthorization, OAuth2 token refresh errors, occasional owner-setting failures, and Tag Manager metadata-object lookup errors.
  • Audit, observability, and health: Fixed audit timestamp precision, internal cross-server audit attribution, audit file/list/client-IP/formatter gaps, Hadoop metrics scheduler cleanup, and /health.html alias handling.
  • Web UI: Fixed unsupported-view error popups, relational view listing against unsupported catalogs, table/view navigation 404s, copy SQL behavior, view SQL display, bucketed Iceberg table editing, service-admin metalake creation button loading, and invalid metalake enable/disable switches.
  • Core and Storage: Fixed JDBC strict namespace handling, orphaned schema cleanup, table cache config, fileset credential NPEs, S3 fileset macOS bundle/runtime handling, GVFS long-running write failures, and WebUIFilter NPEs.
  • Deployment and Runtime: Fixed Docker image build inputs, published Docker image install path, Iceberg REST Docker environment mappings, Helm startup command, and JDK 8 client behavior.
  • Lance: Fixed declared/materialized table reporting, Web UI column display, runtime dependency size, and purgeTable handling.

Acknowledgements​

Thanks to everyone who contributed to the 1.3.0 work — code, reviews, tests, issue triage, design, and feedback.

A0R0P0I7T, a19920714liou, a638011, Abhijeetsng97, Abyss-lord, Aditi102005, ajw711, AlexGritA, AmitaWhite, anfebladi, arjnklc, babumahesh, bbiiaaoo, bharos, ChisomUma, chl-wxp, danhuawang, dennismdejong, diqiu50, FANNG1, flaming-archer, freesinger, gada121982, gauravrudragit, geniusjun, geyanggang, griffonbyte, hdygxsj, hobostay, Jalina2007, JandyTenedora, jarredhj0214, jerryshao, JoegenUSTC, kdyann, lasdf1234, laserninja, LauraXia123, lhjchn, Lucas61000, LuciferYang, LukasDEDD, markhoerth, mchades, mehakmeet, nikitanagar08, ningsh7, Octavi00, pandeysambhi, paultanay, pithecuse527, puchengy, pythaac, qqqttt123, raboof, rameshreddy-adutla, raushanprabhakar1, robertsilen, romanhorilyi, roryqi, Roshan1299, sachinnn99, sekikn, sgedward, shunki-fujita, sunyuhan1998, tanya0793, Tarantula471, Thakkar-Khushang, TimothyDing, tsungchih, Victory-ET, wangxiaojing, whua3, xxubai, YuF-9468, yunhwane, yuqi1129, yuw1, zhoukangcn

Apache, Apache Flink, Apache Hive, Apache Hudi, Apache Iceberg, Apache Ranger, Apache Spark, Apache Paimon and Apache Gravitino are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

Apache Gravitino 1.2.1

· 3 min read
Qi Yu
Qi Yu
PMC Member

We are glad to announce the release of Apache Gravitino 1.2.1! This is a patch release that focuses on stability, correctness, and performance improvements. It includes bug fixes across the core server, authorization, Iceberg REST, Trino connector, Hive, and various catalog implementations.

Improvements​

Core & Server​

  • Performance: Improve the performance of metadata object retrieval. #10459
  • Batch operations: Support batch get owner for bundle of metadata objects. #9518
  • Owner management: Clean zombie VIEW owner relations when deleting schema or catalog. #10458

Authorization​

  • Performance: Optimize JCasbin policy lookup for improved authorization performance when roles with large grants. #10907

Iceberg REST Catalog​

  • Spec compliance: Remove prefix from config endpoint per Iceberg REST spec. #10640
  • Schema version: Change default Iceberg JDBC schema version to v1. #10851
  • Error handling: Return JSON error body instead of HTML for pre-JAX-RS errors. #10667

Catalogs & Connectors​

  • Hive: Shrink the package size of Hive Metastore 2 and Hive Metastore 3 catalogs. #10457
  • Paimon: Exclude unnecessary tomcat-embed-core transitive dependency from Paimon catalog. #10524
  • Lance: Clean up redundant code in GravitinoLanceTableOperations#createTable. #10496

Helm & Charts​

  • Launch script: Use start-gravitino.sh instead of gravitino.sh to launch the service. #10557
  • Kubernetes: Add serviceAccountName to pod spec in deployment.yaml. #10573

Build​

  • Package size: Remove testing jars from release package. #10331
  • Module structure: Rename updater module to updaters for consistency. #10452
  • Build tooling: Remove release task and centralize JDK 8 compatibility handling. #10262
  • GitHub Actions: Update actions/upload-artifact from v4 to v7. #10646

Bug Fixes​

Core & Server​

  • Fix batchSelect queries missing version-info JOIN and field aliases. #10444
  • Fix missing partition path in getPartition error messages. #10175
  • Fix rollback masking the original post-hook exception. #10217
  • Fix missing @Param("metalakeId") in GroupRoleRelMapper.softDeleteGroupRoleRelByMetalakeId. #10657
  • Avoid blocking dropCatalog on imported schemas. #10737
  • Default ifExists to true when deleting a table index. #10380
  • Close leaked EntityStore lifecycle to fix test flakiness. #10700
  • Harden retention count assertions in TestFunctionMetaService and TestFilesetMetaService. #10700

Trino Connector​

  • Fix TrinoException when using catalog name with metalake. #10717

Iceberg REST Catalog​

  • Fix table scan planning failure due to incorrect SQL generation. #10841
  • Fix IcebergTableHookDispatcher failure with NoSuchTableException for staged creates. #10766

Hive​

  • Fix Hive Metastore OOM by closing HiveClientFactory on pool shutdown. #10844
  • Fix concurrent keytab symlink creation race condition in Hive Kerberos authentication. #10741

ClickHouse​

  • Fix ClickHouse alter-table bugs including autoIncrement issues. #10381

Web UI​

  • Preserve hidden properties when editing a catalog in web-v2. #10837

MCP Server​

  • Fix path encoding issues in MCP REST client for special characters. #10799

Common​

  • Lazy-initialize Version to fix TestVersion without jar task. #10762
  • Block H2 JDBC URL and driver in catalog datasource creation. #10801

CI & Infrastructure​

  • Pin localstack docker image version to 4.14.0 to fix CI problem. #10527
  • Pin docker/* GitHub Actions to SHA-based references (v4.0.0) to comply with ASF policy. #10504

Documentation​

  • Clarify verified JDBC compatibility matrix for ClickHouse. #10308

Acknowledgements​

Thanks to everyone who contributed to the 1.2.1 work — code, reviews, tests, issue triage, design, and feedback.

babumahesh, bharos, danhuawang, diqiu50, FANNG1, geyanggang, hdygxsj, jerryshao, laserninja, mchades, pandeysambhi, pythaac, roryqi, sachinnn99, yuw1, yuqi1129

Apache, Apache Flink, Apache Hive, Apache Hudi, Apache Iceberg, Apache Ranger, Apache Spark, Apache Paimon and Apache Gravitino are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

Apache Gravitino 1.1.1

· 4 min read
Qi Yu
Qi Yu
PMC Member

We are glad to announce the release of Apache Gravitino 1.1.1! This is a patch release that focuses on stability, correctness, and performance improvements. It includes bug fixes across the core server, authorization, Iceberg REST, Spark connector, OAuth, and various catalog implementations.

Improvements​

Core & Server​

  • Cache: Cache non-existent relational data to avoid repeated backend lookups on missing entities. #9799
  • Performance: Reduce the number of catalogInUse calls on the server hot path. #9474
  • Metadata API: Support batchGet for metadata objects to reduce round-trips. #9893
  • Managed entities: Include lakehouse-generic catalogs in managed entities for proper drop behavior. #9490
  • Error handling: Preserve the post-hook exception when rollback fails, so the original error is not swallowed. #10217
  • Partition API: Include partition path parameter in getPartition error messages for easier debugging. #10175

Authorization​

  • Performance: Avoid authorization-plugin overhead when the plugin is not configured. #9170
  • Performance: Convert Jcasbin internal map to a cache to speed up permission lookups. #9770
  • Batch authorization: Support preloading table metadata in batch metadata authorization. #9802

Iceberg REST Server​

  • Performance: Improve the performance of loading tables by reducing redundant HMS calls. #9765
  • Cache: Optimize the catalog wrapper and entity cache expiry strategy. #9782
  • Rename: Support renaming a table across different namespaces in the Gravitino Iceberg catalog. #9571

Lance REST Server​

  • Column operations: Support drop and rename column for Lance tables. #9113
  • Empty table: Refine the concept of createEmptyTable in Lance REST for clearer semantics. #9520
  • Statistics: Add maxStatisticsPerUpdate configuration for Lance partition storage. #9650
  • Helm: Add a complete Lance REST server Helm chart. #9403

Common​

  • Version parsing: Enhance version parsing to support release candidate tags with validation. #9482
  • Code quality: Refactor to reduce duplicated code across modules. #9294
  • Logging: Update log4j2 configuration for Iceberg/Lance REST servers. #9547
  • Build: Add MCP-server changes handling in build workflow. #9921
  • Build: Remove release task and centralize JDK 8 compatibility handling. #10262

Bug Fixes​

Core & Server​

  • Fix loading table failure caused by incorrect SQL in the fetch-column-info query. #10034
  • Fix tag association problem that caused tags to be incorrectly linked. #9635
  • Fix credential issue for filesets with multiple locations. #9500
  • Fix equals and hashCode missing from Policy.java, causing incorrect policy comparison. #10009
  • Default ifExists to true when deleting a table index to prevent spurious errors. #10380

Authorization​

  • Fix NoSuchEntityException caused by schema entity not being imported before authorization checks. #10055
  • Fix schema import to avoid setOwner failures when the schema had not been ingested. #9809
  • Fix PassThroughAuthorizer user verification logic that incorrectly rejected valid users. #9616

Iceberg​

  • Fix wrong namespaces returned when listing tables or views in multi-level namespace configurations. #10397
  • Fix URL decoding of table names in Iceberg REST server request paths. #9936
  • Fix authorization decode issue for table names containing special characters. #9936
  • Fix migrate procedure by preserving the stageCreate flag. #9666

OAuth​

  • Allow JWKS validators to operate without serverUri or tokenPath being mandatory. #9713

Catalogs​

  • Fix altering the JDBC catalog column default value problem. #9816
  • Fix UnsupportedOperationException when updating aliases for a model version created without aliases. #9727

Hive​

  • Perform proper resource cleanup in HiveClientPool.close() to prevent connection leaks. #9581

Lance REST​

  • Handle null mode in registerTableRequest to prevent NPE. #9512

Spark Connector​

  • Fix No SLF4J providers warning/error in Spark connector 3.3. #6906

CI & Infrastructure​

  • Pin all docker/* GitHub Actions to SHA-based references (v4.0.0) to comply with ASF policy. #10502
  • Fix Python CI pipeline failures due to runner image upgrade. #9919
  • Fix Docker container startup failures due to GitHub CI runner image upgrade. #9990
  • Fix MCP-server fastmcp version to avoid breaking CI changes from 3.0.x. #10035
  • Fix UV CI pipeline. HOTFIX
  • Fix JDK8 compatibility issues across modules. #10373

Documentation​

  • Add documents about the Flink catalog name limitation. #9973
  • Update OAuth documentation to clarify correct version endpoints for Azure authentication. #9868
  • Fix the incorrect curl command in the migration guide for set the owner. #10041
  • Add documentation for docker run command in the Hive section. #9876
  • Improve lakehouse-paimon-catalog documentation. #9957
  • Add a guide for Lance REST integration with Spark and Ray. #9622
  • Add REST catalog backend documentation for Iceberg REST Catalog (IRC). MINOR
  • Add warehouse documentation for the REST catalog backend in Iceberg. MINOR
  • Add missing DescribeTable endpoint to lance-rest-service.md. #9662

Acknowledgements​

Thanks to everyone who contributed to the 1.1.1 work — code, reviews, tests, issue triage, design, and feedback.

@FANNG1, @agnes-xinyi-lu, @bharos, @chl-wxp, @danhuawang, @echonesis, @jerryshao, @joeyutong, @mchades, @pandeysambhi, @pythaac, @qqqttt123, @roryqi, @tedyu, @yuqi1129

Apache Gravitino 1.2.0

· 8 min read
Hui Yu
Hui Yu
committer

Apache Gravitino 1.2.0 has been released! This release introduces major new capabilities including a new Table Maintenance Service (TMS), a new ClickHouse catalog, end-to-end UDF management, authorization for Iceberg view operation, a redesigned Web UI, and broad improvements across connectors, authorization, and clients.

Release Date: 2026-03-13 Previous Version: 1.1.0 (2025-12-16)

Highlights​

  • Table Maintenance Service (TMS) foundational framework — Data platforms can now shift from reactive firefighting to proactive table health. Gravitino analyzes your tables and automatically schedules the right maintenance operations, at the right times.
  • ClickHouse catalog — Teams running real-time analytics on ClickHouse can now govern it alongside their lakehouse, with one metadata layer for both streaming and batch workloads.
  • Scan planning offload — Query engines like DuckDB and Spark can now offload scan planning to Gravitino's IRC server, reducing query latency and client-side complexity — making Gravitino a more capable catalog server for the growing Iceberg ecosystem.
  • Ecosystem reach — Multi-version Trino connector (435–478), Flink user authentication, and multi-cluster fileset support and Web UI v2 broaden Gravitino's integration surface across the modern data stack.

New Features​

  1. Table Maintenance Service (TMS) #9546, #9652, #9653, #9654, #9983, #9984, #9985, #9986, #10096, #10097, #10098, #10140, #9541, #9543

    Maintaining a healthy lakehouse table requires ongoing maintenance work — compaction, rewriting data files, and expiring snapshot cleanup. Gravitino 1.2.0 introduces the Table Maintenance Service (TMS), which analyzes table statistics and automatically schedules the right maintenance operations at the right time. This allows data platforms to move from reactive firefighting to proactive table health management.

  2. ClickHouse Catalog #9738, #9754, #9755, #9756, #9820, #9865

    ClickHouse is widely adopted for large-scale real-time analytics. Gravitino 1.2.0 introduces a full-featured ClickHouse catalog with full DDL support including distributed and partitioned cluster modes.

  3. User-Defined Function (UDF) Management #9525, #9527, #9528, #9529, #9530, #9531, #9532, #9561

    Teams can now centrally register, update, and govern UDFs, and Spark can automatically discover and invoke these functions through Gravitino's catalog interface. Gravitino 1.2.0 introduces end-to-end UDF management: Java API, server-side REST interface, relational storage backend, Java client support, Spark FunctionCatalog integration, Python client, Web UI management for visual browsing/creation, and complete documentation with OpenAPI specifications.

  4. Iceberg REST Catalog: Support scan planning cache #9048

    Adding a cache for scan planning improves performance by caching the results of the scan planning phase, which determines which data files need to be read for a query. Since scan planning involves reading and parsing Iceberg metadata and manifest files, caching avoids repeating this expensive work for similar queries. As a result, it reduces metadata I/O, lowers query latency, and decreases load on object storage and the planning service.

  5. Authorization support for Iceberg REST catalog view operations #9744, #9745, #9746, #9747, #9915

    Views are an important abstraction for data access control but have long been a governance blind spot. Gravitino 1.2.0 manages Iceberg views as first-class entities, introducing view management, generic storage, view-level permissions, and full authorization support for IRC view operations. This enables teams to govern views alongside tables and apply fine-grained access control policies to view access while maintaining consistent enforcement of underlying table permissions.

  6. Generic Lakehouse Catalog: Delta Lake External Table Support #9647

    Delta Lake is one of the most widely deployed open table formats. Gravitino now supports registering and managing external Delta tables through the generic lakehouse catalog, enabling unified governance for Delta alongside Iceberg, Hudi, and other formats without data migration. This allows organizations to manage Delta Lake workloads through an open catalog architecture, reducing dependence on proprietary catalog services such as Unity Catalog.

  7. Trino Connector Multi-version Support #9718, #9719, #9894, #9952, #9961, #9964, #10091

    Gravitino now supports Trino versions 435 through 478, with a connector architecture designed to maintain compatibility with future Trino releases.

  8. Multi-cluster Fileset Support #9568, #9312
    Production data platforms often span multiple storage clusters. Through GVFS multi-cluster filesets, Gravitino now allows a single fileset to reference storage locations across different clusters.

  9. Flink Connector: User Authentication #9564
    The Gravitino Flink connector now supports user authentication, enabling secure access to Gravitino-managed metadata from Flink jobs.

  10. Web UI v2 Reconstruction #9758 The Gravitino Web UI now supports managing ClickHouse catalogs and UDFs, along with improved views for tags, policies, and task templates. The updated interface provides a more streamlined experience for navigating governance resources. Web v1 remains available during the transition period.

Improvements​

Core & Server​

  • Block non-cascading schema deletions when topics still exist (#9078)
  • Cache non-existent relational data to avoid repeated lookups (#9799)
  • Optimize in-use status checks for catalogs and metalakes (#9586)
  • Improve authorization performance by skipping overhead when plugin is empty (#9170)
  • Support renaming tables to a different schema in ManagedTableOperations (#9477)
  • Include generic lakehouse catalogs in managed entities for proper drop behavior (#9490)
  • Add JDBC storage backend for partition statistics (#9838)
  • Make gravitino-api and related dependencies compileOnly in catalog modules (#10195)
  • Optimize JDBC driver deregistration logic to avoid possible OOM (#10253)

Authorization​

  • Convert JCasbin internal map to a cache for improved performance (#9770)
  • Support preloading table metadata in batch for authorization checks (#9802)
  • Rename model privilege names to follow operation name conventions (#9381)
  • Support overriding privileges for roles (#9269)
  • Fileset supports credential vending with correct access privileges (#9506)

Iceberg REST Catalog (IRC)​

  • Upgrade Apache Iceberg to 1.10.1 (#9989)
  • Optimize IRC catalog wrapper and entity cache expiry strategy (#9782)
  • Improve IRC table load performance under high concurrency with authorization enabled (#9765)
  • IRC uses internal catalog fetcher instead of HTTP interface for better performance (#9825)
  • IRC timely expires catalog wrapper cache (#9966)
  • Improve table load performance by tuning the cooperation between Iceberg service and Gravitino server (#9277)
  • Cross-Namespace Table Rename: Gravitino Iceberg catalog now supports renaming tables across different namespaces (#9517)

Catalogs & Connectors​

  • Support Paimon REST backend (#9791)
  • Support hash distribution in Paimon catalog (#9731)
  • Refactor Hive and Hudi catalogs to use the shared HiveClient (#9459)
  • Add support to skip catalog in Trino connector (#9492)
  • Flink connector supports generic Hive tables (#9504)
  • Spark connector supports TableWritePrivilege for Spark 3.5+ authorization (#10181)
  • Upgrade Kyuubi Hive connector from 1.10 to 1.11 (#10040)

OAuth & Authentication​

  • Support JWT tokens with multiple audiences in the aud claim (#9733)
  • Support regex for user principal mapping in OAuth (#9767)
  • Allow JWKS validators to work without serverUri or tokenPath (#9713)
  • Enhance version parsing to support release candidate version strings (#9482)

Clients​

  • Add configurable HTTP connection pool settings for Gravitino client (#9468)
  • Allow disabling client-server version check via environment variable (#9760)

Lance REST Service​

  • Add Helm chart for deploying Lance REST server as a standalone service on Kubernetes (#9403)
  • Add dataset version tracking in Lance REST loadTable and createTable (#9792)
  • Add documentation for Lance REST server setup and Spark/Ray integration (#9169, #9622)
  • Refine createEmptyTable semantics in Lance REST (#9520)

Web UI​

  • Add global banner to guide users to Web UI v2 (#9996)
  • Support ClickHouse catalog management in Web UI v2 (#9865)
  • Support associated roles view for tags, policies, and job templates (#9807)

Bug Fixes​

  • Fix credential vending for filesets with multiple storage locations (#9500)
  • Fix IRC connection failure after idle timeout (#9383)
  • Fix tag association cache inconsistency (#9635)
  • Fix Iceberg migrate procedure failing with "table already exists" for JDBC catalogs (#9666)
  • Fix clearing column comments in MySQL table ALTER operations (#9694)
  • Fix UnsupportedOperationException when updating model version aliases (#9727)
  • Fix table loading failure due to incorrect SQL in fetch column info (#10034)
  • Fix NoSuchEntityException in authorization when schema entity is not imported (#10055)
  • Fix owner assignment failure due to unimported schema entity (#9809)
  • Fix Hive SerDe incompatibility between Gravitino Flink connector and native Flink client (#9508)
  • Fix JDBC catalog column default value alteration (#9816)
  • Fix Trino connector distribution to include JARs for all supported versions (#10139)
  • Fix TMS built-in rewrite adapter resolution by template name (#10311)
  • Fix JDBC catalog pool size properties not being loaded on catalog creation (#10284)
  • Fix SLF4J provider not found error in Spark connector 3.3 (#6906)
  • Fix PassThroughAuthorizer user verification logic blocking new user creation (#9616)
  • Fix IRC URL decoding for table names containing special characters (#9936)
  • Fix lazy authorization configuration check for IRC to allow server startup (#9247)
  • Fix maxStatisticsPerUpdate configuration for Lance partition storage (#9650)

Breaking Changes​

  • Dropped Python 3.9 Support — Python 3.9 has reached end of life. The Gravitino Python client no longer supports this version. Users need to upgrade to Python 3.10 or higher. (#10011)

Acknowledgements​

Thanks to everyone who contributed to the 1.2.0 work — code, reviews, tests, issue triage, design, and feedback. Below is a consolidated list of contributor GitHub IDs extracted from issue and PR activity.

Apache, Apache Flink, Apache Hive, Apache Hudi, Apache Iceberg, Apache Ranger, Apache Spark, Apache Paimon and Apache Gravitino are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

Apache Gravitino 1.1.0 - An AI-native metadata management platform

· 6 min read
Qi Yu
Qi Yu
PMC Member

We are glad to announce the release of Apache Gravitino 1.1.0! This release builds upon the solid foundation laid by Apache Gravitino 1.0.0, introducing a range of new features, improvements, and bug fixes that enhance the platform's capabilities, performance, and security.

Highlights​

  • Broader catalog support (initial Lance REST service, a reusable lakehouse-generic catalog, and Hive3) to simplify integration with diverse lakehouse deployments.
  • Stronger metadata-level authorization and security hardening for the Iceberg REST surface.
  • Multi-cluster fileset support and Python client improvements for real-world multi-region and migration workflows.
  • Stability, performance and observability work across the entity-store, caches, scan planning, connectors and CI — reducing operational friction and test flakiness.

New Features​

  1. Built for the Future of AI Data: Lance REST service. #8889

As AI and ML workflows become central to data platforms, efficient access to vector data is crucial. The new Lance REST service exposes Lance datasets through a managed HTTP interface. This allows remote clients—such as inference services or notebooks—to access vector data with the high performance of the Lance format, all while adhering to Apache Gravitino's centralized security and governance policies.

  1. Generic lakehouse catalog. #8828

The lakehouse ecosystem is diverse and rapidly evolving, with new table formats and engines emerging frequently. To keep pace, we introduced a generic lakehouse catalog framework. This abstraction reduces the boilerplate code required to integrate new engines, standardizing how capabilities are negotiated and how namespaces are handled. This means faster support for new formats and a more consistent experience for developers and users alike.

  1. Access control for Iceberg REST service. #4290

The Iceberg REST catalog is becoming the standard for open table access, but production use demands robust security. We have hardened the Iceberg REST service with comprehensive authentication and authorization checks. This ensures that data accessed via standard Iceberg clients is fully protected, making Apache Gravitino a secure choice for multi-tenant and public-facing data lake deployments.

  1. Hive 3 catalog support. #5912

Many enterprises still rely on Hive 3 for their core data infrastructure, making migration a risky and complex endeavor. This feature allows users to register existing Hive 3 metastores directly as Apache Gravitino catalogs. By doing so, organizations can instantly bring their legacy data under Apache Gravitino's unified governance and management umbrella without moving data or disrupting existing workloads, paving the way for a smoother transition to modern lakehouse architectures.

  1. Multiple HDFS clusters support. #9117, #9288

In large-scale production environments, data is often distributed across multiple HDFS clusters to ensure isolation and disaster recovery. Previously, Apache Gravitino was limited in how it handled these complex topologies. With this release, users can manage filesets across multiple HDFS clusters within a single Apache Gravitino instance. This capability simplifies cross-cluster data management, improves resource isolation, and provides greater flexibility for multi-tenant architectures.

  1. Metadata authorization for IRC, statistics, tags, jobs, and policies. #4361, #8752, #8944, #8943

True governance requires securing every aspect of the metadata platform. We have expanded fine-grained authorization to cover auxiliary resources like tags, statistics, and background jobs. This enhancement closes previous security gaps, ensuring that all user interactions with the system—whether viewing statistics or managing tags—are strictly governed by least-privilege policies.

  1. New Iceberg REST endpoints. #6336

To support the full range of capabilities expected by modern analytics tools, we have implemented additional endpoints from the Iceberg REST specification. This improves compatibility with the latest query engines and clients, ensuring that users can leverage advanced planning and catalog operations without running into compatibility issues.

Improvements​

Core & Server​

  • Entity store and Cache: Fixed several performance and logic issues to improve stability and speed. #8697, #8743, #8815, #8817, #8710, #9148, #7916, #8546
  • Metrics: Expose more metrics for server and catalogs to enhance observability. #8594
  • Authorization: Refined permission checks. #7942.
  • Resource management: Improved resource release and closure mechanisms to prevent leaks. #8981, #9002, #8999
  • JDBC metric store: Support storing Iceberg metrics in JDBC. #8899
  • Job system enhancement: Support job alteration. #8638, #8814

Catalogs & Connectors​

  • Iceberg catalog: Support metadata cache. #8314
  • Upgrade Iceberg to 1.10.0 to support scan planning. #9046
  • Improve dynamic config provider for better usability. #8970
  • Fileset catalog: Prevented filesystem instances from hanging for a long time. #9280
  • Trino connector: Support SQL UPDATE/DELETE/MERGE. #8241
  • Fix getTableStatistics in GravitinoMetadata. #9100

Clients​

  • GVFS client: Improved stability and error handling. #8752, #8882, #8948, #8953.
  • Fileset bundle JARs: Refactored for a more detailed delivery strategy. #9106
  • Python client: Added support for relational catalog. #5198

Developer Experience & Operations​

  • Helm chart: Enhanced configuration options and stability. #8747, #8174
  • GitHub templates: Added templates to support AI coding. #9227.
  • Tests: Refactoring and enhancement of test suites. #9223, #9107
  • Docker: Changed Apache Gravitino Docker base image. #8817
  • Code Style: Upgrade Google Java Format to support JDK 17. #8792.

Frontend Updates​

  • Added pagination for files list. #8987
  • Displayed the index type in UI. #6997
  • Upgraded dependabot affected versions. #9357
  • Fixed routing issue where path '/' may not route to 'metalakes'. #9354

Bug Fixes​

  • Create topic encounters NoSuchTopicException when Kafka is deployed with 3 brokers on EKS. #4168
  • Apache Gravitino IRC server returns java.lang.NoSuchMethodError: void org.apache.hadoop.security.HadoopKerberosName.setRuleMechanism. #8754
  • Several bugs in SQL provider. #8659, #9166
  • Unknown error when using fsspec through JNI. #8858

Still, there are many bug fixes that have not been listed due to limited space. Please refer to the full list of issues and pull requests merged since the 1.0.0 release for more details.

Acknowledgements​

Thanks to everyone who contributed to the 1.1.0 work — code, reviews, tests, issue triage, design, and feedback. Below is a consolidated list of contributor GitHub IDs extracted from issue and PR activity.

Apache, Apache Flink, Apache Hive, Apache Hudi, Apache Iceberg, Apache Ranger, Apache Spark, Apache Paimon and Apache Gravitino are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

Apache Gravitino 1.0.1 - Release Notes

· 3 min read
Minghuang Li
Minghuang Li
PMC Member

We are pleased to announce the release of Gravitino 1.0.1. This version introduces comprehensive support for job template alterations, along with significant improvements and bug fixes across the core engine, various catalogs, and clients.

Major Features & Improvements​

Job and Job Template​

Gravitino Core​

  • Refactored tag operations by leveraging the entity store's relation operations. #7916
  • Made several optimizations to the Caffeine cache, including adjusting weight values, resolving a performance issue with reverseIndex, and prioritizing the eviction of tags and policies when the cache is full, and so on. #8697, #8743, #8815, #8871, #8937

Catalogs​

  • Kafka: Fixed an issue where topic creation was asynchronous, ensuring the operation is now synchronous. #4168
  • Iceberg: Fixed a failure in starting the Iceberg REST server within a Docker environment. #8733
  • Doris, StarRocks, PostgreSQL: Fixed incorrect parsing of column default values and types for these data sources. #8277

Python Client​

  • Added metadata objects to the Python client. #8627
  • Fixed an incorrect credential URL and a fileset test issue on GCS. #8935, #8969

Authorization​

  • Authorization is supported for the testCatalogConnection operation. #7893

Web UI​

  • Fixed an issue with reconfiguring submission parameters when creating a catalog. #8694
  • Added pagination support for the fileset file list. #8987

Bug Fixes​

  • Fixed a Null Pointer Exception (NPE) in TableFormat.java when a user has no roles. #8202
  • Corrected exception handling in the setPolicy operation. #8661
  • Fixed missing policy operations in the OpenAPI entry point. #8706
  • Fixed a build failure in the gvfs-fuse module. #8830
  • Fixed an issue where the hard deletion of statistics would fail. #9038
  • Corrected index names for statistics and job names in the database upgrade script. #8979
  • Fixed deletePolicyAndVersionMetasByLegacyTimeline error. #9031
  • Fixed role didn't update when the table is deleted. #8824

Credits​

We would like to thank the following contributors for their valuable contributions to this release:

@dyrnq @yuqi1129 @LauraXia123 @jerryshao @danhuawang @playasim @keepConcentration @KayMas2808 @jerqi @mchades @HugoSalaDev @FANNG1 @diqiu50 @hdygxsj @tsungchih

Apache, Apache Flink, Apache Hive, Apache Hudi, Apache Iceberg, Apache Ranger, Apache Spark, Apache Paimon and Apache Gravitino are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

Apache Gravitino 1.0.0 - From Metadata Management to Contextual Engineering

· 8 min read
Jerry Shao
Jerry Shao
PMC Member

Apache Gravitino was designed from day one to provide a unified framework for metadata management across heterogeneous sources, regions, and clouds—what we define as the metadata lake (or metalake). Throughout its evolution, Gravitino has extended support to multiple data modalities, including tabular metadata from Apache Hive, Apache Iceberg, MySQL, and PostgreSQL; unstructured assets from HDFS and S3; streaming and messaging metadata from Apache Kafka; and metadata for machine learning models. To further strengthen governance in Gravitino, we have also integrated advanced capabilities, including tagging, audit logging, and end-to-end lineage capture.

After all enterprise metadata has been centralized through Gravitino, it forms a data brain: a structured, queryable, and semantically enriched representation of data assets. This enables not only consistent metadata access but also knowledge grounding, contextual reasoning, tool using and others. As we approach the 1.0 milestone, our focus shifts from pure metadata storage to metadata-driven contextual engineering—a foundation we call the Metadata-driven Action System, to provide the building blocks for the contextual engineering.

The release of Apache Gravitino 1.0.0 marks a significant engineering step forward, with robust APIs, extensible connectors, enhanced governance primitives, improved scalability and reliability in distributed environments. In the following sections, I will dive into the new features and architectural improvements introduced in Gravitino 1.0.0.

Metadata-driven action system​

In version 1.0.0, we introduced three new components that enable us to build jobs to accomplish metadata-driven actions, such as table compaction, TTL data management, and PII identification. These three new components are: the statistics system, the policy system, and the job system.

Taking table compaction as an example:

  • Firstly, users can define the table compaction policy in Gravitino and associate this policy with the tables that need to be compacted.
  • Then, users can save the statistics of the table to Gravitino.
  • Also, users can define a job template for the compaction.
  • Lastly, users can use the statistics with the defined policy to generate the compaction parameters and use these parameters to trigger a compaction job based on the defined job templates.

Statistics system​

The statistics system is a new component for the statistics store and retrieval. You can define and store the table/partition level statistics in Gravitino, and also fetch them through Gravitino for different purposes.

For the details of how we design this component, please see #7268. For instructions on using the statistics system, refer to the documentation here.

Policy system​

The policy system enables you to define action rules in Gravitino, like compaction rules or TTL rules. The defined policy can be associated with the metadata, which means these rules will be enforced on the dedicated metadata. Users can leverage these enforced polices to decide how to trigger an action on the dedicated metadata.

Please refer to the policy system documentation to know how to use it. For more information on the policy system's implementation details, please refer to #7139.

Job system​

The job system is another feature that allows you to submit and run jobs through Gravitino. Users can register a job template, then trigger a job based on the specific job template. Gravitino will help submit the job to the dedicated job executor, such as Apache Airflow. Gravitino can manage the job lifecycle and save the job status in it. With the job system, users can run a self-defined job to accomplish a metadata-driven action system.

In version 1.0.0, we have an initial version to support running the jobs as a local process. If you want to know more about the design details, you can follow issue #7154. Also, a user-facing documentation can be found here.

The whole metadata-driven action system is still in an alpha phase for version 1.0.0. The community will continue to evolve the code and take the Iceberg table maintenance as a reference implementation in the next version. Please stay tuned.

Agent-ready through the MCP server​

MCP is a powerful protocol to bridge the gap between human languages and machine interfaces. With MCP, users can communicate with the LLM using natural language, and the LLM can understand the context and invoke the appropriate tools.

In version 1.0.0, the community officially delivered the MCP server for Gravitino. Users can launch it as a remote or local MCP server and connect to various MCP applications, such as Cursor and Claude Desktop. Additionally, we exposed all metadata-related interfaces as tools that MCP clients can call.

With the Gravitino MCP server, users can manage and govern metadata, as well as perform metadata-driven actions using natural language. Please follow issue #7483 for more details. Additionally, you can refer to the documentation for instructions on how to start the MCP server locally or in Docker.

Unified access control framework​

Gravitino introduced the RBAC system in the previous version, but it only offers users the ability to grant privileges to roles and users, without enforcing access control when manipulating the secure objects. In 1.0.0, we complete this missing piece in Gravitino.

Currently, users can set access control policies through our RBAC system and enforce these controls when accessing secure objects. For details, you can refer to the umbrella issue #6762.

Add support for multiple locations model management​

The model management is introduced in Gravitino 0.9.0. Users have since requested support for multiple storage locations within a single model version, allowing them to select a model version with a preferred location.

In 1.0.0, the community added multiple locations for model management. This feature is similar to the fileset’s support for multiple locations. Users can check the document here for more information. For more information on implementation details, please refer to this issue #7363.

Support the latest Apache Iceberg and Paimon versions​

In Gravitino 1.0.0, we have upgraded the supported Iceberg version to 1.9.0. With the new version, we will add more feature support in the next release. Additionally, we have upgraded the supported Paimon version to 1.2.0, introducing new features for Paimon support.

You can see the issue #6719 for Iceberg upgrading and issue #8163 for Paimon upgrading.

Various core features​

Core:

  • Add the cache system in the Gravitino entity store #7175.
  • Add Marquez integration as a lineage sink in Gravitino #7396.

Server:

  • Add Azure AD login support for OAuth authentication #7538.

Catalogs:

  • Support StarRocks catalog management in Gravitino #3302.

Clients:

Spark connector:

  • Upgrade the supported Kyubbi version #7480.

UI:

  • Add web UI for listing files / directories under a fileset #7477.

Deployment:

  • Add hem char deployment for Iceberg REST catalog #7159.

Behavior changes​

Compatible changes:​

  • Rename the Hadoop catalog to fileset catalog #7184.
  • Allowing event listener changes Iceberg create table request #6486.
  • Support returning aliases when listing model version #7307.

Breaking changes:​

  • Change the supported Java version to JDK 17 for the Gravitino server.
  • Remove the Python 3.8 support for the Gravitino Python client #7491.
  • Fix the unnecessary double encoding and decoding issue for fileset get location and list files interfaces #8335. This change is incompatible with the old version of Java and Python clients. Using old version clients with a new version server will meet a decoding issue in some unexpected scenarios.

Overall​

There are still lots of features, improvements, and bug fixes that are not mentioned here. We thank the community for their continued support and valuable contributions.

Apache Gravitino 1.0.0 opens a new chapter from the data catalog to the smart catalog. We will continue to innovate and build, to add more Data and AI features. Please stay tuned!

Credits​

This release acknowledges the hard work and dedication of all contributors who have helped make this release possible.

1161623489@qq.com, Aamir, Aaryan Kumar Sinha, Ajax, Akshat Tiwari, Akshat kumar gupta, Aman Chandra Kumar, AndreVale69, Ashwil-Colaco, BIN, Ben Coke, Bharath Krishna, Brijesh Thummar, Bryan Maloyer, Cyber Star, Danhua Wang, Daniel, Daniele Carpentiero, Dentalkart399, Drinkaiii, Edie, Eric Chang, FANNG, Gagan B Mishra, George T. C. Lai, Guilherme Santos, Hatim Kagalwala, Jackeyzhe, Jarvis, JeonDaehong, Jerry Shao, Jimmy Lee, Joonha, Joonseo Lee, Joseph C., Justin Mclean, KWON TAE HEON, Kang, KeeProMise, Khawaja Abdullah Ansar, Kwon Taeheon, Kyle Lin, KyleLin0927, Lord of Abyss, MaAng, Mathieu Baurin, Maxspace1024, Mikshakecere, Mini Yu, Minji Kim, Minji Ryu, Nithish Kumar S, Pacman, Peidian li, Praveen, Qian Xia, Qiang-Liu, Qiming Teng, Raj Gupta, Ratnesh Rastogi, Raveendra Pujari, Reuben George, RickyMa, Rory, Sambhavi Pandey, Sébastien Brochet, Shaofeng Shi, Spiritedswordsman, Sua Bae, Surya B, Tarun, Tian Lu, Tianhang, Timur, Viral Kachhadiya, Will Guo, XiaoZ, Xiaojian Sun, Xun, Yftach Zur, Yuhui, Yujiang Zhong, Yunchi Pang, Zhengke Zhou, _.mung, ankamde, arjun, danielyyang, dependabot[bot], fad, fanng, gavin.wang, guow34, jackeyzhe, kaghatim, keepConcentration, kerenpas, kitoha, lipeidian, liuxian, liuxian131, lsyulong, mchades, mingdaoy, predator4ann, qbhan, raveendra11, roryqi, senlizishi, slimtom95, taylor.fan, taylor12805, teo, tian bao, vishnu, yangyang zhong, youngseojeon, yuhui, yunchi, yuqi, zacsun, zhanghan, zhanghan18, 梁自强, 박용현, 배수아, 신동재, 이승주, 이준하

Apache, Apache Flink, Apache Hive, Apache Hudi, Apache Iceberg, Apache Ranger, Apache Spark, Apache Paimon and Apache Gravitino are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.

Apache Gravitino 0.9.1

· 2 min read
Rory Qi
Rory Qi
PMC Member

Model Management​

Support updating aliases for model versions #6814,#7158

Add file viewer support for Filesets #6860

Implement ListFilesEvent in FilesetEventDispatcher #7314

Support setOwner/getOwner event operations #7646

Trino Connector​

Auto-load multiple metalakes in Trino connector #7288

JDBC Validation​

Validate JDBC URLs during store initialization #7547

Bug Fixes

Core & Catalogs​

Fix H2 backend file lock issues during deletion #7406

Prevent SQL session commit errors #7403

Correct OAuth token refresh in web UI #7426

Validate namespace string conversions #7516

Improve server force-kill shutdown logic #7513

Fix bypass key handling in Hive catalog #7416

Filter empty Hadoop storage locations #7190

Fix model catalog error messages #7346

Connectors​

Spark Connector​

Remove conflicting slf4j dependency #7287

Fix S3 credential test errors #7432

Trino Connector​

Handle unsupported catalog providers #7322

Python Client​

Fix storage handler mappings for S3/OSS/ABS #7225

Improve Java client error messages #7344

Filesets​

Fix multi-location file paths #7371

Improvements

Core & Catalogs​

Optimize column deletion logic (#7415)(https://github.com/apache/gravitino/issues/7415)

Auto-register mappers via SPI #7529

Validate JDBC entity store URLs #7614

Fix catalog index existence checks #7660

CLI & Clients​

Remove duplicate owner field in CLI #7639

URL-encode paths in Java client #7686

Testing​

Refactor Hadoop catalog test stubbing #7280

Fix precondition message mismatches #7521

Documentation​

Add Trino REST catalog example #7121

Iceberg IRC guides for StarRocks/Doris #7368

OpenAPI specs for Fileset/File #6860

Fix access control docs #7195

Update model privilege docs #7555

Typo fixes #7448, #7647

Remove incubating status markers #7492

Add 0.9.1 release notes #7485

Build & Infra​

Fix Helm chart versioning #7129, #7134

Upgrade Kyuubi dependency #7480

Credits

FANNG1 Abyss-lord jerqi jerryshao slimtom95 flaming-archer yunchipang KyleLin0927 xiaozcy diqiu50 yuqi1129 ziqiangliang carl239 LauraXia123 guov100 senlizishi fivedragon5 justinmclean Jackeyzhe Spiritedswordsman su8y

Apache Gravitino 0.9.0 - Focus on AI, data governance, and security with multi-dimensional feature upgrade

· 4 min read
Rory Qi
Rory Qi
PMC Member

Gravitino 0.9.0 focuses on advancements in AI, data governance, and security. Many of its new features are already being used in production environments. The release has attracted strong interest from users from well-known companies, with AI and security capabilities drawing attention.

In this version, the community optimized the user experience for fileset catalogs and model catalogs, making it easier for users to manage their unstructured AI data and model data.

The community added a new data lineage interface. Users can now implement a custom data lineage plugin to adapt to their own system.

For security, the community has corrected some privilege semantics and fixed authorization plugin corner cases to make the entire system more robust.

Model Catalog​

Before 0.9.0, the model catalog was immutable, which was not flexible. In the new version, users can alter models and model versions and add tags #6626 #6222.

Fileset Catalog​

Gravitino now supports multiple named storage locations within a single fileset and placeholder-based path generation.

With multiple location support, users can reference data across different file systems (HDFS, S3, GCS, local, etc.) through a unified fileset interface, each with a unique location name.

The placeholder feature allows dynamic storage path generation using the {{placeholder}} syntax, automatically replacing placeholders with corresponding fileset properties.

These enhancements significantly improves the flexibility for multi-cloud environments and complex data organization patterns while maintaining a clean abstraction layer for data assets management #6681.

GVFS (Gravitino Virtual File System)​

GVFS has been enhanced to support accessing multiple locations within filesets. Users can now select which location to use through configuration properties, environment variables, or fileset default settings.

GVFS has also been refactored with a pluggable architecture allowing custom operations and hooks. This enables users to extend functionality through operations_class and hook_class configuration options for more flexible integration with their specific infrastructure #6938.

Security​

The new version has added privileges for the data model and corrected some privilege semantics. It has also fixed some bugs with the Ranger path-based plugin #6620 #6575 #6821 #6864. All of the user-related, group-related, and role-related events are now supported for the event system #2969.

Data Lineage​

The community added a data lineage interface that follows the OpenLineage API specification. Users can implement their custom data lineage plugin to adapt to their system #6617.

Core​

The community cared about performance. Performance was improved by reducing the scope of the lock and batch reading data from storage #6744 #6560 #2870.

CLI​

Additionally, there is one more change worth mentioning. Users no longer need to rely on the alias command to use the CLI. Instead, the community provided a convenient script located at ./bin/gcli.sh so that a user can directly use the CLI client #5383.

Connector​

Both the Flink connector and the Spark connector added JDBC support #6233 #6164.

Chart​

Deploying Gravitino on Kubernetes with a fully customizable configuration #6594.

Overall​

Gravitino 0.9.0 focuses on advancements in AI, data governance, and security. We thank the Gravitino community for their continued support and valuable contributions. We can continue to innovate and build thanks to all our users' feedback. Thank you for taking the time to read this! To dive deeper into the Gravitino 0.9.0 release, explore the full documentation. Your feedback is greatly valued and helps shape the future of the Gravitino project and community.

Credits​

JavedAbdullah AndreVale69 Brijeshthummar02 cool9850311 liuchunhao danhuawang unknowntpo FANNG1 tsungchih jerryshao justinmclean zhoukangcn Abyss-lord amazingLyche yuqi1129 Pranaykarvi puchengy LauraXia123 tengqm rud9192 antony0016 frankvicky TEOTEO520 TungYuChiang sunxiaojian xunliu LuciferYang diqiu50 zhengkezhou1 caican00 granewang yunchipang jerqi mchades rickyma Xander-run flaming-archer waukin lsyulong luoshipeng FourFriends this-user vitamin43 hdygxsj liangyouze

Apache, Apache Fink, Apache Hive, Apache Hudi, Apache Iceberg, Apache Ranger, Apache Spark, Apache Paimon and Apache Gravitino are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.