[590] Add RFC for XCatalogSync - Synchronize tables across catalogs #605
Conversation
vinishjail97
force-pushed
the
590-RFC-CatalogSync
branch
from
d6d886b to
41c67b6
Compare
vinishjail97
force-pushed
the
590-RFC-CatalogSync
branch
from
41c67b6 to
a6e8be2
Compare
vinishjail97
force-pushed
the
590-RFC-CatalogSync
branch
from
a3cb743 to
ece8d98
Compare
| ## Abstract | ||
|
|
||
| Users of Apache XTable (Incubating) today can translate metadata across table formats (iceberg, hudi, and delta) and use the tables in different platforms depending on their choice. | ||
| Today there's still some friction involved in terms of usability because users need to explicitly [register](https://xtable.apache.org/docs/catalogs-index) the tables in the catalog of their choice (glue, HMS, unity, bigLake etc.) |
There was a problem hiding this comment.
Hi @vinishjail97, thanks for sharing the details. I think catalog sync is a useful feature. One of the key value adds of catalogs is governance, particularly access control. All the catalogs mentioned here provide the ability to grant different privileges to roles. The proposed catalog sync in XTable replicates the table across catalogs. What are your thoughts about porting the governance features?
There was a problem hiding this comment.
I think this is a good long term vision but deserves its own RFC to draw up the entities since it will expand beyond the tables now.
| ## Implementation | ||
|
|
||
| Introducing the following interfaces. [[PR]]( https://github.com/apache/incubator-xtable/pull/603) | ||
| 1. `CatalogSyncClient`: This interface contains methods that are responsible for creating table, refreshing table metadata, dropping table etc. in target catalog. Consider this interface as a translation layer between InternalTable and the catalog's table object. |
There was a problem hiding this comment.
I don't think table DDL operations are related to InternalTable. But you bring up an important point. The table format and the catalog layer are two different layers in the analytics stack. Currently, XTable only supports conversion of table format level metadata, which is captured by current InternalTable. However, the proposed feature extends to the catalog layer where catalog level metadata translation takes place. So, in effect, this feature syncs InternalCatalog object. What are your thoughts?
There was a problem hiding this comment.
Currently the feature allows you to sync to a catalog without a source catalog by just registering those tables in the target catalogs. I think in its current form this is still syncing at the table level. If we want to support new entities to sync like access control and permissions level metadata, we are in a position to expand what the clients can do.
| 1. `catalogId`: A user-defined unique identifier for the catalog, allows user to sync table to multiple catalogs of the same name/type eg: HMS catalog with url1, HMS catalog with url2. | ||
| 2. `catalogType`: The type of the source catalog. This might be a specific type understood by XTable, such as Hive, Glue etc. | ||
| 3. `catalogSyncClientImpl`(optional): A fully qualified class name that implements the interface for `CatalogSyncClient`, it can be used if the implementation for catalogType doesn't exist in XTable. | ||
| 4. `catalogConversionSourceImpl`(optional): A fully qualified class name that implements the interface for `CatalogConversionSource`, it can be used if the implementation for catalogType doesn't exist in XTable. |
There was a problem hiding this comment.
It is not clear what is the difference in the role of CatalogConversionSource and CatalogSyncClient. Could you please clarify. In case of table sync, only TableSource exists, there is no TableClient.
Important Read
What is the purpose of the pull request
Adds RFC for XCatalogSync - Synchronize tables across catalogs.
Brief change log
(for example:)
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.