HIVE-29672#6552
Open
dengzhhu653 wants to merge 22 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors the standalone metastore RawStore surface by introducing dedicated sub-store interfaces/implementations (e.g., column stats, constraints, WLM) and wiring many former RawStore methods through new default delegations. It also updates metastore tests and verification helpers to exercise the new store boundaries (including DirectSQL vs JDO comparison paths).
Changes:
- Add new metastore sub-store interfaces (
ColStatsStore,ConstraintStore,WLMStore) and implementations (ColStatsStoreImpl,ConstraintStoreImpl,WLMStoreImpl). - Convert a number of
RawStoremethods intodefaultmethods delegating tounwrap(...)stores; adjustCachedStoreand tests accordingly. - Update
VerifyingObjectStoreto verify column-stats and partition-stats behavior viaColStatsStore/TableStoreunder DirectSQL configurator toggling.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/VerifyingObjectStore.java | Switches verification logic to ColStatsStore/TableStore with DirectSQL toggling. |
| standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java | Updates a column-validation test to call ColStatsStoreImpl.validateTableCols. |
| standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java | Adds default delegations to ColStatsStore/ConstraintStore/WLMStore via unwrap. |
| standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metastore/impl/WLMStoreImpl.java | New WLM store implementation extracted into its own class. |
| standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metastore/impl/TableStoreImpl.java | Refactors table/partition conversion helpers and adds getMTable(TableName, ...). |
| standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metastore/impl/ConstraintStoreImpl.java | New constraints store implementation (PK/FK/UK/NN/DC/CC operations). |
| standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metastore/impl/ColStatsStoreImpl.java | New column-stats store implementation (read/update/delete/aggregate paths). |
| standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metastore/iface/WLMStore.java | New WLM store interface with @MetaDescriptor. |
| standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metastore/iface/TableStore.java | Adds AttachedMTableInfo and getMTable(TableName, ...). |
| standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metastore/iface/ConstraintStore.java | New constraint store interface with @MetaDescriptor. |
| standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metastore/iface/ColStatsStore.java | New column-stats store interface with @MetaDescriptor. |
| standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metastore/GetHelper.java | Exposes savepoint helpers (now public) for use by new store implementations. |
| standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java | Removes methods now satisfied by RawStore defaults (delegation via unwrap). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+347
to
+351
| final String fkColType = getColumnFromTableColumns(childCols, fkColumnName).getType(); | ||
| fkSignature.append( | ||
| generateColNameTypeSignature(fkColumnName, fkColType)); | ||
| referencedKSignature.append( | ||
| generateColNameTypeSignature(pkColumnName, fkColType)); |
Comment on lines
+538
to
+543
| FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. MetaException(message:JDODataStoreException: Update of object with id "1[OID]org.apache.hadoop.hive.metastore.model.MWMResourcePlan" using statement "UPDATE WM_RESOURCEPLAN SET "NAME"=? WHERE RP_ID=?" failed : org.apache.derby.shared.common.error.DerbySQLIntegrityConstraintViolationException: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'UNIQUE_WM_RESOURCEPLAN' defined on 'WM_RESOURCEPLAN'. | ||
| #### A masked pattern was here #### | ||
| Caused by: ERROR 23505: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'UNIQUE_WM_RESOURCEPLAN' defined on 'WM_RESOURCEPLAN'. | ||
| #### A masked pattern was here #### | ||
|
|
||
| Root cause: ERROR 23505: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'UNIQUE_WM_RESOURCEPLAN' defined on 'WM_RESOURCEPLAN'.) |
asf-ci-hive
added
tests unstable
tests pending
and removed
tests pending
tests unstable
labels
dengzhhu653
commented
Jun 20, 2026
| PREHOOK: type: ALTER RESOURCEPLAN | ||
| PREHOOK: Output: dummyHostnameForTest | ||
| FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. AlreadyExistsException(message:Resource plan name should be unique: ) | ||
| FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. MetaException(message:JDODataStoreException: Update of object with id "1[OID]org.apache.hadoop.hive.metastore.model.MWMResourcePlan" using statement "UPDATE WM_RESOURCEPLAN SET "NAME"=? WHERE RP_ID=?" failed : org.apache.derby.shared.common.error.DerbySQLIntegrityConstraintViolationException: The statement was aborted because it would have caused a duplicate key value in a unique or primary key constraint or unique index identified by 'UNIQUE_WM_RESOURCEPLAN' defined on 'WM_RESOURCEPLAN'. |
Member
Author
There was a problem hiding this comment.
This diff is due to the time transaction commits, before the change alterResourcePlan throws this DerbySQLIntegrityConstraintViolationException on commitTransaction, and in this method will catch and re-throw Metastore specific exception, as we move the transaction management to TransactionHandler, the time transaction commits happens after the call, so we wont be able to catch the exception in WMStoreImpl and transform the exception as before.
|
Member
Author
|
I assume this is the last biggest refactors, @ngsg @deniskuzZ @wecharyu @okumin could you take a look if have secs? Thanks in advance! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?