Ice catalog hangs if metadata files are accidentally deleted #57
Description
Deleting metadata files for Iceberg tables managed by ice-rest-catalog using a command like 'aws s3 rm' causes SHOW TABLES commands on the corresponding ClickHouse database to hang. There is no obvious way to clean up the missing metadata other than deleting storage for etcd and restarting.
(Obviously this is not something you would do intentionally but it's easy to do by accident or if there is a bug.)
To reproduce:
- Set up a database in ClickHouse that points to the REST catalog.
- Create a table in the ice catalog using 'ice insert default.foo -p '.
- Confirm that
SHOW TABLES FROM <ice-catalog-name>shows the new table without hanging. - Use
aws s3 rm --recursive s3 s3:<bucket>/default/foo/metadata/to remove the table metadata. - Run SHOW TABLES again. It will hang.
If you inspect the log in the ice-rest-catalog you will see that it is filled with messages like the following:
2025-08-29 03:34:56 [15657818-37] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token GET v1/namespaces/default/tables/datetime_examples
2025-08-29 03:34:56 [15657818-37] ERROR c.a.i.r.c.i.r.RESTCatalogServlet > @token GET v1/namespaces/default/tables/datetime_examples
org.apache.iceberg.exceptions.NotFoundException: Location does not exist: s3://rhodges-ice-rest-catalog-demo/default/datetime_examples/metadata/00001-2e96c257-96a8-4c71-9505-9bcf9d3679ae.metadata.json
at org.apache.iceberg.aws.s3.S3InputStream.openStream(S3InputStream.java:242)
at org.apache.iceberg.aws.s3.S3InputStream.openStream(S3InputStream.java:225)
at org.apache.iceberg.aws.s3.S3InputStream.positionStream(S3InputStream.java:221)
at org.apache.iceberg.aws.s3.S3InputStream.read(S3InputStream.java:143)
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:287) [6 skipped]
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:280)
at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:180)
at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$2(BaseMetastoreTableOperations.java:206)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:206)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:176)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:167)
at com.altinity.ice.rest.catalog.internal.etcd.EtcdCatalog$EtcdCatalogTableOperations.doRefresh(EtcdCatalog.java:468)
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:88)
at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:71)
at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:49)
at org.apache.iceberg.rest.CatalogHandlers.loadTable(CatalogHandlers.java:328)
at com.altinity.ice.rest.catalog.internal.rest.RESTCatalogAdapter.handle(RESTCatalogAdapter.java:213)
at com.altinity.ice.rest.catalog.internal.rest.RESTCatalogMiddlewareTableConfig.handle(RESTCatalogMiddlewareTableConfig.java:33)
at com.altinity.ice.rest.catalog.internal.rest.RESTCatalogMiddlewareTableAWCredentials.handle(RESTCatalogMiddlewareTableAWCredentials.java:40)
at com.altinity.ice.rest.catalog.internal.rest.RESTCatalogServlet.handle(RESTCatalogServlet.java:126)
at com.altinity.ice.rest.catalog.internal.rest.RESTCatalogServlet.doGet(RESTCatalogServlet.java:157)
at com.altinity.ice.rest.catalog.internal.rest.RESTCatalogAuthorizationHandler.next(RESTCatalogAuthorizationHandler.java:84) [6 skipped]
at com.altinity.ice.rest.catalog.internal.rest.RESTCatalogAuthorizationHandler.handle(RESTCatalogAuthorizationHandler.java:54)
[28 skipped]
Caused by: software.amazon.awssdk.services.s3.model.NoSuchKeyException: The specified key does not exist. (Service: S3, Status Code: 404, Request ID: AS080NV1RYZPY3TG, Extended Request ID: Qunxuc54ZkvhAdoulVmLFdblAW905kdJkFMgwhEJk0NdBSEeN5qrQFRCrIZr2WQHPhpSAdssd5l63nK4lvzUa4HZZaaT1r04R8Skqa1Pjhs=) (SDK Attempt Count: 1)
at software.amazon.awssdk.services.s3.model.NoSuchKeyException$BuilderImpl.build(NoSuchKeyException.java:146)
at software.amazon.awssdk.services.s3.model.NoSuchKeyException$BuilderImpl.build(NoSuchKeyException.java:94)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52) [21 skipped]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:60)
at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:5864)
at org.apache.iceberg.aws.s3.S3InputStream.openStream(S3InputStream.java:240)
at org.apache.iceberg.aws.s3.S3InputStream.openStream(S3InputStream.java:225)
at org.apache.iceberg.aws.s3.S3InputStream.positionStream(S3InputStream.java:221)
at org.apache.iceberg.aws.s3.S3InputStream.read(S3InputStream.java:143)
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:287) [6 skipped]
at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:280)
at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:180)
at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$2(BaseMetastoreTableOperations.java:206)
at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413)
at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:206)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:176)
at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:167)
at com.altinity.ice.rest.catalog.internal.etcd.EtcdCatalog$EtcdCatalogTableOperations.doRefresh(EtcdCatalog.java:468)
at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:88)
at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:71)
at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:49)
at org.apache.iceberg.rest.CatalogHandlers.loadTable(CatalogHandlers.java:328)
at com.altinity.ice.rest.catalog.internal.rest.RESTCatalogAdapter.handle(RESTCatalogAdapter.java:213)
at com.altinity.ice.rest.catalog.internal.rest.RESTCatalogMiddlewareTableConfig.handle(RESTCatalogMiddlewareTableConfig.java:33)
... 66 common frames omitted