Skip to content

Conversation

davidrabinowitz
Copy link
Member

No description provided.

@davidrabinowitz
Copy link
Member Author

/gcbrun

@yigress
Copy link
Collaborator

yigress commented Apr 17, 2023

does this pass acceptance test?
I tried with it on dataproc-21 got failures

681318825121_0036_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE
Vertex Input: nation initializer failed, vertex=vertex_1681318825121_0036_1_00 [Map 1], java.lang.NoSuchFieldError: VIRTUAL_COLUMN_NAMES
at com.google.cloud.hive.bigquery.connector.input.BigQueryInputSplit.createSplitsFromBigQueryReadStreams(BigQueryInputSplit.java:180)
at com.google.cloud.hive.bigquery.connector.input.BigQueryInputFormat.getSplits(BigQueryInputFormat.java:44)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:519)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:765)
at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:243)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:281)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:272)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:272)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:256)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

interesting that the complaint field VirtualColumn.VIRTUAL_COLUMN_NAMES is from hive-exec.

import org.slf4j.LoggerFactory;
import shaded.hivebqcon.com.google.cloud.bigquery.connector.common.*;
import shaded.hivebqcon.com.google.common.annotations.VisibleForTesting;
import shaded.hivebqcon.com.google.common.collect.ImmutableList;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't know if the issue of java.lang.NoSuchFieldError: VIRTUAL_COLUMN_NAMES is relate to the change here.

@functicons
Copy link
Contributor

/gcbrun

1 similar comment
@davidrabinowitz
Copy link
Member Author

/gcbrun

Copy link
Contributor

@functicons functicons left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@davidrabinowitz davidrabinowitz merged commit 769ff40 into GoogleCloudDataproc:main Apr 20, 2023
@davidrabinowitz davidrabinowitz deleted the shading branch April 20, 2023 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants