![]() : .extensions.IcebergSparkSessionExtensionsĪt java.base/(Unknown Source)Īt java.base/(Unknown Source)Īt java.base/0(Native Method)Īt java.base/(Unknown Source)Īt .Utils$.classForName(Utils.scala:206)Īt .SparkSession$.$anonfun$applyExtensions$1(SparkSession.scala:1160)Īt .SparkSession$.$anonfun$applyExtensions$1$adapted(SparkSession.scala:1158)Īt .foreach(ResizableArray.scala:62)Īt .foreach$(ResizableArray.scala:55)Īt .foreach(ArrayBuffer.scala:49)Īt .SparkSession$.org$apache$spark$sql$SparkSession$$applyExtensions(SparkSession.scala:1158)Īt .SparkSession.(SparkSession.scala:101)Īt java.base/.newInstance0(Native Method)Īt java.base/.newInstance(Unknown Source)Īt java.base/.newInstance(Unknown Source)Īt java.base/.newInstance(Unknown Source)Īt (MethodInvoker.java:247)Īt (ReflectionEngine.java:357)Īt (ConstructorCommand.java:80)Īt (ConstructorCommand.java:69)Īt py4j.Gatewa圜n(Gatewa圜onnection.java:238)Īt java.base/(Unknown Source)Īt .$(ProgressReporter.scala:350)Īt .(StreamExecution.scala:69)Īt .(MicroBatchExecution.scala:564)Īt .$anonfun$runActivatedStream$2(MicroBatchExecution.scala:223)Īt 0$mcV$sp.apply(JFunction0$mcV$sp.java:23)Īt .(ProgressReporter.scala:352)Īt .$anonfun$runActivatedStream$1(MicroBatchExecution.scala:191)Īt .(TriggerExecutor.scala:39)Īt .(MicroBatchExecution.scala:185)Īt .$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:334)Īt .$$anon$1.run(StreamExecution.scala:245)Ģ2/11/24 09:18:45 ERROR MicroBatchExecution: Query terminated with error `22/11/24 09:18:36 WARN SparkSession: Cannot use .extensions.IcebergSparkSessionExtensions to configure session extensions. ".extensions.IcebergSparkSessionExtensions", REMOTE_CATALOG_NAME: str = "remote_glue_catalog" We have tried with the following versions:Įxtra configuration included into the spark session: We have used iceberg successfully before in a local spark session before (not structured streaming). ![]() Spark structured streaming worked well until we tried to add the iceberg extension. The application is containerised and behind the scenes running on EKS. We are trying to use spark structured streaming in combination with iceberg.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |