text:
How to use spark-csv in a Spark standalone cluster mode

Hi @falaki @HyukjinKwon , I'm getting started with Spark, but I have a trouble with spark-csv.
When I use local mode, it works.
  lazy val conf = {
    new SparkConf(false)
      .setMaster("local[*]")
      .setAppName("Spark Tutorial")
  }

  lazy val sparkContext = new SparkContext(conf)
  lazy val sparkSQLContext = SQLContext.getOrCreate(sparkContext)

However, if i use spark standalone cluster mode, an error occurred. The codes as following:
  lazy val conf = {
    new SparkConf(false)
      .setMaster("spark://spark-master:7077")
      .setAppName("Spark Tutorial")
  }

  lazy val sparkContext = new SparkContext(conf)
  lazy val sparkSQLContext = SQLContext.getOrCreate(sparkContext)

Error message:
16/07/14 09:59:12 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, spark-master): java.lang.ClassNotFoundException: com.databricks.spark.csv.CsvRelation$$anonfun$firstLine$1
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:68)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
    at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
    at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

16/07/14 09:59:12 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 1, spark-master, partition 0,PROCESS_LOCAL, 2196 bytes)
16/07/14 09:59:12 INFO TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1) on executor spark-master: java.lang.ClassNotFoundException (com.databricks.spark.csv.CsvRelation$$anonfun$firstLine$1) [duplicate 1]
16/07/14 09:59:12 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID 2, spark-master, partition 0,PROCESS_LOCAL, 2196 bytes)
16/07/14 09:59:12 INFO TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2) on executor spark-master: java.lang.ClassNotFoundException (com.databricks.spark.csv.CsvRelation$$anonfun$firstLine$1) [duplicate 2]
16/07/14 09:59:12 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID 3, spark-master, partition 0,PROCESS_LOCAL, 2196 bytes)
16/07/14 09:59:12 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on spark-master:42484 (size: 2.8 KB, free: 511.1 MB)
16/07/14 09:59:13 INFO TaskSetManager: Lost task 0.3 in stage 0.0 (TID 3) on executor spark-master: java.lang.ClassNotFoundException (com.databricks.spark.csv.CsvRelation$$anonfun$firstLine$1) [duplicate 3]
16/07/14 09:59:13 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
16/07/14 09:59:13 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
16/07/14 09:59:13 INFO TaskSchedulerImpl: Cancelling stage 0
16/07/14 09:59:13 INFO DAGScheduler: ResultStage 0 (first at CsvRelation.scala:267) failed in 2.467 s
16/07/14 09:59:13 INFO DAGScheduler: Job 0 failed: first at CsvRelation.scala:267, took 2.578672 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, spark-master): java.lang.ClassNotFoundException: com.databricks.spark.csv.CsvRelation$$anonfun$firstLine$1
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:68)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
    at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
    at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
    at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1328)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
    at org.apache.spark.rdd.RDD.take(RDD.scala:1302)
    at org.apache.spark.rdd.RDD$$anonfun$first$1.apply(RDD.scala:1342)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
    at org.apache.spark.rdd.RDD.first(RDD.scala:1341)
    at com.databricks.spark.csv.CsvRelation.firstLine$lzycompute(CsvRelation.scala:267)
    at com.databricks.spark.csv.CsvRelation.firstLine(CsvRelation.scala:263)
    at com.databricks.spark.csv.CsvRelation.inferSchema(CsvRelation.scala:240)
    at com.databricks.spark.csv.CsvRelation.<init>(CsvRelation.scala:73)
    at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:162)
    at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:44)
    at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109)
    at org.lavenderx.tutorial.spark.sql.datasource.CSVFileExample$.main(CSVFileExample.scala:12)
    at org.lavenderx.tutorial.spark.sql.datasource.CSVFileExample.main(CSVFileExample.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.lang.ClassNotFoundException: com.databricks.spark.csv.CsvRelation$$anonfun$firstLine$1
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:68)
    at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
    at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
    at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
    at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
16/07/14 09:59:13 INFO SparkContext: Invoking stop() from shutdown hook

BTW, the Scala version is 2.10.6, Spark version is 1.6.2 and spark-csv is the latest.
Is there something with me? Could you provide some advice?
Thanks in advance!:blush:

Do you mind if I ask whether it happens only with this library? I just want to be sure if this is an issue specific to this library.

@lavenderx did you manage to resolve this ? I am facing the same issue and any help will be appreciated.

authors: Guru-B HyukjinKwon lavenderx

repo: databricks/spark-csv


text:
Support for different encodings

https://github.com/databricks/spark-xml/issues/18
Currently, this library only support UTF-8. This PR lets this library support different encodings.

Current coverage is 86.82%

Merging #54 into master will increase coverage by +0.99% as of c107780

@@            master     #54   diff @@
======================================
  Files           12      12
  Stmts          600     615    +15
  Branches        82      83     +1
  Methods          0       0
======================================
+ Hit            515     534    +19
  Partial          0       0
+ Missed          85      81     -4

Review entire Coverage Diff as of c107780

Powered by Codecov. Updated on successful CI builds.

I am going to merge this.

Merged.

Merged.

source: gharchive/pull-request

authors: HyukjinKwon codecov-io

repo: databricks/spark-xml

license_type: permissive

license_source: github-api


text:
Hate crime data download

The hate crime data is available at https://ucr.fbi.gov/hate-crime . The data is divided into 14 different tables(and a 15th participation details table) for each year. The scripts goes through the range of input years and collects the download links for all the tables, finally downloading and storing them.
NOTE: the requirements.txt is modified to add beautiful soup package to parse through the pages.

Thanks for the review @ajaits I have tried to address most the comments. I have avoided using breaks and continue for easier debugging and I usually try to avoid using them. If you want i can make the requested changes.

repo: datacommonsorg/data

license_source: bigquery


text:
Document how to ingest from IETF Mailman 2

IETF has switched to Mailman 2 for its mailing list archives.
This makes the URL-based scraper ineffective for mailing lists that are active after 2022.
We need better documentation on how to use BigBang with the new mailing list archive.
The quickest route is

log in to the IETF archives and download the mboxes
ingest from the mboxes

There may be a way to get the mbox directly through HTTP with a valid username/password, which would certaintly make a collect-mail script work more easily.
https://mail.python.org/archives/list/mailman-users@python.org/thread/QYWDGQ4TNO2SJAH4MDDYIOPTKWR74E7B/

There is already a Mailmain 2 ingestor that pulls data into a MongoDB database in the glasgow-IPL ietfdata package.
We might wrap/use this:
https://github.com/glasgow-ipl/ietfdata/blob/cc78853f3dc8435c1cbe2711e38ead32354eec17/ietfdata/mailarchive2.py

source: gharchive/issue

repo: datactive/bigbang

license: mit


text:
Correction for raw affiliation UMR ABSys - Agrosystèmes Biodiversifiés ( INRAE - Campus Supagro Montpellier 2 place Viala 34060 MONTPELLIER CEDEX 2  - France)

Correction needed for raw affiliation UMR ABSys - Agrosystèmes Biodiversifiés ( INRAE - Campus Supagro Montpellier 2 place Viala 34060 MONTPELLIER CEDEX 2  - France)
raw_affiliation_name: UMR ABSys - Agrosystèmes Biodiversifiés ( INRAE - Campus Supagro Montpellier 2 place Viala 34060 MONTPELLIER CEDEX 2  - France)
new_rors: 003vg9w96;03rnk6m14;00n2c0309
previous_rors: 003vg9w96;03rnk6m14
works_examples: W3152720850;W4287104733;W4207087745;W4200174583;W4401408386;W3125635929;W4210339903;W4387229115;W4322620623;W4380574474;W4385127864;W4318408276;W4379517179;W4307825432;W4376104434;W4380049587;W4397022621
contact: 6e4daaedafe7bded48c91802a804e7a1:aaa54d3105872bb7ae73b2cf17408ca52153 @ inrae.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-09. The new affiliations should be visible within the next 7 days.

repo: dataesr/openalex-affiliations

license: MIT


text:
Correction for raw affiliation  aperfeiçoar;  avaliação cega pelos pares de todos manuscritos publicados, com base em critérios de neutralidade e imparcialidade acadêmica. Editora Artemis Curitiba-PR Brasil www.;  confiabilidade é exclusiva dos autores. A Editora Artemis, em seu compromisso de manter;  qualidade e confiabilidade dos trabalhos que publica, conduz; Curitiba-PR Brasil; Editora Artemis Curitiba-PR Brasil; pelo conteúdo dos artigos e seus dados, em sua forma, correção

Correction needed for raw affiliation  aperfeiçoar;  avaliação cega pelos pares de todos manuscritos publicados, com base em critérios de neutralidade e imparcialidade acadêmica. Editora Artemis Curitiba-PR Brasil www.;  confiabilidade é exclusiva dos autores. A Editora Artemis, em seu compromisso de manter;  qualidade e confiabilidade dos trabalhos que publica, conduz; Curitiba-PR Brasil; Editora Artemis Curitiba-PR Brasil; pelo conteúdo dos artigos e seus dados, em sua forma, correção
raw_affiliation_name:  aperfeiçoar;  avaliação cega pelos pares de todos manuscritos publicados, com base em critérios de neutralidade e imparcialidade acadêmica. Editora Artemis Curitiba-PR Brasil www.;  confiabilidade é exclusiva dos autores. A Editora Artemis, em seu compromisso de manter;  qualidade e confiabilidade dos trabalhos que publica, conduz; Curitiba-PR Brasil; Editora Artemis Curitiba-PR Brasil; pelo conteúdo dos artigos e seus dados, em sua forma, correção
new_rors: 01bv0ma93;029jgxb70
previous_rors: 031ah7413;01bv0ma93;029jgxb70
works_examples: W4293700425;W4293586078;W4293703028;W4293586075;W4293700576;W4293700552;W4293700663;W4293702174;W4293700423;W4293499756;W4293700719;W4293702571;W4293700430;W4293702886;W4293586061;W4293499663;W4293702187;W4293703240;W4293586076;W4293499546;W4293499848;W4293499909;W4293702677
contact: a3a96db353730eee26df6d287ea86353:1c084416db7f1a431397eca862 @ univ-cotedazur.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-10. The new affiliations should be visible within the next 7 days.


text:
Correction for raw affiliation CIRAD, UPR Recyclage et risque, F-34398 Montpellier, France

Correction needed for raw affiliation CIRAD, UPR Recyclage et risque, F-34398 Montpellier, France
raw_affiliation_name: CIRAD, UPR Recyclage et risque, F-34398 Montpellier, France
new_rors: 05kpkpg04;03ze73t75
previous_rors: 01h4rra97;05kpkpg04
works_examples: W4387642818;W4389101476
contact: c928353c0cdeefccb402f70f7373638e:00226608266a418bb9d4721fb6b2 @ cirad.fr

This issue was accepted and ingested by the OpenAlex team on 2024-12-19. The new affiliations should be visible within the next 7 days.


text:
Avoid double-wal

Is your feature request related to a problem? Please describe.
In the RocksDB example, as well as numerous places in the documentation and discussions, it appears that apply_to_state_machine() is meant to be atomic and durable. Unfortunately, for some store implementations, persisting the whole data structure to disk could be quite costly. An alternative approach is shown in the RocksDB example, where rocks writes it's wal (but not the actual data) to disk, and this is better, but doesn't this result in a double-wal? Raft has it's own log (i.e. append_to_log()), then the application has to have it's own wal in order to complete apply_to_state_machine() quickly and durably.
Describe the solution you'd like
Instead of nested wals, would it be possible to allow the application to:

not persist anything during apply_to_state_machine() and keep changes in memory
the app can periodically flush to disk (data + last_applied_index) based on it's own cron job
if the app crashes, it replays the raft logs since last_applied_index (on it's own - without being told by openraft)
now that the state machine is rebuilt, the app re-joins the cluster
Openraft is unaware that apply_to_state_machine() doesn't immediately persist, but this is fine because the app state machine is still restored prior to resuming anything raft related.

Forgive me, I may be misunderstanding how this is supposed to work, but it appears #437 is effectively asking the same thing? I saw another issue (#208) out there about pipelining requests to apply_to_state_machine() to help alleviate this issue, but perhaps the solution above is either better or at least complimentary?
Describe alternatives you've considered

Creating an app wal in addition to flushing log entries to disk (WAL for both append_to_log and apply_to_state_machine)
Flushing full state to disk in apply_to_state_machine() and having a really slow application.

You are right. An application only needs one WAL. Raft is just a distributed WAL.

not persist anything during apply_to_state_machine() and keep changes in memory
the app can periodically flush to disk (data + last_applied_index) based on it's own cron job

It's OK not to persist the state, except you may want to persist the last_applied_index.
If you do not persist any state at all, when a node restarts, just rebuild state machine from the last snapshot, and wait for openraft to inform RaftStorage to re-commit and re-apply other logs.
Or last_applied_index should be persisted before apply_to_state_machine() returns. And when a node starts, rebuild state machine from last snapshot, then re-apply logs up to last_applied_index.

Forgive me, I may be misunderstanding how this is supposed to work, but it appears #437 is effectively asking the same thing?

Yes. it's meant to get rid of the expensive persisting state operation.
#208 is another thing about state machine. It's about batch-applying logs, and won't conflict with what you want to do.

Thank you @drmingdrmer for taking the time to provide this detailed description.
I did have one remaining question regarding:

just rebuild state machine from the last snapshot, and wait for openraft to inform RaftStorage to re-commit and re-apply other logs.

Assuming we haven't flushed either the state or last_applied_index the  during the last apply_to_state_machine(), when the node crashes and restarts, do we need to do any special initialization logic? Or does openraft call last_applied_state() and as long as we return the correct results, openraft will do all the necessary initialization (re-applying logs to the state machine) for us?

n/m, I think you answered the above question here: https://github.com/datafuselabs/openraft/discussions/616#discussioncomment-4444773

Whether to apply these log has to be left to openraft to decide

ty!! :smile:

authors: avantgardnerio drmingdrmer

repo: datafuselabs/openraft


text:
Pinned columns are not displayed properly with some row source options for the table

Steps to reproduce:

Open a table
Pin some columns
Select one of the following options for table's row source:

Selected
MouseOverRow
MouseOverGroup
FilteredSelected
SelectedOrCurrent

Use another viewer to change which rows are displayed in the table (select somthing, filter, mouse over etc.)

Expected results: Pinned columns area display the same rows as the main part of the table.
Actual results: Pinned columns area is not updated properly.
'MouseOverGroup' option (pinned columns are not updated at all):

'Selected' option (pinned columns are updated on click on the pinned columns area, otherwise previous selection is displayed):

The fix for mouse over issue is available in v 1.1.25 of PowerGrid package. Selection or filtering from other viewers cause pinned column to update accordingly. No issue was observed.

@dgosi Could you please specify the package version and let me know if there are any platform updates I need to wait for before testing? I'm still reproducing the bug on PowerGrid 1.1.25.

I am using dev  server. Could you please describe the exact steps and the viewers to reproduce the issue? For me filtering, mouse over and selection do update pinned columns.

We can experience different behavior due to the operating system and browser differences. I use Windows 10, Chrome
To reproduce on dev:

Open SPGI (I use Order and Hide columns to move the Chemist 521 column to the first position).
Add a grid viewer:

Pin the Chemist 521 and ID columns
Set Row Source to Selected, MouseOverGroup, CurrentRow

Select rows in the initial grid.
Add a pie chart, hover over its groups
Actual result: the data in the pinned columns is not updated until you hover over the grid

Thank you for providing the precise steps! MouseOverGroup and switching the rowSource property have been fixed. Everything else seems to work fine. All fixes are in v 1.1.27 of PowerGrid package.

I still reproduce a delay in updating pinned columns, when Row Source is set to:

MouseOverRow
FilteredSelected or Selected
SelectedOrCurrent

PowerGrid 1.1.33

Add 2nd table to the page besides the default grid and pin a couple of columns in it
Configure this additional table to show FilteredSelected rows
Select multiple rows
Filter out some rows, so that additional table shows only 1-2 items
Reset filter

Expected: table with row source FilteredSelected is re-rendered on resetting filters
Actual: pinned columns area is re-rendered only when one of the table on the page is hovered.

Fixed in DG 1.17, PowerGrid 1.2.1

authors: ahadzhanian dgosi katerina-stepanova

repo: datagrok-ai/public


text:
Validation report is not displayed

As reported in https://github.com/datahq/datahub-qa/issues/209
Acceptance criteria

[x] validation reports are shown

Tasks

[x] investigate the sources
[x] fix

Analysis
Datahub reports are generated by using goodtables-py v1.5.1, which was working with goodtables-ui@1.1.x. Since new version 1.2.x released, we're getting errors when rendering reports:
Uncaught TypeError: Cannot set property '2' of undefined

Uncaught TypeError: Cannot read property 'map' of undefined

E.g., see https://jsfiddle.net/n7v8eong/ console logs.
Looking at changelog here https://github.com/frictionlessdata/goodtables-py/#v20 it seems like goodtables-ui@1.2.x doesn't support old reports anymore. I've tried to use v1.1.x by accessing it over CDN - https://unpkg.com/goodtables-ui@1.1.3/dist/goodtables-ui.js - but it didn't help.
Raised an issue here - https://github.com/frictionlessdata/goodtables-ui/issues/23

Should be FIXED once specstore is redeployed

repo: datahq/frontend


text:
datahub performance

Hello,
I have a problem with performance on process which consume messages from kafka and push changes in elasticsearch and neo4j
i was added this envs to my gms
  extraEnvs:
    - name: SPRING_KAFKA_PROPERTIES_MAX_POLL_RECORDS
      value: '10'
    - name: SPRING_KAFKA_PROPERTIES_MAX_POLL_INTERVAL_MS
      value: '120000'
    - name: ES_BULK_REQUESTS_LIMIT
      value: '1500'
    - name: ES_BULK_FLUSH_PERIOD
      value: '2'
    - name: LOGGING_LEVEL_ORG_APACHE_KAFKA_CLIENTS_CONSUMER
      value: DEBUG
    - name: LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_KAFKA
      value: DEBUG
    - name: ELASTICSEARCH_THREAD_COUNT
      value: '15'
    - name: ES_BULK_ENABLE_BATCH_DELETE
      value: 'true'
    - name: LOGGING_LEVEL_ORG_APACHE_KAFKA_CLIENTS_CONSUMER
      value: DEBUG
    - name: LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_KAFKA
      value: DEBUG

[2024-10-18 09:01:22,092 [I/O dispatcher 1] INFO  c.l.m.s.e.update.BulkListener:61 - Successfully fed bulk request 172. Number of events: 5 Took time ms: 3
2024-10-18 09:01:40,463 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:119 - Invoking MCL hook IncidentsSummaryHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_opportunity_view_final,PROD)
2024-10-18 09:01:40,463 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:119 - Invoking MCL hook IngestionSchedulerHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_opportunity_view_final,PROD)
2024-10-18 09:01:40,463 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:119 - Invoking MCL hook EntityChangeEventGeneratorHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_opportunity_view_final,PROD)
2024-10-18 09:01:40,463 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:119 - Invoking MCL hook SiblingAssociationHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_opportunity_view_final,PROD)
2024-10-18 09:01:40,463 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.h.s.SiblingAssociationHook:109 - Urn urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_opportunity_view_final,PROD) with aspect upstreamLineage received by Sibling Hook.
2024-10-18 09:01:40,467 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.h.s.SiblingAssociationHook:244 - Associating urn:li:dataset:(urn:li:dataPlatform:dbt,_bdm.bdm_dim_opportunity_view_final,PROD) and urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_opportunity_view_final,PROD) as siblings.
2024-10-18 09:01:40,473 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:119 - Invoking MCL hook FormAssignmentHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_opportunity_view_final,PROD)
2024-10-18 09:01:40,473 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:137 - Successfully completed MCL hooks for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_opportunity_view_final,PROD)
2024-10-18 09:01:40,473 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:82 - Got MCL event key: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_request,PROD), topic: MetadataChangeLog_Versioned_v1, partition: 0, offset: 119678, value size: 143224, timestamp: 1729168196437
2024-10-18 09:01:40,473 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:106 - Invoking MCL hooks for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_request,PROD), aspect name: upstreamLineage, entity type: dataset, change type: UPSERT
2024-10-18 09:01:40,474 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:119 - Invoking MCL hook UpdateIndicesHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_request,PROD)
2024-10-18 09:01:40,479 [ThreadPoolTaskExecutor-1] INFO  c.l.m.s.e.update.ESBulkProcessor:82 - Added request id: EtcUX9vACyZAw/dPG+Inzw==, operation type: UPDATE, index: system_metadata_service_v1
2024-10-18 09:02:04,472 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:119 - Invoking MCL hook IncidentsSummaryHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_request,PROD)
2024-10-18 09:02:04,473 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:119 - Invoking MCL hook IngestionSchedulerHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_request,PROD)
2024-10-18 09:02:04,473 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:119 - Invoking MCL hook EntityChangeEventGeneratorHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_request,PROD)
2024-10-18 09:02:04,473 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:119 - Invoking MCL hook SiblingAssociationHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_request,PROD)
2024-10-18 09:02:04,473 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.h.s.SiblingAssociationHook:109 - Urn urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_request,PROD) with aspect upstreamLineage received by Sibling Hook.
2024-10-18 09:02:04,476 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.h.s.SiblingAssociationHook:244 - Associating urn:li:dataset:(urn:li:dataPlatform:dbt,_bdm.bdm_dim_request,PROD) and urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_request,PROD) as siblings.
2024-10-18 09:02:04,481 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:119 - Invoking MCL hook FormAssignmentHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_request,PROD)
2024-10-18 09:02:04,481 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeLogProcessor:137 - Successfully completed MCL hooks for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_bdm.bdm_dim_request,PROD)
](url)

but performance is very low, can you help me find bottleneck?

Hi @pilipyukaaa , which version of Datahub you are using ?

im not interesst
Den sön 20 okt. 2024 3:39 PMdeepgarg-visa @.***> skrev:

Hi @pilipyukaaa https://github.com/pilipyukaaa , which version of
Datahub you are using ?
—
Reply to this email directly, view it on GitHub
https://github.com/datahub-project/datahub/issues/11671#issuecomment-2424966781,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BLQW5VENVKYX2IPQH2TKE6LZ4OXCJAVCNFSM6AAAAABQF3O5BGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRUHE3DMNZYGE
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

hello, @deepgarg-visa i am using datahub version 0.13.3

i was update my datahub to 0.14.1 version and its still not good
2024-10-21 13:24:50,425 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:79 - Invoking MCL hook SiblingAssociationHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) 2024-10-21 13:24:50,425 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.h.s.SiblingAssociationHook:121 - Urn urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) with aspect datasetKey received by Sibling Hook. 2024-10-21 13:24:50,433 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.h.s.SiblingAssociationHook:256 - Associating urn:li:dataset:(urn:li:dataPlatform:dbt,_dds.dist_5_dds_CRM_issues,PROD) and urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) as siblings. 2024-10-21 13:24:50,438 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:97 - Successfully completed MCL hooks for consumer: generic-mae-consumer-job-client urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) 2024-10-21 13:24:50,439 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:69 - Invoking MCL hooks for consumer: generic-mae-consumer-job-client urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD), aspect name: siblings, entity type: dataset, change type: RESTATE 2024-10-21 13:24:50,439 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:79 - Invoking MCL hook FormAssignmentHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) 2024-10-21 13:24:50,439 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:79 - Invoking MCL hook UpdateIndicesHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) 2024-10-21 13:24:50,440 [ThreadPoolTaskExecutor-1] INFO  c.l.m.s.e.update.ESBulkProcessor:82 - Added request id: BXh3SoWBZKt7JlYWQUbs+w==, operation type: UPDATE, index: system_metadata_service_v1 2024-10-21 13:24:50,441 [ThreadPoolTaskExecutor-1] INFO  c.l.m.s.e.update.ESBulkProcessor:82 - Added request id: urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Aclickhouse%2C_dds.dist_5_dds_crm_issues%2CPROD%29, operation type: UPDATE, index: datasetindex_v2 2024-10-21 13:24:50,473 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:79 - Invoking MCL hook IncidentsSummaryHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) 2024-10-21 13:24:50,474 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:79 - Invoking MCL hook IngestionSchedulerHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) 2024-10-21 13:24:50,474 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:79 - Invoking MCL hook EntityChangeEventGeneratorHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) 2024-10-21 13:24:50,474 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:79 - Invoking MCL hook SiblingAssociationHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) 2024-10-21 13:24:50,474 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:97 - Successfully completed MCL hooks for consumer: generic-mae-consumer-job-client urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) 2024-10-21 13:24:50,474 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:69 - Invoking MCL hooks for consumer: generic-mae-consumer-job-client urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD), aspect name: upstreamLineage, entity type: dataset, change type: RESTATE 2024-10-21 13:24:50,474 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:79 - Invoking MCL hook FormAssignmentHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) 2024-10-21 13:24:50,474 [ThreadPoolTaskExecutor-1] INFO  c.l.metadata.kafka.MCLKafkaListener:79 - Invoking MCL hook UpdateIndicesHook for urn: urn:li:dataset:(urn:li:dataPlatform:clickhouse,_dds.dist_5_dds_crm_issues,PROD) 2024-10-21 13:24:50,478 [ThreadPoolTaskExecutor-1] INFO  c.l.m.s.e.update.ESBulkProcessor:82 - Added request id: YxtQxbiPZDnFzc4S31sl0A==, operation type: UPDATE, index: system_metadata_service_v1 2024-10-21 13:24:50,478 [ThreadPoolTaskExecutor-1] INFO  c.l.m.s.e.update.ESBulkProcessor:82 - Added request id: urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Aclickhouse%2C_dds.dist_5_dds_crm_issues%2CPROD%29, operation type: UPDATE, index: datasetindex_v2

repo: datahub-project/datahub


text:
fix(report): too long report causes MSG_SIZE_TOO_LARGE in kafka

While investigating following error:
cimpl.KafkaException: KafkaError{code=MSG_SIZE_TOO_LARGE,val=10,str="Unable to produce message: Broker: Message size too large"}

we found that was produced because of the following messages including call stacks were added to the reports:
              'products_events_behaviour.formsaveerror': ['unable to get column information due to an error -> (pyhive.exc.OperationalError) TExecuteStatementResp(status=TStatus(statusCode=3, infoMessages=["*org.apache.hive.service.cli.HiveSQLException:Error running query: org.apache.spark.sql.AnalysisException: s3a://data-sch-products-pro/events_behaviour/formsaveerror doesn\'t exist:140:139", \'org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$:runningQueryError:HiveThriftServerErrors.scala:47\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute:SparkExecuteStatementOperation.scala:435\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:runInternal:SparkExecuteStatementOperation.scala:209\', \'org.apache.hive.service.cli.operation.Operation:run:Operation.java:383\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkOperation$$super$run:SparkExecuteStatementOperation.scala:52\', \'org.apache.spark.sql.hive.thriftserver.SparkOperation:run:SparkOperation.scala:106\', \'org.apache.spark.sql.hive.thriftserver.SparkOperation:run$:SparkOperation.scala:104\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:run:SparkExecuteStatementOperation.scala:52\', \'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:573\', \'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatement:HiveSessionImpl.java:549\', \'org.apache.hive.service.cli.CLIService:executeStatement:CLIService.java:305\', \'org.apache.hive.service.cli.thrift.OSSTCLIServiceIface:ExecuteStatement:ThriftCLIService.java:1059\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.QueryRejectionThriftHandler:ExecuteStatement:QueryRejectionThriftHandler.scala:45\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SQLGatewaySessionConfThriftHandler:ExecuteStatement:SQLGatewaySessionConfThriftHandler.scala:50\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SQLGatewaySessionStateThriftHandler:ExecuteStatement:SQLGatewaySessionStateThriftHandler.scala:307\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.FetchProtocolThriftHandler:ExecuteStatement:FetchProtocolThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.GetResultSetMetadataThriftHandler:ExecuteStatement:GetResultSetMetadataThriftHandler.scala:26\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.InitialNamespaceThriftHandler:ExecuteStatement:InitialNamespaceThriftHandler.scala:33\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SessionCapabilitiesHandler:ExecuteStatement:SessionCapabilitiesHandler.scala:27\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.AsyncMetadataOperationsThriftHandler:ExecuteStatement:AsyncMetadataOperationsThriftHandler.scala:26\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.GetInfoInOpenSessionThriftHandler:ExecuteStatement:GetInfoInOpenSessionThriftHandler.scala:36\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ResultSetMetadataInFetchResultsThriftHandler:ExecuteStatement:ResultSetMetadataInFetchResultsThriftHandler.scala:26\', \'com.databricks.sql.hive.thriftserver.thrift.DirectResultsThriftHandler:ExecuteStatement:DirectResultsThriftHandler.scala:58\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ProtocolVersionThriftHandler:ExecuteStatement:ProtocolVersionThriftHandler.scala:29\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.FilterJDBCStatementBlockListThriftHandler:ExecuteStatement:FilterJDBCStatementBlockListThriftHandler.scala:54\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:super$ExecuteStatement:LocalPropertiesThriftHandler.scala:90\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:$anonfun$ExecuteStatement$1:LocalPropertiesThriftHandler.scala:90\', \'org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties:withLocalProperties:ThriftLocalProperties.scala:123\', \'org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties:withLocalProperties$:ThriftLocalProperties.scala:48\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:withLocalProperties:LocalPropertiesThriftHandler.scala:30\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:ExecuteStatement:LocalPropertiesThriftHandler.scala:90\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:super$ExecuteStatement:SetNextOperationIdThriftHandler.scala:50\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:$anonfun$ExecuteStatement$1:SetNextOperationIdThriftHandler.scala:50\', \'com.databricks.logging.Log4jUsageLoggingShim$:$anonfun$withAttributionContext$1:Log4jUsageLoggingShim.scala:33\', \'scala.util.DynamicVariable:withValue:DynamicVariable.scala:62\', \'com.databricks.logging.AttributionContext$:withValue:AttributionContext.scala:94\', \'com.databricks.logging.Log4jUsageLoggingShim$:withAttributionContext:Log4jUsageLoggingShim.scala:31\', \'com.databricks.logging.UsageLogging:withAttributionContext:UsageLogging.scala:205\', \'com.databricks.logging.UsageLogging:withAttributionContext$:UsageLogging.scala:204\', \'com.databricks.spark.util.PublicDBLogging:withAttributionContext:DatabricksSparkUsageLogger.scala:22\', \'com.databricks.logging.UsageLogging:withAttributionTags:UsageLogging.scala:240\', \'com.databricks.logging.UsageLogging:withAttributionTags$:UsageLogging.scala:225\', \'com.databricks.spark.util.PublicDBLogging:withAttributionTags:DatabricksSparkUsageLogger.scala:22\', \'com.databricks.spark.util.PublicDBLogging:withAttributionTags0:DatabricksSparkUsageLogger.scala:63\', \'com.databricks.spark.util.DatabricksSparkUsageLogger:withAttributionTags:DatabricksSparkUsageLogger.scala:163\', \'com.databricks.spark.util.UsageLogging:$anonfun$withAttributionTags$1:UsageLogger.scala:493\', \'com.databricks.spark.util.UsageLogging$:withAttributionTags:UsageLogger.scala:605\', \'com.databricks.spark.util.UsageLogging$:withAttributionTags:UsageLogger.scala:614\', \'com.databricks.spark.util.UsageLogging:withAttributionTags:UsageLogger.scala:493\', \'com.databricks.spark.util.UsageLogging:withAttributionTags$:UsageLogger.scala:491\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:withAttributionTags:SetNextOperationIdThriftHandler.scala:37\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:ExecuteStatement:SetNextOperationIdThriftHandler.scala:50\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SetSessionHandleProtocolVersionThriftHandler:ExecuteStatement:SetSessionHandleProtocolVersionThriftHandler.scala:51\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:super$ExecuteStatement:ThreadLocalContextThriftHandler.scala:70\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:$anonfun$ExecuteStatement$1:ThreadLocalContextThriftHandler.scala:70\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:createThriftThreadLocalContext:ThreadLocalContextThriftHandler.scala:45\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:ExecuteStatement:ThreadLocalContextThriftHandler.scala:70\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:super$ExecuteStatement:ErrorPropagationThriftHandler.scala:73\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:$anonfun$ExecuteStatement$1:ErrorPropagationThriftHandler.scala:73\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:runSafely:ErrorPropagationThriftHandler.scala:119\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:ExecuteStatement:ErrorPropagationThriftHandler.scala:73\', \'org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:425\', \'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1437\', \'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1422\', \'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:38\', \'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39\', \'org.apache.thrift.server.TServlet:doPost:TServlet.java:83\', \'org.apache.hive.service.cli.thrift.ThriftHttpServlet:doPost:ThriftHttpServlet.java:95\', \'javax.servlet.http.HttpServlet:service:HttpServlet.java:523\', \'javax.servlet.http.HttpServlet:service:HttpServlet.java:590\', \'org.eclipse.jetty.servlet.ServletHolder:handle:ServletHolder.java:799\', \'org.eclipse.jetty.servlet.ServletHandler:doHandle:ServletHandler.java:550\', \'org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:143\', \'org.eclipse.jetty.security.SecurityHandler:handle:SecurityHandler.java:602\', \'org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:235\', \'org.eclipse.jetty.server.session.SessionHandler:doHandle:SessionHandler.java:1624\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:233\', \'org.eclipse.jetty.server.handler.ContextHandler:doHandle:ContextHandler.java:1434\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:188\', \'org.eclipse.jetty.servlet.ServletHandler:doScope:ServletHandler.java:501\', \'org.eclipse.jetty.server.session.SessionHandler:doScope:SessionHandler.java:1594\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:186\', \'org.eclipse.jetty.server.handler.ContextHandler:doScope:ContextHandler.java:1349\', \'org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:141\', \'org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127\', \'org.eclipse.jetty.server.Server:handle:Server.java:516\', \'org.eclipse.jetty.server.HttpChannel:lambda$handle$1:HttpChannel.java:388\', \'org.eclipse.jetty.server.HttpChannel:dispatch:HttpChannel.java:633\', \'org.eclipse.jetty.server.HttpChannel:handle:HttpChannel.java:380\', \'org.eclipse.jetty.server.HttpConnection:onFillable:HttpConnection.java:277\', \'org.eclipse.jetty.io.AbstractConnection$ReadCallback:succeeded:AbstractConnection.java:311\', \'org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:105\', \'org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint:onFillable:SslConnection.java:555\', \'org.eclipse.jetty.io.ssl.SslConnection:onFillable:SslConnection.java:410\', \'org.eclipse.jetty.io.ssl.SslConnection$2:succeeded:SslConnection.java:164\', \'org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:105\', \'org.eclipse.jetty.io.ChannelEndPoint$1:run:ChannelEndPoint.java:104\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:runTask:EatWhatYouKill.java:338\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:doProduce:EatWhatYouKill.java:315\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:tryProduce:EatWhatYouKill.java:173\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:run:EatWhatYouKill.java:131\', \'org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread:run:ReservedThreadExecutor.java:386\', \'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149\', \'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624\', \'java.lang.Thread:run:Thread.java:750\', "*org.apache.spark.sql.AnalysisException:s3a://data-sch-products-pro/events_behaviour/formsaveerror doesn\'t exist:140:1", \'org.apache.spark.sql.AnalysisException:copy:AnalysisException.scala:71\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute:SparkExecuteStatementOperation.scala:427\'], sqlState=None, errorCode=0, errorMessage="Error running query: org.apache.spark.sql.AnalysisException: s3a://data-sch-products-pro/events_behaviour/formsaveerror doesn\'t exist"), operationHandle=None)\n[SQL: DESCRIBE `products_events_behaviour`.`formsaveerror`]\n(Background on this error at: https://sqlalche.me/e/14/e3q8)', 'Ingestion error: (pyhive.exc.OperationalError) TExecuteStatementResp(status=TStatus(statusCode=3, infoMessages=["*org.apache.hive.service.cli.HiveSQLException:Error running query: org.apache.spark.sql.AnalysisException: s3a://data-sch-products-pro/events_behaviour/formsaveerror doesn\'t exist:140:139", \'org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$:runningQueryError:HiveThriftServerErrors.scala:47\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute:SparkExecuteStatementOperation.scala:435\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:runInternal:SparkExecuteStatementOperation.scala:209\', \'org.apache.hive.service.cli.operation.Operation:run:Operation.java:383\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkOperation$$super$run:SparkExecuteStatementOperation.scala:52\', \'org.apache.spark.sql.hive.thriftserver.SparkOperation:run:SparkOperation.scala:106\', \'org.apache.spark.sql.hive.thriftserver.SparkOperation:run$:SparkOperation.scala:104\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:run:SparkExecuteStatementOperation.scala:52\', \'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:573\', \'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatement:HiveSessionImpl.java:549\', \'org.apache.hive.service.cli.CLIService:executeStatement:CLIService.java:305\', \'org.apache.hive.service.cli.thrift.OSSTCLIServiceIface:ExecuteStatement:ThriftCLIService.java:1059\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.QueryRejectionThriftHandler:ExecuteStatement:QueryRejectionThriftHandler.scala:45\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SQLGatewaySessionConfThriftHandler:ExecuteStatement:SQLGatewaySessionConfThriftHandler.scala:50\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SQLGatewaySessionStateThriftHandler:ExecuteStatement:SQLGatewaySessionStateThriftHandler.scala:307\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.FetchProtocolThriftHandler:ExecuteStatement:FetchProtocolThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.GetResultSetMetadataThriftHandler:ExecuteStatement:GetResultSetMetadataThriftHandler.scala:26\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.InitialNamespaceThriftHandler:ExecuteStatement:InitialNamespaceThriftHandler.scala:33\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SessionCapabilitiesHandler:ExecuteStatement:SessionCapabilitiesHandler.scala:27\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.AsyncMetadataOperationsThriftHandler:ExecuteStatement:AsyncMetadataOperationsThriftHandler.scala:26\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.GetInfoInOpenSessionThriftHandler:ExecuteStatement:GetInfoInOpenSessionThriftHandler.scala:36\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ResultSetMetadataInFetchResultsThriftHandler:ExecuteStatement:ResultSetMetadataInFetchResultsThriftHandler.scala:26\', \'com.databricks.sql.hive.thriftserver.thrift.DirectResultsThriftHandler:ExecuteStatement:DirectResultsThriftHandler.scala:58\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ProtocolVersionThriftHandler:ExecuteStatement:ProtocolVersionThriftHandler.scala:29\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.FilterJDBCStatementBlockListThriftHandler:ExecuteStatement:FilterJDBCStatementBlockListThriftHandler.scala:54\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:super$ExecuteStatement:LocalPropertiesThriftHandler.scala:90\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:$anonfun$ExecuteStatement$1:LocalPropertiesThriftHandler.scala:90\', \'org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties:withLocalProperties:ThriftLocalProperties.scala:123\', \'org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties:withLocalProperties$:ThriftLocalProperties.scala:48\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:withLocalProperties:LocalPropertiesThriftHandler.scala:30\', \'com.databricks.sql.hive.thriftserver.thrift.LocalPropertiesThriftHandler:ExecuteStatement:LocalPropertiesThriftHandler.scala:90\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:super$ExecuteStatement:SetNextOperationIdThriftHandler.scala:50\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:$anonfun$ExecuteStatement$1:SetNextOperationIdThriftHandler.scala:50\', \'com.databricks.logging.Log4jUsageLoggingShim$:$anonfun$withAttributionContext$1:Log4jUsageLoggingShim.scala:33\', \'scala.util.DynamicVariable:withValue:DynamicVariable.scala:62\', \'com.databricks.logging.AttributionContext$:withValue:AttributionContext.scala:94\', \'com.databricks.logging.Log4jUsageLoggingShim$:withAttributionContext:Log4jUsageLoggingShim.scala:31\', \'com.databricks.logging.UsageLogging:withAttributionContext:UsageLogging.scala:205\', \'com.databricks.logging.UsageLogging:withAttributionContext$:UsageLogging.scala:204\', \'com.databricks.spark.util.PublicDBLogging:withAttributionContext:DatabricksSparkUsageLogger.scala:22\', \'com.databricks.logging.UsageLogging:withAttributionTags:UsageLogging.scala:240\', \'com.databricks.logging.UsageLogging:withAttributionTags$:UsageLogging.scala:225\', \'com.databricks.spark.util.PublicDBLogging:withAttributionTags:DatabricksSparkUsageLogger.scala:22\', \'com.databricks.spark.util.PublicDBLogging:withAttributionTags0:DatabricksSparkUsageLogger.scala:63\', \'com.databricks.spark.util.DatabricksSparkUsageLogger:withAttributionTags:DatabricksSparkUsageLogger.scala:163\', \'com.databricks.spark.util.UsageLogging:$anonfun$withAttributionTags$1:UsageLogger.scala:493\', \'com.databricks.spark.util.UsageLogging$:withAttributionTags:UsageLogger.scala:605\', \'com.databricks.spark.util.UsageLogging$:withAttributionTags:UsageLogger.scala:614\', \'com.databricks.spark.util.UsageLogging:withAttributionTags:UsageLogger.scala:493\', \'com.databricks.spark.util.UsageLogging:withAttributionTags$:UsageLogger.scala:491\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:withAttributionTags:SetNextOperationIdThriftHandler.scala:37\', \'com.databricks.sql.hive.thriftserver.thrift.SetNextOperationIdThriftHandler:ExecuteStatement:SetNextOperationIdThriftHandler.scala:50\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.SetSessionHandleProtocolVersionThriftHandler:ExecuteStatement:SetSessionHandleProtocolVersionThriftHandler.scala:51\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:super$ExecuteStatement:ThreadLocalContextThriftHandler.scala:70\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:$anonfun$ExecuteStatement$1:ThreadLocalContextThriftHandler.scala:70\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:createThriftThreadLocalContext:ThreadLocalContextThriftHandler.scala:45\', \'com.databricks.sql.hive.thriftserver.thrift.ThreadLocalContextThriftHandler:ExecuteStatement:ThreadLocalContextThriftHandler.scala:70\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement:DelegatingThriftHandler.scala:63\', \'com.databricks.sql.hive.thriftserver.thrift.DelegatingThriftHandler:ExecuteStatement$:DelegatingThriftHandler.scala:62\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:super$ExecuteStatement:ErrorPropagationThriftHandler.scala:73\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:$anonfun$ExecuteStatement$1:ErrorPropagationThriftHandler.scala:73\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:runSafely:ErrorPropagationThriftHandler.scala:119\', \'com.databricks.sql.hive.thriftserver.thrift.ErrorPropagationThriftHandler:ExecuteStatement:ErrorPropagationThriftHandler.scala:73\', \'org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:425\', \'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1437\', \'org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1422\', \'org.apache.thrift.ProcessFunction:process:ProcessFunction.java:38\', \'org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39\', \'org.apache.thrift.server.TServlet:doPost:TServlet.java:83\', \'org.apache.hive.service.cli.thrift.ThriftHttpServlet:doPost:ThriftHttpServlet.java:95\', \'javax.servlet.http.HttpServlet:service:HttpServlet.java:523\', \'javax.servlet.http.HttpServlet:service:HttpServlet.java:590\', \'org.eclipse.jetty.servlet.ServletHolder:handle:ServletHolder.java:799\', \'org.eclipse.jetty.servlet.ServletHandler:doHandle:ServletHandler.java:550\', \'org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:143\', \'org.eclipse.jetty.security.SecurityHandler:handle:SecurityHandler.java:602\', \'org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:235\', \'org.eclipse.jetty.server.session.SessionHandler:doHandle:SessionHandler.java:1624\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextHandle:ScopedHandler.java:233\', \'org.eclipse.jetty.server.handler.ContextHandler:doHandle:ContextHandler.java:1434\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:188\', \'org.eclipse.jetty.servlet.ServletHandler:doScope:ServletHandler.java:501\', \'org.eclipse.jetty.server.session.SessionHandler:doScope:SessionHandler.java:1594\', \'org.eclipse.jetty.server.handler.ScopedHandler:nextScope:ScopedHandler.java:186\', \'org.eclipse.jetty.server.handler.ContextHandler:doScope:ContextHandler.java:1349\', \'org.eclipse.jetty.server.handler.ScopedHandler:handle:ScopedHandler.java:141\', \'org.eclipse.jetty.server.handler.HandlerWrapper:handle:HandlerWrapper.java:127\', \'org.eclipse.jetty.server.Server:handle:Server.java:516\', \'org.eclipse.jetty.server.HttpChannel:lambda$handle$1:HttpChannel.java:388\', \'org.eclipse.jetty.server.HttpChannel:dispatch:HttpChannel.java:633\', \'org.eclipse.jetty.server.HttpChannel:handle:HttpChannel.java:380\', \'org.eclipse.jetty.server.HttpConnection:onFillable:HttpConnection.java:277\', \'org.eclipse.jetty.io.AbstractConnection$ReadCallback:succeeded:AbstractConnection.java:311\', \'org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:105\', \'org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint:onFillable:SslConnection.java:555\', \'org.eclipse.jetty.io.ssl.SslConnection:onFillable:SslConnection.java:410\', \'org.eclipse.jetty.io.ssl.SslConnection$2:succeeded:SslConnection.java:164\', \'org.eclipse.jetty.io.FillInterest:fillable:FillInterest.java:105\', \'org.eclipse.jetty.io.ChannelEndPoint$1:run:ChannelEndPoint.java:104\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:runTask:EatWhatYouKill.java:338\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:doProduce:EatWhatYouKill.java:315\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:tryProduce:EatWhatYouKill.java:173\', \'org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:run:EatWhatYouKill.java:131\', \'org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread:run:ReservedThreadExecutor.java:386\', \'java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149\', \'java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624\', \'java.lang.Thread:run:Thread.java:750\', "*org.apache.spark.sql.AnalysisException:s3a://data-sch-products-pro/events_behaviour/formsaveerror doesn\'t exist:140:1", \'org.apache.spark.sql.AnalysisException:copy:AnalysisException.scala:71\', \'org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation:org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute:SparkExecuteStatementOperation.scala:427\'], sqlState=None, errorCode=0, errorMessage="Error running query: org.apache.spark.sql.AnalysisException: s3a://data-sch-products-pro/events_behaviour/formsaveerror doesn\'t exist"), operationHandle=None)\n[SQL: DESCRIBE FORMATTED `products_events_behaviour`.`formsaveerror`]\n(Background on this error at: https://sqlalche.me/e/14/e3q8)'],

This is preventing (or reducing the change of) the issue by limiting the reason message when adding a warn or a failure to the report, while keeping the full message in the logger.
Checklist

[ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

This is looking great to me. Awaiting final review from @hsheth2 and team!

CI failure is unrelated.


text:
add self tests for test type custom_sql

issue #4 for custom_sql

ready for review @dividor
I'll be adding test types one after the other so that I create huge PRs that stay often for ever

repo: datakind/Data-Observation-Toolkit


text:
Towards more user-friendly navigation using aliases and ID concepts

This PR is in response to https://github.com/datalad/datalad-catalog/issues/423, the issue contains a detailed walk-through of the concepts being introduced here.
The PR has a quite large diff. Firstly, it includes changes to the following core modules:

webcatalog: new methods are added for extracting all IDs in a catalog, and all VERSIONs for respective IDs, including properties like name, alias, and more. This allows users to iterate through a list comprising all catalog datasets (which is necessary for assigning aliases to datasets, and creating alias and ID metadata files for all datasets in a catalog, and also for supporting the upcoming 'tree' command) and allows the 'report' functionality which summarizes some stats of a catalog.
node: adds a method for sorting metadata_sources by time, as a proxy for determining a last_updated value for a dataset version, which forms part of the output of the new webcatalog methods for reporting.
get: adds the property 'report', allowing a summary report of the catalog to be printed out.
utils: changes supporting above methods.
tests: minor changes to allow test success, new tests are TODO.

Then, the dataset schema was updated to include the alias field, and then name and alias fields were added to the existing metadata of all datasets in the catalog, in preparation for the next step. Then a script was added (and executed) that uses these new python methods to run through all datasets in the demo catalog and then creates associated concept ID and alias metadata files. These files have the format:
{
   "type": "redirect",
   "dataset_id": "...",
   "dataset_version": "..."
}

and they are located at predictable locations based on the ID or ALIAS of the dataset being accessed. This allows the web app to easily navigate using a friendly URL (e..g. https://mycatalog.de/dataset/studyforrest_alias or https://mycatalog.de/dataset/abcd_id), which is then routed internally to the correct ID and VERSION metadata file.
Lastly, web asset changes to support this navigation are included.
There are several aspects still to be developed to make this functionality compatible with the full pipeline of catalog creation and entry generation and maintenance. TODO:

add functionality to always create/update concept and ID metadata files when metadata is added/updated to the catalog
support a list of versions in the concept id metadata file
update script to also include assigning alias fields to metadata from a tsv file.

Codecov Report
Attention: Patch coverage is 23.40426% with 36 lines in your changes are missing coverage. Please review.

Project coverage is 83.62%. Comparing base (44ab1da) to head (ab2fbea).

Files
Patch %
Lines

datalad_catalog/webcatalog.py
8.00%
23 Missing :warning:

datalad_catalog/get.py
25.00%
9 Missing :warning:

datalad_catalog/node.py
33.33%
4 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #430      +/-   ##
==========================================
+ Coverage   82.12%   83.62%   +1.49%
==========================================
  Files          46       45       -1
  Lines        2999     2919      -80
==========================================
- Hits         2463     2441      -22
+ Misses        536      478      -58

Flag
Coverage Δ

unittests
77.06% <23.40%> (-0.99%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

With the current version of the script at https://github.com/datalad/datalad-catalog/blob/ab2fbeabb51696a534daeb0f43c2e969d8e1a1b7/tools/create_alias_concept_metadata.py, you can apply all necessary updates to a catalog's metadata to allow the new functionality provided by this PR.
For example:
python tools/create_alias_concept_metadata.py --catalog ../abcdj/data-catalog/catalog --aliases abcdj-aliases.tsv

where abcdj-aliases.tsv is a tsv file with the following content:
alias	dataset_id	dataset_version
ocr-PIRA-cohort	db7592d0-6206-5684-a29c-8059a6033241	0.1.0
jumax	3f8a45c0-08fc-479c-b561-cb6f744d2b5c	32f3303308c89b037fe1700547348470539a30cb
movies	0036b2c6-f131-4660-9ef9-945087ad02d3	1139b4c82d7ecd0d9ab25bf4c00ec2c06461d6da
ema_pilot	372e66b3-e654-4a0f-ba4c-6394bf314f2f	d468e9a9455b38b959db34bc5335fd66e42b2884
abcdj	1015ed7c-0a3d-4dfc-9c4f-11fe71673a41	c58f0f563011222c618a58951fa21b61c8eb189b

After (or before) these changes have been applied to a catalog, the only other changes necessary are those made to the web assets and schema at:

datalad_catalog/catalog/assets/app_component_dataset.js
datalad_catalog/catalog/assets/app_globals.js
datalad_catalog/catalog/assets/app_router.js
datalad_catalog/catalog/schema/jsonschema_dataset.json

@mslw: ☝ for reference if we want to apply this to SFB1451 catalog

Some more thoughts after revisiting this PR:

It feels to me that it makes more sense to assign an alias to an ID concept (i.e. dataset) only, and not to the ID+VERSION. The idea is that the dataset retains an alias, irrespective of continued version bumps. If this is regarded as the way to go, then:

the alias metadata file should not be populated with a specific version as it is currently.
the ID concept metadata file seems  like the appropriate place to include a list of the multiple versions of a dataset that are contained in a catalog
the alternative (actually ideal) is not to track available versions explicitly in a file, but rather to determine it at runtime by looking at the metadata directories and files in a dataset ID directory, but we only have a client side application and we're not working with a local filesystem
tracking versions in a file would mean that we would also need to update the file both when new versions are added or when existing versions are removed.

Still TODO:

add functionality to always create/update concept and ID metadata files when metadata is added/updated to the catalog

I spot check the live demo and it worked nicely. Thanks!
I agree that aliasing for a dataset (unversioned) makes most sense. Conceptually, however, aliased dataset versions also nake sense (they are tagged versions).
I have no good concept on the structural consequences of this PR. Would two uncoordinated, parallel updates of a dataset still work, or is there the change to overwrite information?
If I understood correctly, there is now a need (or at least the ability) to enumerate catalog items. What would you think the maximum number of catalag record could be, in a real world scenario?

Some more thoughts after revisiting this PR:
It feels to me that it makes more sense to assign an alias to an ID concept (i.e. dataset) only, and not to the ID+VERSION. The idea is that the dataset retains an alias, irrespective of continued version bumps. If this is regarded as the way to go, then:

the alias metadata file should not be populated with a specific version as it is currently.

(...)

I think the "concept" URL would be the most valuable addition (but tagged versions could also be useful, as mih points out). Without looking into the meat of the PR, the file abcdj-aliases.tsv seems to be a reasonable way to get this done. Updates to a dataset (adding metadata for a new version) would require updating the file to keep it pointing at "latest" (unless the version is, kept literally at "latest"...). However, this would not be a complicated operation, and we already follow the same procedure whenever the superdataset version progresses (catalog set-super, which changes super.json). I would be willing to pay the price.
I think figuring out at runtime is not necessary (consider this as potential complication: I don't typically update "older" dataset versions, but in general the metadata update time/order does not have to match version time/order).

@mih

I have no good concept on the structural consequences of this PR. Would two uncoordinated, parallel updates of a dataset still work, or is there the change to overwrite information?

Two uncoordinated, parallel updates of a dataset could still work (I'm assuming "coordinated" would mean that the contributors have to speak/chat to each other in order to make both updates work successfully?) in the way that it currently works, by depending on a standard git-based collaborative workflow.
Also, there is as much chance of overwriting information as there was before this PR. E.g. if two agents have write permissions to the same catalog repo, and they both provide updates to the same property of a dataset, and if both metadata sources have a higher level of priority than the source that populated the existing property value, these two changes could overwrite each other. But this is expected behaviour and source priority is a configuration-level setting per catalog or per dataset.

If I understood correctly, there is now a need (or at least the ability) to enumerate catalog items. What would you think the maximum number of catalag record could be, in a real world scenario?

The ability: yes. The need stems from me wanting to run through all datasets in a catalog to assign aliases. I though this functionality could be useful in future as well for any type of catalog-dataset-level update.
The maximum number of catalog records for a real world scenario is difficult to answer. It would depend on how the user interface (and code underneath it) deals with delays and whether that translates to user frustration. The bottle neck at the moment is the fetching of subdataset-level metadata when navigating to a dataset. So the issue won't be maximum number of datasets for the whole catalog, but if a single dataset has many subdatasets. I think we need to test this to start having a better understanding of the practicalities, and the SFB1451 catalog revamping would be a good first test. But we should stress test it more.
This single-dataset-with-many-subdatastes scenario is kind of built-in in our current usage of a catalog home page, but isn't strictly required. The alternative is to have a more traditional "data portal" where the front page isn't also a (conceptual) dataset, and is rather used to query across all datasets in the catalog. This is something I think we should discuss when taling about the design of the new catalog toolset.

@mslw

Without looking into the meat of the PR, the file abcdj-aliases.tsv seems to be a reasonable way to get this done. Updates to a dataset (adding metadata for a new version) would require updating the file to keep it pointing at "latest" (unless the version is, kept literally at "latest"...). However, this would not be a complicated operation, and we already follow the same procedure whenever the superdataset version progresses (catalog set-super, which changes super.json). I would be willing to pay the price.

Just to be clear, the file abcdj-aliases.tsv is not intended to serve as a continuously maintained and referenced source of aliases for dataset-id-versions of a catalog instance. I suggested its use only when running a script to assign aliases, i.e. once-off usage. Afterwards, normal catalog operation depends on two main aspects:

The alias, which is not referenced from a catalog-central source (such as the 'super.json' file for catalog homepage) but which is found within the so-called "alias metadata file" (which follows the same structure as the concept-id metadata file) and which contains a pointer to the correct dataset ID (which is in turn the concept-id metadata file)
The dataset version that needs to be navigated to. In the current state of the PR, this is specified in the alias metadata file, but I am thinking of making that at least optional (not required as it is currently), and maintaining the correct version in the concept-id metadata file.

I think figuring out at runtime is not necessary (consider this as potential complication: I don't typically update "older" dataset versions, but in general the metadata update time/order does not have to match version time/order).

This is a good point. At the moment the proxy for "latest" is "update time", but like you say these might not be equal. Thinking about the use case for multi-versioned datasets in a single catalog, there are two scenarios:

A dataset is accessed as a specific version, either via a versioned subdataset relation or via direct navigation. Here we do not have to care about what version is considered latest, because the user is interested in this specific version.
A dataset is accessed via the concept-ID or via an (unversioned) alias, meaning no version is specified and should be derived somehow. Currently, from my POV the most sensible way for this to be achieved is to have a kind of "canonical version" in the concept-id file, which could be updated every time a new dataset version is added to the catalog (but does not necessarily have to be). Alternatively, by proxy of update time (or whatever other time-based field we might prefer), the "latest" version could also be implicitly used as the canonical version. In addition storing the canonical version, the concept-id metadata file could also store a list of other versions (that is, as opposed to determining all versions of a dataset at runtime), for example for easy navigation between versions.

In order to spend the least amount of time necessary on this PR, I propose:

fixing the remaining issues with navigation using url query strings (main priority)
NOT (at least, not yet) adding alias-related functionality to the catalog-add command, i.e. not adding code that would add an alias metadata file and concept ID file when a dataset record is added to a catalog
NOT (at least, not yet) adding a separate command that would create an alias metadata file and/or concept ID file for a specific dataset(-version) in a catalog
proposing the already committed script for use when alias- and concept-id metadata files need to be created for dataset-versions in an existing catalog

These items are pending a discussion with @mslw about how the SFB1451 catalog restructuring will take place.

Bringing a note from @mih  into this PR:

Is it true that some kind of alias listing has to be done, and this needs a consolidation in case a new (lastest) version of an existing resource is added to the catalog?

This is somewhat undecided. With the current state of the new functionality in this PR, a catalog instance (and its maintainers) can choose to complete ignore aliases and concept IDs if they want to, and carry on using the id+version-based navigation as supported in the version of SFB1451 that is currently in production. If they choose to support navigation based on aliases and concept ids (i.e. where the url does not contain a specified version), the necessary additional metadata files can be added once-off (using the script added in this PR), i.e. no listing has to be maintained. What has to be maintained though, is a pointer to the "current canonical" version of a dataset from (at least) the concept id metadata file (this can also be pointed to from the alias metadata file, thus supporting versioned tags). The alternative is to not maintain any pointer to the current canonical version and let the client application determine this at runtime from some field in the metadata of all available versions of the dataset being navigated to. E.g. using a last_updated_at field as a proxy for latest, but as @mslw pointed out there isn't necessarily a 1-to-1 relationship between the last_updated_at and canonical version.
Somewhat related, if we want to support a dropdown list of versions available for a given dataset, this would necessitate running through all versions of a current dataset, which would mean that finding the latest one is a small additional step.

UPDATE: Since https://github.com/datalad/datalad-catalog/pull/440 has been merged, this PR needs some resolving first. And then the last bugs related to the query strings on navigation to subdatasets need to be fixed.

Given that:

dataset_version is optional for a alias metadata/redirect file (it will then redirect to the dataset-id metadata/redirect file)
the canonical dataset_version is required for the dataset-id metadata/redirect file
new versions of a dataset, to serve as the canonical versions of that dataset, will likely be added to a catalog

I think the script that creates the redirect files (and takes a tsv with dataset_id | dataset_version | alias as input) should create the alias files without the dataset_version field. This is not currently the case.
@mslw thoughts?

I don't understand the linting errors:
...
would reformat /home/runner/work/datalad-catalog/datalad-catalog/tools/create_alias_concept_metadata.py

Oh no! 💥 💔 💥
1 file would be reformatted, 48 files would be left unchanged.

When I run black locally on that file, no changes are suggested. So I'll ignore these failures, meaning there are no blockers to merge anymore.

Given that:
* `dataset_version` is optional for a alias metadata/redirect file (it will then redirect to the dataset-id metadata/redirect file)

* the canonical `dataset_version` is required for the dataset-id metadata/redirect file

* new versions of a dataset, to serve as the canonical versions of that dataset, will likely be added to a catalog

I think the script that creates the redirect files (and takes a tsv with dataset_id | dataset_version | alias as input) should create the alias files without the dataset_version field. This is not currently the case.
@mslw thoughts?

I don't think there is a strict should - but I agree with your reasoning, and I'd personally be more inclined to create an alias for a dataset ID, and then use the dataset id metadata/redirect file to keep it pointing to "latest".

authors: codecov-commenter jsheunis mih mslw

repo: datalad/datalad-catalog


text:
Revert "Refactoring line prepare function (#346)"

This reverts commit 4c57f5a8e21e25417c6852f2028307506d3747bb.

Merge without full teamcity e2e check (last time commit didn't catch this problem)

repo: datalens-tech/datalens-ui


text:
Using Serilog format specifier for ints makes buffer file invalid

Symptom
I accidentally used an invalid Serilog format specifier in a message template. Although it was my mistake, it kind of had severe consequences.
No logs of this application ever arrive in Seq when using durable log shipping. It works fine when setting bufferBaseFilename to null.
Cause
When using the :l format specifier, and passing an int as property value, the durable buffer file is corrupted. It contains invalid json. This causes the HttpLogShipper to receive 500 response from Seq.
Even after fixing the incorrect format specifier, the logs never arrived in Seq. I needed to delete the buffer file, causing loss of log entries.
Replay
using System;
using Serilog;
using Serilog.Events;

namespace ConsoleApp1
{
    class Program
    {
        static void Main(string[] args)
        {
            var loggerConfiguration = new LoggerConfiguration()
                .MinimumLevel.Verbose()
                .Enrich.WithProperty("Application", "ConsoleApp1", destructureObjects: false)
                .WriteTo.Seq(
                    restrictedToMinimumLevel: LogEventLevel.Debug,
                    serverUrl: "http://localhost:5341",
                    bufferBaseFilename: Environment.ExpandEnvironmentVariables("%ALLUSERSPROFILE%\\ConsoleApp1")
                );

            Log.Logger = loggerConfiguration.CreateLogger();

            Log.Logger.Information("I will be logged with a string argument {0:l}.", "abc");
            Log.Logger.Information("I will corrupt the buffer file {0:l}.", 123); // Notice it's an int, not a string
            Log.Logger.Information("This will never be logged. HttpLogShipper receives 500.");

            Log.CloseAndFlush();
        }
    }
}

Causes the following text in the buffer file (notice that it's invalid json):
{"@t":"2021-10-25T12:22:38.0452847Z","@mt":"I will be logged with a string argument {0:l}.","@r":["abc"],"0":"abc","Application":"ConsoleApp1"}
{"@t":"2021-10-25T12:22:38.1094993Z","@mt":"I will corrupt the buffer file {0:l}.","@r":[{"@t":"2021-10-25T12:22:38.1141918Z","@mt":"This will never be logged. HttpLogShipper receives 500.","Application":"ConsoleApp1"}

And the Seq server-side error:
[12:19:40 ERR] HTTP "POST" "/api/events/raw" responded 500 in 5.7451 ms
System.InvalidCastException: Cannot cast Newtonsoft.Json.Linq.JObject to Newtonsoft.Json.Linq.JToken.
   at Newtonsoft.Json.Linq.Extensions.Convert[T,U](T token)
   at Seq.Server.Storage.StorageEventNormalizer.<>c.<FromClefFormat>b__3_1(PropertyToken p, JToken j)
   at System.Linq.Enumerable.ZipIterator[TFirst,TSecond,TResult](IEnumerable`1 first, IEnumerable`1 second, Func`3 resultSelector)+MoveNext()
   at System.Linq.Lookup`2.Create(IEnumerable`1 source, Func`2 keySelector, IEqualityComparer`1 comparer)
   at System.Linq.GroupedEnumerable`2.GetEnumerator()
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   at Seq.Server.Storage.StorageEventNormalizer.FromClefFormat(Int32 lineNumber, Object evt, InputSettings inputSettings, StorageEventCreationData& creationData, String& error)
   at Seq.Server.Web.Api.IngestionController.TryParseClefBody(Input input, InputSettings inputSettings, StorageEventCreationData[]& events, String& errorMessage)
   at Seq.Server.Web.Api.IngestionController.TryParseBody(Input input, InputSettings inputSettings, StorageEventCreationData[]& events, String& errorMessage)
   at Seq.Server.Web.Api.IngestionController.Ingest()
   at lambda_method532(Closure , Object , Object[] )
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Seq.Server.Web.Middleware.WebSocketAcceptMiddleware.Invoke(HttpContext context)
   at Seq.Server.Web.Middleware.RequestAuthenticationMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Seq.Server.Web.Middleware.ServerStatusMiddleware.Invoke(HttpContext context)
   at Seq.Server.Web.Middleware.BrowserSecurityOptionsMiddleware.Invoke(HttpContext context)
   at Seq.Server.Web.Middleware.RequestCompletionMiddleware.Invoke(HttpContext httpContext)

Expected behaviour

The invalid use of a format specifier should not corrupt the buffer file.
Even when this occurs, I would prefer to continue logging the entries that actually do work, ignoring the invalid entry (maybe warn to the console or something).

Imho, making such a trivial mistake in a message template, should not make all logging to Seq fail (until you delete the buffer file).
Any pointers to where to start looking, would be nice.

Fixed now in 5.0.2-dev-00203 on NuGet; will queue up a stable/main build once we're happy we haven't broken anything :-)

Hi @DavidLievrouw - did you have a chance to check out the fixed build? Thanks!

Yes, it works with 5.0.2-dev-00203. For the above replay-scenario, the output is:

And more importantly, the entries that are logged after the erroneous message, arrive in Seq correctly.
Thank you for your assistance.

Awesome, thanks for the follow-up, David.

authors: DavidLievrouw nblumhardt

repo: datalust/serilog-sinks-seq