text:
can't persist onetomany, manytoone(Bidirectional Association) in Hbase, its also in Unidirectional Association

@Entity
@Table(name = "CLASSROOM", schema = "EDUCATION@hbase")
@IndexCollection(columns = {
    @Index(name = "name")
})
public class ClassroomEntity implements Serializable {
    private static final long serialVersionUID = 8020522006409395867L;

    /**
     * ID
     */
    @Id
    @NotNull
    @GeneratedValue(strategy = GenerationType.TABLE)
    @Column(name = "CLASSROOM_ID")
    private Integer classroomId;

// ignore name definition

    @OneToMany(fetch = FetchType.LAZY, cascade = CascadeType.ALL, mappedBy = "classroom",   targetEntity = StudentEntity.class)
    private Set<StudentEntity> students;

    ...set and get
}

@Entity
@Table(name = "STUDENT", schema = "EDUCATION@hbase")
@IndexCollection(columns = {
    @Index(name = "name")
})
public class StudentEntity implements Serializable
{
    private static final long serialVersionUID = 8020522006409395867L;

    /**
     * ID
     */
    @Id
    @NotNull
    @GeneratedValue(strategy = GenerationType.TABLE)
    @Column(name = "STUDENT_ID")
    private Integer studentId;

//ignore name definition

    /**
     * CLASS ENTITY
     */
    @ManyToOne(fetch = FetchType.LAZY)
    @JoinColumn(name = "CLASSROOM_ID")
    public ClassEntity classroom;

    ...set and get
}

In the controller
    Set<StudentEntity> students = new HashSet<>();

    StudentEntity s1 = new StudentEntity();
    s1.setStudentId(10);
    s1.setName("aaa");
    students.add(s1);

    StudentEntity s2 = new StudentEntity();
    s2 setStudentId(20);
    s2.setName("bbb");
    students.add(s2);

    ClassroomEntity classroom = new ClassroomEntity();
    classroom.setClassroomId(100);
    classroom.setName("ccc");
    classroom.setStudents(students);

    someService.insertClassroom(classroom);

Persistence Unit:
<persistence-unit name="hbase">
        <provider>com.impetus.kundera.KunderaPersistence</provider>
        <properties>
            <property name="kundera.nodes" value="localhost" />
            <property name="kundera.port" value="2181" />
            <property name="kundera.keyspace" value="EDUCATION" />
            <property name="kundera.dialect" value="hbase" />
            <property name="kundera.ddl.auto.prepare" value="create" />
            <property name="kundera.client.lookup.class" value="com.impetus.client.hbase.HBaseClientFactory" />
            <property name="kundera.show.query" value="true" />
            <property name="kundera.batch.size" value="5" />
            <property name="kundera.datastore.file.path" value="META-INF/datastore-hbase.xml" />
        </properties>
    </persistence-unit>

I use hbase shell, can list classroom name and student name, but can't list classroomId, studentId and the onetomany & manytoone association data

HI @jackyang1984,
What are HBase and Kundera version you using?

Please have a look at existing association test case - https://github.com/Impetus/Kundera/tree/trunk/src/kundera-hbase/kundera-hbase-v2/src/test/java/com/impetus/client/hbase/crud/association

@devender-yadav
hadoop: 2.7.6
hbase: 2.0.0
kundera: 3.12
yes, i try the crud test, but didn't work

@jackyang1984 can you please error logs.

@devender-yadav have not error, two entities can persist data to hbase, but the association between them can not

@jacksonic I will try to replicate it today in our system with particular versions.
Are you using kundera-hbase-v2?

@devender-yadav yes

@devender-yadav i saw many example about kundera with cassandra, more than kundera with hbase, is kundera the best friend with cassandra?

@devender-yadav below is my maven pom

4.0.0
<groupId>com.abc</groupId>
<artifactId>cs</artifactId>
<version>v1</version>
<packaging>jar</packaging>

<name>cs</name>

<parent>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-parent</artifactId>
	<version>2.0.2.RELEASE</version>
	<relativePath/>
</parent>

<properties>
	<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
	<java.version>1.8</java.version>
    <hadoop.version>2.7.6</hadoop.version>
	<hbase.version>1.2.6</hbase.version>
	<kundera.version>3.12</kundera.version>
</properties>

<dependencies>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-web</artifactId>
		<exclusions>
			<exclusion>
				<artifactId>spring-boot-starter-tomcat</artifactId>
				<groupId>org.springframework.boot</groupId>
			</exclusion>
		</exclusions>
	</dependency>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-jetty</artifactId>
	</dependency>
	<dependency>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-test</artifactId>
		<scope>test</scope>
	</dependency>
    <dependency>
        <groupId>org.springframework.data</groupId>
        <artifactId>spring-data-jpa</artifactId>
    </dependency>

    <dependency>
        <groupId>com.impetus.kundera.client</groupId>
        <artifactId>kundera-hbase-v2</artifactId>
        <version>${kundera.version}</version>
    </dependency>
    <!--<dependency>-->
        <!--<groupId>com.impetus.kundera.core</groupId>-->
        <!--<artifactId>fallback-impl</artifactId>-->
        <!--<version>${kundera.version}</version>-->
        <!--<scope>test</scope>-->
    <!--</dependency>-->
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-client</artifactId>
        <version>${hbase.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-server</artifactId>
        <version>${hbase.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-server</artifactId>
        <version>${hbase.version}</version>
        <classifier>tests</classifier>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-common</artifactId>
        <version>${hbase.version}</version>
        <classifier>tests</classifier>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-common</artifactId>
        <version>${hbase.version}</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-hadoop-compat</artifactId>
        <version>${hbase.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>${hadoop.version}</version>
    </dependency>
</dependencies>

@jackyang1984,
Kundera will Cassandra is more popular. But kundera-hbase and kundera-mongodb are also active and commonly used in Kundera community.

@jackyang1984,
There is no JoinTable annotation in your entities. Thus, Join table will not be created in your particular example.
This is working fine at my end.

@devender-yadav
https://github.com/Impetus/Kundera/wiki/Polyglot-Persistence  the example have not also, so maybe you can add it

@jackyang1984 Join table is generally used for Many to Many associations. Is there any reason that you need separate join table for One to Many association.
What issue are you facing with example - https://github.com/Impetus/Kundera/wiki/Polyglot-Persistence

@devender-yadav yes, i do as the Polyglot-Persistence, but didn't work, but i need use onetomany and manytoone, so kundera is not support both of them?

@jackyang1984 Kundera does support both of them. That's why we have added the wiki for polyglot persistence - https://github.com/Impetus/Kundera/wiki/Polyglot-Persistence
What exact issue are you facing with polyglot persistence? Please share error logs and expected results.

@devender-yadav
@Entity
@Table(name = "CLASSROOM", schema = "EDUCATION@hbase")
public class ClassroomEntity {
@Id
@NotNull
@GeneratedValue(strategy = GenerationType.TABLE)
@Column(name = "CLASSROOM_ID")
private Integer classroomId;
@Column(name = "CLASSROOM_NAME")
private String classroomName;

@OneToMany(fetch = FetchType.LAZY, cascade = CascadeType.ALL)
@JoinColumn(name = "CLASSROOM_STUDENT")
private Set<StudentEntity> students;

}
@Entity
@Table(name = "STUDENT", schema = "EDUCATION@hbase")
public class StudentEntity {
@Id
@NotNull
@GeneratedValue(strategy = GenerationType.TABLE)
@Column(name = "STUDENT_ID")
private Integer studentId;
@Column(name = "STUDENT_NAME")
private String studentName;

}
// Controller
ClassroomEntity classroom = new ClassroomEntity();
classroom.setClassroomName("classroom1");
    Set<StudentEntity> students = new HashSet<>();
    StudentEntity s1 = new StudentEntity();
    s1.setStudentName("test1");

    StudentEntity s2 = new StudentEntity();
    s2.setStudentName("test2");

    students.add(s1);
    students.add(s2);

    classroom.setStudents(students);

    service.insertClassroom(classroom);

My expected is save classroom data, student data and onetomany data to hbase,  but......,   in fact, onetomany data don't save to hbase, as the same time havn't error log to display, it's really havn't error log!!!!!!

Hi @jackyang1984,
I tried to replicate this at my end. Its working fine.
Entities:
@Entity
@Table(name = "CLASSROOM", schema = "kundera_db@hbase_pu")
public class ClassroomEntity
{
    @Id
    @NotNull
    @GeneratedValue(strategy = GenerationType.TABLE)
    @Column(name = "CLASSROOM_ID")
    private Integer classroomId;

    @Column(name = "CLASSROOM_NAME")
    private String classroomName;

    @OneToMany(fetch = FetchType.LAZY, cascade = CascadeType.ALL)
    @JoinColumn(name = "CLASSROOM_STUDENT")
    private Set<StudentEntity> students;

    // getter setters
}

@Entity
@Table(name = "STUDENT", schema = "kundera_db@hbase_pu")
public class StudentEntity
{
    @Id
    @NotNull
    @GeneratedValue(strategy = GenerationType.TABLE)
    @Column(name = "STUDENT_ID")
    private Integer studentId;

    @Column(name = "STUDENT_NAME")
    private String studentName;

    // getters and setters
}

Test Class:
    EntityManagerFactory emf = Persistence.createEntityManagerFactory("hbase_pu");
    EntityManager em = emf.createEntityManager();

    ClassroomEntity classroom = new ClassroomEntity();
    classroom.setClassroomId(501);
    classroom.setClassroomName("class1");

    Set<StudentEntity> students = new HashSet<>();

    StudentEntity s1 = new StudentEntity();
    s1.setStudentId(101);
    s1.setStudentName("student1");
    students.add(s1);

    StudentEntity s2 = new StudentEntity();
    s2.setStudentId(102);
    s2.setStudentName("student2");
    students.add(s2);

    classroom.setStudents(students);

    em.persist(classroom);

Persistence Unit
<persistence-unit name="hbase_pu">
	<provider>com.impetus.kundera.KunderaPersistence</provider>
	<class>com.dev.test.StudentEntity</class>
	<class>om.dev.test.ClassroomEntity</class>
	<exclude-unlisted-classes>true</exclude-unlisted-classes>
	<properties>
		<property name="kundera.nodes" value="localhost" />
		<property name="kundera.port" value="60000" />
		<property name="kundera.keyspace" value="kundera_db" />
		<property name="kundera.dialect" value="hbase" />
		<property name="kundera.client.lookup.class" value="com.impetus.client.hbase.HBaseClientFactory" />
		<property name="kundera.ddl.auto.prepare" value="create" />
	</properties>
</persistence-unit>

Data in HBase:

scan "kundera_db:CLASSROOM"

ROW                                                  COLUMN+CELL
 \x00\x00\x01\xF5                                    column=CLASSROOM:CLASSROOM_NAME, timestamp=1526367106622, value=class1
1 row(s) in 0.1210 seconds

scan "kundera_db:STUDENT"

ROW                                                  COLUMN+CELL
 \x00\x00\x00e                                       column=STUDENT:CLASSROOM_STUDENT, timestamp=1526367106633, value=\x00\x00\x01\xF5
 \x00\x00\x00e                                       column=STUDENT:STUDENT_NAME, timestamp=1526367106633, value=student1
 \x00\x00\x00f                                       column=STUDENT:CLASSROOM_STUDENT, timestamp=1526367106636, value=\x00\x00\x01\xF5
 \x00\x00\x00f                                       column=STUDENT:STUDENT_NAME, timestamp=1526367106636, value=student2
2 row(s) in 0.1150 seconds

@devender-yadav I find my codes bug where is
https://github.com/Impetus/Kundera/blob/trunk/examples/container/jboss/data-keeper/src/main/java/com/impetus/kundera/datakeeper/dao/DataKeeperDaoImpl.java
@Override
public void insert(Object entity)
{
em.persist(entity);
em.clear();
}
if comment em.clear(), it's running

@jackyang1984 but it should work in both cases i.e. with and without clearing persistence cache.

@devender-yadav
Unidirectional onetomany now is ok, but now Bidirectional onttomany is not, all datas and Bidirectional relation data is stored, but  i can't get the many side data
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.StackOverflowError
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1006) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) ~[spring-webmvc-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) ~[jetty-servlet-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) ~[jetty-servlet-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:215) ~[websocket-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) ~[jetty-servlet-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) ~[jetty-servlet-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) ~[jetty-servlet-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) ~[jetty-servlet-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.0.6.RELEASE.jar!/:5.0.6.RELEASE]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) ~[jetty-servlet-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) ~[jetty-servlet-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[jetty-security-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) ~[jetty-servlet-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.Server.handle(Server.java:531) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) ~[jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [jetty-server-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) [jetty-io-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) [jetty-io-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [jetty-io-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:760) [jetty-util-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:678) [jetty-util-9.4.10.v20180503.jar!/:9.4.10.v20180503]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_171]
Caused by: java.lang.StackOverflowError: null
at java.io.StringWriter.write(StringWriter.java:112) ~[na:1.8.0_171]
at com.google.gson.stream.JsonWriter.string(JsonWriter.java:591) ~[gson-2.8.4.jar!/:na]
at com.google.gson.stream.JsonWriter.writeDeferredName(JsonWriter.java:402) ~[gson-2.8.4.jar!/:na]

@jackyang1984,
Please share your latest Bidirectional OneToMany relational entities.

@devender-yadav have kundera-hbase-v2 support left join operation? if support, how to write the query?

@jackyang1984 you can't write left join query on kundera-hbase-v2.

@devender-yadav ok.  Two tables is ManyToOne Relationship, i just select one table by another table field, do you know how to do that?

repo: Impetus/Kundera


text:
Not recognizing vext types

In some cases it doesn’t know NetEvents, WebUI etc.

And it doesn't know about json.encode. Not sure if it’s the same issue

source: gharchive/issue

repo: Imposter/vscode-lua-vu

license: MIT

license_type: permissive

license_source: github-api


text:
Test TimelinePage: should how a tooltip on hover fails

The mentioned test sometimes passes and sometimes fails with the following message:
● TimelinePage › should show a tooltip on hover

    expect(element).not.toBeInTheDocument()

    expected document not to contain element, found

Deborah Knoll V

born in Lavamünd on March  1, 1918

died in Graz on February 21, 1936

lived in Untertilliach on June 29, 1927

stayed in Pitschgau on February  2, 1925

stayed in Neukirchen an der Enknach on October 23, 1932

statue erected in Arriach on January 28, 1950

Modi quo sapiente sunt repudiandae dolor maiores dolor. Non possimus provident perferendis tenetur dolores iure quia nemo neque. Enim neque aut ut voluptates ea molestias est. Sequi et in voluptatibus. Dolorem culpa minus tempora eaque. Eius quia dolorem deserunt voluptatem distinctio totam dolore occaecati debitis.

 instead

    Ignored nodes: comments,

repo: InTaVia/web


text:
Get rid of insane overloads on SimChat

Inside Scene.PacketHandlers.cs, there are an insane number of confusing overloads on the SimChat function. These have already caused a regression and are prone to cause more coding errors especially if the method signatures need to change again in the future.
I propose that only the first SimChat method that takes all of the parameters be left in place and any parameters that now completed as part of the overloads be left as optional parameters with sane defaults.

Full path to the file is OpenSim\Region\Framework\Scenes

No takers for this? Definitely a more simple way to get your feet wet in the code

Pull request #191

These changes were included as part of the 0.9.23 release. ;)

authors: MatthewBeardmore ddaeschler jimtarber

repo: InWorldz/halcyon

license_source: bigquery


text:
class for wrapping the functions, arrange attendance df , warning df

attendance df - excel df with the addition of columns for each session. In each session, the column contains the zoom name of the person who wrote the excel name

list of warning df - list of data frames where each one represents all the chat data of zoom users that wrote something in the chat but it was not inputted in the excel, which means it might be a typo error, this will be represented in the website for consideration of the teacher

code review comment addressed and viewed

source: gharchive/pull-request


text:
feat: migrate emulator to device_preview

Changes

migrated emulator to use device_preview

UI

Unfortunately, this plugin is reproducing the same issue. One way to know is to check the nav-bar.
With device_preview:

With a stand-alone iOS simulator:

You can tell that the icons and spacing in the navbar on the device_simulator and device_preview are a little bit distorted and not properly aligned. While on the individual simulator, the nav-bar looks normal.

authors: HelenGezahegn eivenlour

repo: IndustryBear/MentorApp


text:
can`t use query with protobuf-java under 3.22

Specifications

Install  protobuf-java 3.21.10
call query
Error:

Exception in thread "main" java.lang.IllegalAccessError: class org.apache.arrow.flight.impl.Flight$FlightDescriptor tried to access method 'com.google.protobuf.LazyStringArrayList com.google.protobuf.LazyStringArrayList.emptyList()' (org.apache.arrow.flight.impl.Flight$FlightDescriptor and com.google.protobuf.LazyStringArrayList are in unnamed module of loader 'app')
	at org.apache.arrow.flight.impl.Flight$FlightDescriptor.<init>(Flight.java:7034)
	at org.apache.arrow.flight.impl.Flight$FlightDescriptor.<clinit>(Flight.java:7738)
	at org.apache.arrow.flight.impl.FlightServiceGrpc.getGetFlightInfoMethod(FlightServiceGrpc.java:106)

Code sample to reproduce problem
  val manage = InfluxDbManager(host, token, database)
  val result = manage.query("SELECT * FROM  \"tes\" WHERE  time >= now() - interval '300 hour'", (data: Seq[AnyRef]) => data.head)
  println(result)

Expected behavior
run the query
Actual behavior
exception
Additional info
related to:
https://github.com/quarkusio/quarkus/issues/31240
line in code:

Hi @noam-stein,
Thank you for using our client.
The protobuf-java dependency is utilized by flight-grpc. If you wish to use version 3.21, you'll need to use this version of flight-grpc.
You can exclude protobuf-java and flight-grpc from influxdb3-java and add these dependencies separately in your pom.xml or Gradle configuration.
Regards

Thanks for the quick response.
unfortunately, I can't do it.
my project uses Bazel and some projects need flight-grpc latest version.
I can't really change that :-(

Which version of the flight-grpc  do you use?

13.0.0

Why you want to install protobuf-java 3.21.10? The flight-grpc:13.0.0 depends on version 3.23.1.

The project is a big monorepo.
I don't have control over the version of protobuf-java. I can't update protobuf-java in the whole project because there are a lot of breaking changes.

So, currently, you are using flight-grpc:13.0.0 along with protobuf-java:3.21.10, and you also want to use influxdb3-java:0.3.0, correct?

thanks for the help,
I was able to update the dependencies

authors: bednar noam-stein


text:
Skrivefeil Egenskap: tilhører modul (modelldcatnobelongsToModule)

I beskrivelsen av egenskapen står det: Brukes til å referere til en modellmodul/delmodell som modellen inngår i.
Det skal stå: Brukes til å referere til en modellmodul/delmodell som egenskapen inngår i.

Fiksa

Fiksa

authors: ToHelland jimjyang


text:
Markdownfix

After the features/smhi_root is merged, this little fix to correct the name of SMHIs parent Ministry could be considered.
Closes #15

Did you base this branch on main or some other branch? Given the many commits unrelated to the name change.

Sorry, based on the other branch, that's why I figured we'd fix that first.
Guess I can fix that, but can't we just sort that out first?
On Fri, 3 Feb 2023, 00:10 Mladen Gibanica, @.***> wrote:

Did you base this branch on main or some other branch? Given the many
commits unrelated to the name change.
—
Reply to this email directly, view it on GitHub
https://github.com/Ingenjorsarbete-For-Klimatet/ifk-smhi/pull/48#issuecomment-1414494897,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AE7PLB7IM7X5XMUEVT723VTWVQ5GBANCNFSM6AAAAAAUPOOXFA
.
You are receiving this because you authored the thread.Message ID:
@.***>

authors: docNord mgcth

repo: Ingenjorsarbete-For-Klimatet/ifk-smhi


text:
Won't start

Hi,
It won't start.
❯ radiotray-ng-mpris
Traceback (most recent call last):
  File "/usr/bin/radiotray-ng-mpris", line 33, in <module>
    sys.exit(load_entry_point('radiotray-ng-mpris==0.1.1', 'console_scripts', 'radiotray-ng-mpris')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/radiotray-ng-mpris", line 25, in importlib_load_entry_point
    return next(matches).load()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.11/site-packages/radiotray_ng_mpris/cli.py", line 18, in <module>
    from .wrap import setup_signal_handling, wrap_radiotray_ng
  File "/usr/lib/python3.11/site-packages/radiotray_ng_mpris/wrap.py", line 14, in <module>
    from mpris_server import Metadata, MetadataObj
  File "/usr/lib/python3.11/site-packages/mpris_server/__init__.py", line 5, in <module>
    from . import adapters, base, interfaces, mpris, server, types
  File "/usr/lib/python3.11/site-packages/mpris_server/adapters.py", line 6, in <module>
    from .base import ActivePlaylist, DEFAULT_DESKTOP, DEFAULT_ORDERINGS, DEFAULT_PLAYLIST_COUNT, DEFAULT_RATE, DbusObj, \
  File "/usr/lib/python3.11/site-packages/mpris_server/base.py", line 27
    type Properties = Collection[Property]
         ^^^^^^^^^^
SyntaxError: invalid syntax

Hi, thanks for your report. Do you use Manjaro Linux? On Manjaro, the dependency python-mpris_server is broken, because it requires Python 3.12, but Manjaro still uses Python 3.11.
What you can do:

Wait until Python is upgraded (should happen in the next weeks)
Downgrade mpris_server to the previous AUR version 0.4.2.


text:
🛑 Amiya IN is down

In 8c54d67, Amiya IN ($URL) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Amiya IN is back up in c74cc2c after 10 minutes.


text:
Blog, Polars, premier jet pour discussion

@linogaliana un premier jet assez brut comme support à la discussion. J'itère ensuite en fonction de tes retours.

@romaintailhurat avec 6acce27693baf7b73aaf0242d27f6c2cad95ee1e je t'ai créé un profil sur le site, que tu peux enrichir si tu le désires

authors: linogaliana romaintailhurat

repo: InseeFrLab/ssphub


text:
macOS Python GetNameOfClass test illegal instruction

Fix the PythonGetNameOfClass test on macOS. Currently fails with:
WARNING: In /Users/vsts/agent/2.141.1/work/1/s/Modules/Core/Common/src/itkProcessObject.cxx, line 903
BlockMatchingImageFilter (0x7f9a94af0170): Input already "FeaturePoints" already required!

WARNING: In /Users/vsts/agent/2.141.1/work/1/s/Modules/Segmentation/LevelSets/include/itkNarrowBandLevelSetImageFilter.h, line 350
NarrowBandLevelSetImageFilter (0x7f9a949d8500): The current implmentation of this solver does not compute maximum RMS change. The maximum RMS error value will not be set or used.

WARNING: In /Users/vsts/agent/2.141.1/work/1/s/Modules/Segmentation/LevelSets/include/itkNarrowBandLevelSetImageFilter.h, line 350
NarrowBandLevelSetImageFilter (0x7f9a94f762f0): The current implmentation of this solver does not compute maximum RMS change. The maximum RMS error value will not be set or used.

itkTestDriver: Process exception: Illegal instruction

Duplicate of #96

repo: InsightSoftwareConsortium/ITK


text:
ENH: wrap StreamingImageFilter for all the dimensions

Needed for fixing the wrapping in https://github.com/InsightSoftwareConsortium/ITKBoneEnhancement/pull/20.

I was just looking at that documentation, because my local build was failing too.

This error:
/Users/Kitware/Dashboards/ITK/ITKPythonPackage/scripts/../ITK-2.7-macosx_x86_64/Wrapping/Typedefs/itkEigenToMeasureImageFilter.i:82: Warning 401: Nothing known about base class 'itk::StreamingImageFilter< itk::Image< itk::Vector< float,3 >,3 >,itk::Image< itk::Vector< float,3 >,3 > >'. Ignored.
/Users/Kitware/Dashboards/ITK/ITKPythonPackage/scripts/../ITK-2.7-macosx_x86_64/Wrapping/Typedefs/itkEigenToMeasureImageFilter.i:82: Warning 401: Maybe you forgot to instantiate 'itk::StreamingImageFilter< itk::Image< itk::Vector< float,3 >,3 >,itk::Image< itk::Vector< float,3 >,3 > >' using %template.

Indicates
itk_wrap_image_filter("${WRAP_ITK_SCALAR}" 2)

should be:
itk_wrap_image_filter("${WRAP_ITK_SCALAR};${WRAP_ITK_VECTOR_REAL}" 2)

I was getting a warning that StreamingImageFilter is unknown to wrapping.

@dzenanz please see the comment above.

authors: dzenanz thewtex


text:
Paraview glance + ITK.js

Hello,
I was trying to have images saved to a temporary directory in Paraview Glance, which is based on vue.js
when I call the function readImageLocalFile or writeImageLocalFile I get the error Error: Cannot find module '/ImageIOs/itkMetaImageIOJSBinding'
Is there a way I can solve this?

Hello, thanks for your answer.
I'm trying to develop some changes for glance. Part of my workflow consists of saving/loading datasets to temporary directories on the filesystem. For this reason I wrote this
import { mapState } from 'vuex';
import writeImageLocalFile from 'itk/writeImageLocalFile';
import readImageLocalFile from 'itk/readImageLocalFile';

export default {
  name: 'TomaatWidgetVolume',
  components: {
    Datasets,
  },
  computed: mapState({
      proxyManager: 'proxyManager',
  }),
  methods: {
    updateList: function () {
      this.datasets = this.proxyManager.getSources();
    },
    chooseSource: function (index) {
      var currentDataset = this.datasets[index]
      console.log(currentDataset.getDataset())
      // not the real thing. just trying out similar to example online
      // real thing would be to write the dataset to disk (not read and write back)
      readImageLocalFile('/tmp/volume1.mha')
        .then((image) => {
          const useCompression = true
          writeImageLocalFile(useCompression, image, '/tmp/volume2.mha')
        })
        .catch((error) => {
          console.error('Error during conversion:\n')
          console.error(error)
        })
    },
  },
  data() {
    return {
      datasets: [],
    };
  },
};

Hello @floryst,
thanks for your help (also for glance, as I saw that you have answered me). I'm an absolute beginner in JS even though I know ITK/VTK for python and C++.
The final goal for me is to create a client for TOMAAT (tomaatcloud.github.io) to be able to analyze images with DL approaches (and beyond) from Paraview Glance from Chrome web browser (and similar).
what I am trying to do is to use ITK-js in Paraview Glance. The idea would be to take Glance datasets (VTK) and transform them into ITK images and save them. After they are saved I want to have a plain/text representation of the MHA file which I would like to include in a POST request. POST request to be made multipart with axios.
All in all it should be similar to this https://github.com/faustomilletari/TOMAAT-Slicer/blob/master/TOMAAT/TOMAAT.py#L366 (where I load the MHA file as plain text) and this https://github.com/faustomilletari/TOMAAT-Slicer/blob/master/TOMAAT/TOMAAT.py#L516 (where I send the POST request to the server).
This project is open source and has no commercial purposes whatsoever. It's a personal project. I have already implemented most of it but I truly have no idea about how I can get a string representing the content of my MHA data.
I tried to save a dummy ITK image like this:
    var mock_image = {
      imageType : {
        dimension: 2,
        componentType: IntTypes.UInt8,
        pixelType: PixelTypes.Scalar,
        components: 1,
      },
      origin: [0, 0],
      spacing: [1, 1],
      direction: [1, 0, 0, 1],
      size: [2, 2],
      data: [0, 1, 2, 3],
    }

    return writeImageArrayBuffer(null, false, mock_image, 'porcodio.png')
    .then(function (writtenArrayBuffer) {
      console.log(writtenArrayBuffer)
    })

and I keep getting errors such as
index.js:116 Uncaught (in promise) TypeError: Failed to execute 'postMessage' on 'Worker': Value at index 0 is an untransferable 'undefined' value.
    at eval (index.js:116)
    at new Promise (<anonymous>)
    at Worker.postMessage (index.js:113)
    at eval (writeImageArrayBuffer.js:15)

as I said I'm not very experienced with JS and this kind of stuff so I'm struggling quite a bit. If you have any suggestion I would really appreciate it.
Regards,
Fausto

In substance the only thing I would need is to be able to select datasets from Paraview glance and obtain a mha compressed file out of them in form of a string (which I need to pass into a http post).

I think I am getting closer to the solution. I have written a function to convert the VTK data into ITK and then the hope is to be able to write a ArrayBuffer and interpret it as a string to include into post request. You can find this code here
function convertVtkToItkImage(vtkImage) {
  // ItkImage --
  //imageType: The ImageType for this image.
  //origin: An Array with length dimension that describes the location of the center of the lower left pixel in physical units.
  //spacing: An Array with length dimension that describes the spacing between pixel in physical units.
  //-direction: A dimension by dimension Matrix that describes the orientation of the image at its origin. The orientation of each axis are the orthonormal columns.
  //-size: An Array with length dimension that contains the number of pixels along dimension.
  //-data: A TypedArray containing the pixel buffer data.

  // ImageType --
  // dimension: An integer that describes the dimension for the image, typically 2 or 3.
  // pixelType: The PixelType. For example, Scalar, Vector, CovariantVector, or SymmetricSecondRankTensor.
  // componentType: The type of the components in a pixel. This is one of the IntTypes or FloatTypes.
  // components: The number of components in a pixel. For a Scalar pixelType, this will be 1.

  var itkImg = {}

  var scalars = vtkImage.getPointData().getScalars()
  var direction = vtkImage.getDirection()  // are these deep copies?
  var direction_ = vtkImage.getDirection()

  // transposition
  for (let i = 0; i < 3; i++) {
    for (let j = 0; j < 3; j++) {
      direction[i + j * 3] = direction[j + i * 3]
    }
  }

  itkImg['origin'] = vtkImage.getOrigin()
  itkImg['spacing'] = vtkImage.getSpacing()
  itkImg['data'] = scalars.getData()
  itkImg['size'] = vtkImage.getDimensions()
  itkImg['direction'] = direction

  var itkType = {
    dimension: 3, // vtk image is always 3
    pixelType: 1, // 1 == scalar for now...
    componentType: IntTypes.UInt8, // for now...
    components: scalars.getNumberOfComponents(),
  }

  itkImg['imageType'] = itkType

  writeImageArrayBuffer(null, true, itkImg, 'hello.mha')
  .then(function (writtenArrayBuffer) {
    console.log(writtenArrayBuffer)
  })
}

unfortunately this code errors (surely because of my ignorance of JS) and I would really appreciate some help. the error is here:
"TypeError: Cannot read property 'NaN' of undefined
    at getMatrixElement (http://10.110.21.58:9999/itk/WebWorkers/ImageIO.worker.js:39:64)
    at writeImageEmscriptenFSFile (http://10.110.21.58:9999/itk/WebWorkers/ImageIO.worker.js:63:818)
    at writeImage (http://10.110.21.58:9999/itk/WebWorkers/ImageIO.worker.js:30:3370)
    at http://10.110.21.58:9999/itk/WebWorkers/ImageIO.worker.js:30:4216
    at i (http://10.110.21.58:9999/itk/WebWorkers/ImageIO.worker.js:2:2049)
    at o (http://10.110.21.58:9999/itk/WebWorkers/ImageIO.worker.js:2:1874)
    at http://10.110.21.58:9999/itk/WebWorkers/ImageIO.worker.js:2:2423"

Regards,

Hello, I think I have been able to write a function to transform VTK images back to ITK images and save them to ArrayBuffer.
function convertVtkToItkImage(vtkImage) {
  var scalars = vtkImage.getPointData().getScalars()
  var direction = vtkImage.getDirection()

  let type = new ImageType(3, IntTypes.UInt8, 1, scalars.getNumberOfComponents())

  let image = new Image(type)

  // transposition
  for (let i = 0; i < 3; i++) {
    for (let j = 0; j < 3; j++) {
      image.direction.data[i + j * 3] = direction[j + i * 3]
    }
  }

  image.origin = vtkImage.getOrigin()
  image.spacing = vtkImage.getSpacing()
  image.data = scalars.getData()
  image.size = vtkImage.getDimensions()

  var writtenArrayBuffer = writeImageArrayBuffer(null, true, image, 'image.mha')
}

now the missing piece is to be able to get a string out of what I have just written. Do you have some ideas about how to do it?
Regards,

As this thread is closely related with Paraview, the discussion will follow here: https://github.com/Kitware/paraview-glance/issues/180#issuecomment-408853685

repo: InsightSoftwareConsortium/itk-js


text:
Make Section Controllers Generic on ObjectType

Changes in this pull request

IGListSectionController is now generic on ObjectType.
IGListBindingSectionController is now generic also on ViewModelType.

Issue fixed: #622
@jessesquires I can't open the iOS examples project on my machine. Xcode 8.3 complains that the project is unreadable. So I believe I updated the example projects to be compatible but I'm not 100% sure.

@Adlai-Holler -- yep, known issue. (fixed in #631) the project file got messed up from a bad merge

@Adlai-Holler - just landed fixes. mind rebasing?

Well, this thing is dead-on-arrival. As of Swift 3, you can't use a MYSectionController<String> in place of an IGListSectionController<IGListDiffable>, even if the generic is specified as covariant which is absolutely stupid.
Anyway, that's the case for now. Too bad so sad.

@Adlai-Holler take a look at this, been kicking it around for a while, I think this could be really great!

authors: Adlai-Holler jessesquires rnystrom

repo: Instagram/IGListKit

license: mit


text:
Get "Users In Photo" information

Hi,
Not an issue but a feature request: any plans to add the "Users in Photo" information when getting media details from calls such as "recent media"?
Thanks.

Thank you for reporting this issue and appreciate your patience. We've  notified the core team for an update on this issue. We're looking for a response within the next 30 days or the issue may be closed.

repo: Instagram/python-instagram


text:
Minor issues - first click-through 02/22

Landing page: set to optimal ratio (height to width)
Fix log box during simulation - header color ("#6BAED6"), background color ("#C6DBEF") and text color (black/dark grey)
Fix looping bug: Configuration -> Target HRH utilization box
Fix population source: Configuration-> TotalPop, this should change with selection of region
Fix plot bug: Compare results -> By service category - legend missing colors for some service categories
Fix plot bug: Compare results -> Change over time - why is there "-" on the legend?
Fix plot bug: Compare results -> Change over time - use geom_label_repel to fix the labels
Remove "Next" on Run Simulation step, and remove the "view results" page.
Add "Print to pdf" button to "Compare results from previous runs" page.

Moving plot display bugs to a new issue. Other bugs are fixed.

repo: InstituteforDiseaseModeling/PACE-HRH-UI


text:
Reproduce Linux ext4 Fuzzer Results

Hello,
Awesome project, many thanks for maintaining it.
I'm attempting to reproduce the Linux ext4 fuzzer as described in the kAFL USENIX paper. I'm using the -vm_dir/-vm_ram method to start an Ubuntu 16.10 VM, as follows:
python3 kAFL-Fuzzer/kafl_fuzz.py \
  -vm_dir snapshot \
  -vm_ram snapshot/ram.qcow2 \
  -mem 512 \
  -seed_dir fs-seed \
  -p 8 \
  --purge \
  -abort_time 32 \
  -work_dir /dev/shm/kafl-fs-fuzz \
  -catch_resets \
  -i 2048-65535 \  # As the paper mentions, limit the deterministic mutations to the first two KB (and last 4B) of the image
  -ip0 0xffffffff812bc000-0xffffffff81357af8

Some more details on this configuration:

My machine has a Xeon 4214 (48 cores @ 2.20GHz) and 128GB DDR4 RAM.
I'm using debootstrap to create the Ubuntu image. The kernel I'm using is a precompiled 4.8.0-22-generic.
I'm using dd/mkfs.ext4/tune2fs to generate a few 64KB seed images, then I'm appending 4 bytes to the end for the flags to mount. In practice I usually need to pass at least 4 seeds to stop the fuzzer from getting stuck on the import phase.
To get the -ip0 range I'm looking through /proc/kallsyms for ext4 functions. My first attempt used the entire kernel core range, but that seemed to slow down the fuzzer too much.
I modified fs-fuzzer.c to add HYPERCALL_KAFL_LOCK and a handshake at the beginning so I wouldn't need to use the loader binary.
I made a couple of other modifications to kAFL itself, namely changing the machine type from q35 to pc to get loadvm working, and the parallel -vm_ram fix mentioned here: https://github.com/IntelLabs/kAFL/issues/15#issuecomment-683359851

So far my fuzzer appears to run in a stable manner, but the performance I'm seeing makes me suspicious that something isn't configured right:

Execs/s hovers around 400-600 after the first hour or so
Number of paths gets to about 1000 after the full 32 hours
No timeouts or panics yet

Any idea if there's an issue in my configuration that could be hampering performance? Any guidance is greatly appreciated.

Hi there,
Congrats on getting this to work! Perhaps you can elaborate on the changes you made / issues you encountered? I've seen a couple reports that PT feedback and/or VM loading did not work properly for this use case and it would be great to have a working tutorial again.
Regarding the "lost seeds", I usually see this in connection with noisy coverage output due to non-deterministic target or buggy PT decode. The -funky option may help with this as it spends some more time to validate new findings.
I don't recall what performance we can expect for ext4 fuzzing. If the performance of your corpus is degrading fast, it may help to set a more aggressive timeout initially so slow findings are discarded:
https://github.com/IntelLabs/kAFL/blob/8c45d915c5f8fd858377ad23e9e1528e0b64f252/kAFL-Fuzzer/common/qemu.py#L561
There is currently no logic to adapt the timeout dynamically, so you would have to restart fuzzing after some time with higher timeout. Check kafl_plot.py to see the execution time of individual inputs (perf=).
Some stages are also a lot slower than others and you may get 'stuck' in these slow stages when facing a target with many or very long paths. You can run your corpus just using the havoc stage to see what speed you may get in the best case (add -D, and don't use -redqueen -grimoire -radamsa). Grimoire is probably not very useful for ext4. I would also disable Radamsa initially as it does a lot of repetition+length extension which can produce slow inputs. If you then see the fuzzer stuck for a long time in deterministic stages or even redqueen, consider disabling deterministic (-D) or limiting redqueen/deterministic to run only on favorite inputs (in state_logic.py handle_redqueen(), return if len(metadata["fav_bits"] ==0 or <=1).

Hi there,
Congrats on getting this to work! Perhaps you can elaborate on the changes you made / issues you encountered? I've seen a couple reports that PT feedback and/or VM loading did not work properly for this use case and it would be great to have a working tutorial again.
Regarding the "lost seeds", I usually see this in connection with noisy coverage output due to non-deterministic target or buggy PT decode. The -funky option may help with this as it spends some more time to validate new findings.
I don't recall what performance we can expect for ext4 fuzzing. If the performance of your corpus is degrading fast, it may help to set a more aggressive timeout initially so slow findings are discarded:
https://github.com/IntelLabs/kAFL/blob/8c45d915c5f8fd858377ad23e9e1528e0b64f252/kAFL-Fuzzer/common/qemu.py#L561
There is currently no logic to adapt the timeout dynamically, so you would have to restart fuzzing after some time with higher timeout. Check kafl_plot.py to see the execution time of individual inputs (perf=).
Some stages are also a lot slower than others and you may get 'stuck' in these slow stages when facing a target with many or very long paths. You can run your corpus just using the havoc stage to see what speed you may get in the best case (add -D, and don't use -redqueen -grimoire -radamsa). Grimoire is probably not very useful for ext4. I would also disable Radamsa initially as it does a lot of repetition+length extension which can produce slow inputs. If you then see the fuzzer stuck for a long time in deterministic stages or even redqueen, consider disabling deterministic (-D) or limiting redqueen/deterministic to run only on favorite inputs (in state_logic.py handle_redqueen(), return if len(metadata["fav_bits"] ==0 or <=1).

Thanks for the reply. So far disabling deterministic mode is improving perf considerably (up to about 2000 execs/s). Funky mode also does help decrease the frequency of "lost seeds", so I'll use that as well. I still haven't found any panics; my next steps are to look at different kernel configurations (make ext4 loadable, enable KASAN).
As for my VM loading setup, here's a more detailed write-up:

Edit targets/linux_x86_64/src/fuzzer/fs_fuzzer.c and add the following to the start of main:

/* Take a VM snapshot */
hprintf("Taking snapshot\n");
kAFL_hypercall(HYPERCALL_KAFL_LOCK, 0);
hprintf("Done\n");

/* Fuzzer handshake */
kAFL_hypercall(HYPERCALL_KAFL_ACQUIRE, 0);
kAFL_hypercall(HYPERCALL_KAFL_RELEASE, 0);

Change the following in kAFL-Fuzzer/common/qemu.py:
a.  Change the default QEMU machine type from q35 to pc (I couldn't get q35 to play nicely with savevm/loadvm)
b. Add console=ttyS0 to the bootparam to get Linux serial working
c. Change the -hdb option from self.config.argument_values['vm_ram'] to self.config.argument_values['vm_dir'] + "/ram_" + self.qemu_id + ".qcow2". This allows us to load a VM snapshot on multiple threads.

Create a basic init script for the guest:

$ cat init
mount -t proc none /proc
mount -t sysfs none /sys

echo 0 > /proc/sys/kernel/randomize_va_space

exec /bin/bash

Download or compile Linux v4.8.0.

Use debootstrap to create an Ubuntu 16.10 install (could also install from the ISO if you want):

mkdir snapshot && cd snapshot
sudo debootstrap --variant=minbase yakkety yakkety
cp ../targets/linux_x86_64/bin/fuzzer/ext4 yakkety/bin/fuzzer
cp ../targets/linux_x86_64/bin/info/info yakkety/bin/info
cp ../init yakkety/init

mke2fs -L '' -N 0 -d yakkety -m 5 -r 1 -t ext4 "yakkety.ext4" 1G
qemu-img convert -f raw -O qcow2 yakkety.ext4 yakkety.qcow2

Setup initial overlays:

qemu-img create -f qcow2 ram.qcow2 512
qemu-img create -b $(realpath yakkety.qcow2) -f qcow2 overlay_0.qcow2
qemu-img create -b $(realpath ram.qcow2) -f qcow2 ram_0.qcow2

Start the VM:

../qemu-5.0.0/x86_64-softmmu/qemu-system-x86_64 \
        -enable-kvm \
        -nographic \
        -net none \
        -machine pc \
        -kernel /path/to/vmlinuz-4.8.0-22-generic \
        -hda overlay_0.qcow2 \
  -hdb ram_0.qcow2 \
        -m 512 \
        -append "root=/dev/sda rw console=ttyS0 nokaslr oops=panic nopti mitigations=off init=/init"

In the VM, run the fuzzer binary to take a snapshot (and kill QEMU once this is complete):

root@(none):/# fuzzer
Taking snapshot
Done

In the snapshot dir, copy overlay_0.qcow2 and ram_0.qcow2 for each thread (overlay_1.qcow2, overlay_2.qcow2 ... overlayN.qcow2).

See my above comments on creating a seed and finding ip0 ranges for ext4.

Should be ready to launch kafl_fuzz.py.

Thanks for the reply. So far disabling deterministic mode is improving perf considerably (up to about 2000 execs/s). Funky mode also does help decrease the frequency of "lost seeds", so I'll use that as well. I still haven't found any panics; my next steps are to look at different kernel configurations (make ext4 loadable, enable KASAN).
As for my VM loading setup, here's a more detailed write-up:

Edit targets/linux_x86_64/src/fuzzer/fs_fuzzer.c and add the following to the start of main:

/* Take a VM snapshot */
hprintf("Taking snapshot\n");
kAFL_hypercall(HYPERCALL_KAFL_LOCK, 0);
hprintf("Done\n");

/* Fuzzer handshake */
kAFL_hypercall(HYPERCALL_KAFL_ACQUIRE, 0);
kAFL_hypercall(HYPERCALL_KAFL_RELEASE, 0);

Change the following in kAFL-Fuzzer/common/qemu.py:
a.  Change the default QEMU machine type from q35 to pc (I couldn't get q35 to play nicely with savevm/loadvm)
b. Add console=ttyS0 to the bootparam to get Linux serial working
c. Change the -hdb option from self.config.argument_values['vm_ram'] to self.config.argument_values['vm_dir'] + "/ram_" + self.qemu_id + ".qcow2". This allows us to load a VM snapshot on multiple threads.

Create a basic init script for the guest:

$ cat init
mount -t proc none /proc
mount -t sysfs none /sys

echo 0 > /proc/sys/kernel/randomize_va_space

exec /bin/bash

Download or compile Linux v4.8.0.

Use debootstrap to create an Ubuntu 16.10 install (could also install from the ISO if you want):

mkdir snapshot && cd snapshot
sudo debootstrap --variant=minbase yakkety yakkety
cp ../targets/linux_x86_64/bin/fuzzer/ext4 yakkety/bin/fuzzer
cp ../targets/linux_x86_64/bin/info/info yakkety/bin/info
cp ../init yakkety/init

mke2fs -L '' -N 0 -d yakkety -m 5 -r 1 -t ext4 "yakkety.ext4" 1G
qemu-img convert -f raw -O qcow2 yakkety.ext4 yakkety.qcow2

Setup initial overlays:

qemu-img create -f qcow2 ram.qcow2 512
qemu-img create -b $(realpath yakkety.qcow2) -f qcow2 overlay_0.qcow2
qemu-img create -b $(realpath ram.qcow2) -f qcow2 ram_0.qcow2

Start the VM:

../qemu-5.0.0/x86_64-softmmu/qemu-system-x86_64 \
        -enable-kvm \
        -nographic \
        -net none \
        -machine pc \
        -kernel /path/to/vmlinuz-4.8.0-22-generic \
        -hda overlay_0.qcow2 \
  -hdb ram_0.qcow2 \
        -m 512 \
        -append "root=/dev/sda rw console=ttyS0 nokaslr oops=panic nopti mitigations=off init=/init"

In the VM, run the fuzzer binary to take a snapshot (and kill QEMU once this is complete):

root@(none):/# fuzzer
Taking snapshot
Done

In the snapshot dir, copy overlay_0.qcow2 and ram_0.qcow2 for each thread (overlay_1.qcow2, overlay_2.qcow2 ... overlayN.qcow2).

See my above comments on creating a seed and finding ip0 ranges for ext4.

Should be ready to launch kafl_fuzz.py.

authors: JohnPMerrill il-steffen

repo: IntelLabs/kAFL


text:
Auto exposure when the camera is in inter cam sync mode 259 / 260

Required Info

Camera Model
D455

Firmware Version
5.16.0.1

Operating System & Version
Linux (Ubuntu 20)

Kernel Version (Linux Only)
5.15.49

Platform
PC

SDK Version
2.55.1

Language
C++

Segment
Robot

Issue Description
Hello, I am interested in using the Realsense cameras in a gen locked mode and taking infrared frames with emitter On and emitter Off, but I am having trouble at having an auto exposure. It seems that when the camera is working in inter cam sync mode 260, the exposure is fixed even if the auto exposure option is enabled.
After seeing this issue: https://github.com/IntelRealSense/librealsense/issues/11908#issuecomment-2013161828
I would like to confirm if it is possible that auto exposure does not work when inter cam sync mode is set to 260 the same way it does not work when the option emitter on off is enabled (both gain level and exposure time do not seem to change when using SDK 2.55.1 and FW 5.16.0.1).
I have also noticed differences in emitter on off and auto exposure behaviours between different SDK and FW versions. In these two videos, I am trying to compare one camera using realsense viewer with SDK version 2.50.0 and camera with FW version 5.13.0.50 with emitter on off enabled and it is possible to see in the metadata that although the exposure time does not vary, the gain does vary (this would also be suitable for me).
https://github.com/IntelRealSense/librealsense/assets/62122095/04eac1da-f87d-4013-b045-6edf6a2844fd
However, when using SDK version 2.55.1 and a camera with FW 5.16.0.1 (emitter on off enabled) the metadata shows that both the exposure time or the gain do not vary. What is the reason of this?
https://github.com/IntelRealSense/librealsense/assets/62122095/4ba536a7-72c1-468a-ae4e-30d17e0b55d4
Thank you very much for your help in advance!

Hi @jeferal  The main change to exposure and gain between SDK versions 2.50.0 and 2.55.1 is that in 2.55.1 and firmware 5.16.0.1, the Auto-Exposure Limit and Auto-Gain Limit options were re-introduced for all 400 Series camera models except D415 and D457.  These options - originally introduced in SDK 2.42.0 - are described below.
RS2_OPTION_AUTO_EXPOSURE_LIMIT
Limit maximal exposure used in Auto-Exposure to preserve FPS.
RS2_OPTION_AUTO_GAIN_LIMIT
Limit maximal digital gain value used in Auto-Exposure.
Regarding auto-exposure not working in Inter Cam Sync Mode 259 and 260: I would think that it would almost certainly have the same auto-exposure characteristic as https://github.com/IntelRealSense/librealsense/issues/11908#issuecomment-2013161828 because the IR emitter is being turned on and off by these modes.
However, it is not something that can be tested for confirmation as Intel no longer provide technical support for the genlock hardware sync modes, only for the original Inter Cam Sync Modes 1 and 2 (though the genlock modes continue to be usable in the SDK).

Hi @MartyG-RealSense, thank you very much for your quick reply and your detailed explanation, I really appreciate it. I am also aware of the new release of FW and SDK, which supports these limit parameters and I understand that the modes 259 and 260 are not supported.
However, how would you explain that with the new FW 5.16.0.1 and SDK 2.55.1 when having the stereo sensor in emitter on off mode and auto exposure, the gain does not vary. But when using the FW 5.13.0.50 and SDK 2.50.0 the gain varies when there are changes in the intensity of the environment?

There have been so many changes to the SDK since 2.50.0 that it would be very difficult to identify precisely what could have caused an apparent change in exposure / gain behaviour when using genlock hardware sync, especially as the effect of changes on genlock modes would likely not be tested for since genlock is no longer supported.  I do apologize.

Thank you so much. I completely understand. Those differences that I pointed in the videos were happening when the camera is not in gen locked mode. It is a configuration of a single camera, working with the sensor option emitter_on_off set to true. This is the scenario where I notice the difference in how autoexposure works among different firmware/sdk versions. I apologize for not being very clear.

I checked the firmware release notes.  Firmware 5.15.0.2 (which corresponds to SDK 2.54.1) added an enhanced auto-exposure algorithm for D455.

Thank you @MartyG-RealSense, I will bear that in mind. One more question, is it possible that you could confirm me that auto exposure does not work in the latest FW version 5.16.0.1 and SDK version 2.55.1 when a single camera (D455) is working in emitter_on_off mode? When I mean that it does not work, I mean that both the exposure time and the gain are fixed.
Thank you very much for your help again.

I ran tests to observe the Actual Exposure metadata in RealSense Viewer 2.55.1 with a D455 and firmware 5.15.0.2 and then 5.16.0.1.  Actual Exposure updated when auto-exposure was enabled and emitter on_off was disabled.
Then I tested with auto-exposure enabled and emitter_on_off enabled.  With both firmware 5.15.0.2 and 5.16.0.1 the Actual Exposure did not update when emitter on_off was enabled and gain did not change.
THen I tested it in older Viewer versions with on-off enabled.  In 2.54.2 and 2.54.1 the Actual Exposure and Gain did not change.
Finally, I tested with Viewer 2.50.0 and firmware 5.13.0.50 (as newer firmwares than that do not work with 2.50.0).  Actual Exposure and Gain were able to change.  This replicates what you experienced.
However, because 5.13.0.50 is the newest firmware that works with 2.50.0, it cannot be confirmed whether they are able to change because of SDK 2.50.0 or firmware 5.13.0.50.

I really appreciate your effort in conducting these tests, the information is very valuable.
Therefore, what is the expected behaviour of auto exposure when it works together with emitter on_off?
Just to understand if the fact that both exposure time and gain level do not change is a bug or if it is supposed to work this way.

Thank you very much @MartyG-RealSense. That makes sense. The application I am working with is quite extreme in the sense that I require a IR picture with emitter on and a IR picture with emitter off as well as being able to very different light conditions (that's why I was interested in the auto exposure). Is there any way I could achieve this with the D455 camera?

If you are using hardware sync then mode 259 / 260 is the only way to control emitter on-off over multiple cameras, otherwise each camera will be individually controlling its on-off unsychronized with the other cameras.
The High-Dynamic Range (HDR) feature enables the camera to handle very different lighting conditions by enabling two different exposure values to be used so that the camera can cope with scenes that have both light and dark areas.  When using HDR though, auto-exposure is disabled and emitter on-off cannot be used.
https://dev.intelrealsense.com/docs/high-dynamic-range-with-stereoscopic-depth-cameras

Hi @jeferal  Do you require further assistance with this case, please?  Thanks!

Case closed due to no further comments received.

authors: MartyG-RealSense jeferal

repo: IntelRealSense/librealsense


text:
D455 Depth- 3D coordinates (x,y,z) differs from python and realsense viewer

Greetings!

Required Info

Camera Model
2 multicameras:  { D455 }

Firmware Version
(Open RealSense Viewer --> latest

Operating System & Version
{Win (11)

Platform
PC

SDK Version
latest

Language
python

Issue Description
I'm using 2 D455 cameras that are 120 degrees apart but tilted to focus on the same region. I require the 3D points from each camera and then I transform them to a center coordinate system using a transformation matrix(transformation is irrelevant to the problem).
When I try to use the Depth 3D coordinates from the intel real sense viewer, I get 3D points from each camera which are completely different from the 3D points I get from the python, when mouse-clicking on the same object while streaming. Below is the python code:
'''
import pyrealsense2 as rs
import numpy as np
import cv2
import json
Load transformation matrices for each camera from provided text files
def load_transformation_matrix(file_path: str) -> np.ndarray:
return np.loadtxt(file_path).reshape(4, 4)
transformation_matrix_cam0 = load_transformation_matrix("d455_camera0_transformation.txt")
transformation_matrix_cam1 = load_transformation_matrix("d455_camera1_transformation.txt")
Initialize both RealSense pipelines
pipeline_0 = rs.pipeline()
pipeline_1 = rs.pipeline()
config_0 = rs.config()
config_1 = rs.config()
config_0.enable_device('234222303235')  # Serial number for Camera 0
config_1.enable_device('241122302541')  # Serial number for Camera 1
Increase resolution and enable both color and depth streams
config_0.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config_0.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
config_1.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config_1.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
Start pipelines
profile_0 = pipeline_0.start(config_0)
profile_1 = pipeline_1.start(config_1)
Function to apply JSON settings to sensors
def apply_settings_from_json(device, json_file):
with open(json_file, 'r') as f:
settings = json.load(f)
for sensor in device.sensors:
    for option_name, option_value in settings.items():
        try:
            option = getattr(rs.option, option_name)
            if sensor.supports(option):
                sensor.set_option(option, option_value)
                print(f"Set {option_name} to {option_value} on {sensor.get_info(rs.camera_info.name)}")
        except AttributeError:
            print(f"Option {option_name} not found.")

Apply JSON settings to both devices
apply_settings_from_json(profile_0.get_device(), "cameraright.json")
apply_settings_from_json(profile_1.get_device(), "cameraright.json")
Depth scaling
depth_scale_0 = profile_0.get_device().first_depth_sensor().get_depth_scale() * 1000  # Convert to mm
depth_scale_1 = profile_1.get_device().first_depth_sensor().get_depth_scale() * 1000  # Convert to mm
Intrinsics and rectification
intrinsics_0 = profile_0.get_stream(rs.stream.color).as_video_stream_profile().get_intrinsics()
intrinsics_1 = profile_1.get_stream(rs.stream.color).as_video_stream_profile().get_intrinsics()
map1_0, map2_0 = cv2.initUndistortRectifyMap(
np.array([[intrinsics_0.fx, 0, intrinsics_0.ppx],
[0, intrinsics_0.fy, intrinsics_0.ppy],
[0, 0, 1]]),
np.array(intrinsics_0.coeffs), None,
None, (intrinsics_0.width, intrinsics_0.height), cv2.CV_16SC2)
map1_1, map2_1 = cv2.initUndistortRectifyMap(
np.array([[intrinsics_1.fx, 0, intrinsics_1.ppx],
[0, intrinsics_1.fy, intrinsics_1.ppy],
[0, 0, 1]]),
np.array(intrinsics_1.coeffs), None,
None, (intrinsics_1.width, intrinsics_1.height), cv2.CV_16SC2)
Transformation function
def apply_transformation(point: np.ndarray, transformation_matrix: np.ndarray) -> np.ndarray:
point_homogeneous = np.append(point, 1)
transformed_point = transformation_matrix @ point_homogeneous
return transformed_point[:3] / transformed_point[3]
Mouse callback functions
def mouse_callback_0(event, x, y, flags, param):
if event == cv2.EVENT_LBUTTONDOWN:
depth_value_mm = depth_image_0[y, x] * depth_scale_0
point_3d_m = rs.rs2_deproject_pixel_to_point(intrinsics_0, [x, y], depth_value_mm / 1000)
point_3d_mm = np.array(point_3d_m) * 1000
transformed_point_mm = apply_transformation(point_3d_mm, transformation_matrix_cam0)
print(f"Camera 0 - Original Point (mm): {point_3d_mm}, Transformed Point (mm): {transformed_point_mm}")
def mouse_callback_1(event, x, y, flags, param):
if event == cv2.EVENT_LBUTTONDOWN:
depth_value_mm = depth_image_1[y, x] * depth_scale_1
point_3d_m = rs.rs2_deproject_pixel_to_point(intrinsics_1, [x, y], depth_value_mm / 1000)
point_3d_mm = np.array(point_3d_m) * 1000
transformed_point_mm = apply_transformation(point_3d_mm, transformation_matrix_cam1)
print(f"Camera 1 - Original Point (mm): {point_3d_mm}, Transformed Point (mm): {transformed_point_mm}")
Set up windows and callbacks
cv2.namedWindow('Camera 0 Stream')
cv2.namedWindow('Camera 1 Stream')
cv2.setMouseCallback('Camera 0 Stream', mouse_callback_0)
cv2.setMouseCallback('Camera 1 Stream', mouse_callback_1)
try:
while True:
frames_0 = pipeline_0.wait_for_frames()
frames_1 = pipeline_1.wait_for_frames()
    depth_frame_0 = frames_0.get_depth_frame()
    color_frame_0 = frames_0.get_color_frame()
    depth_frame_1 = frames_1.get_depth_frame()
    color_frame_1 = frames_1.get_color_frame()

    if not depth_frame_0 or not color_frame_0 or not depth_frame_1 or not color_frame_1:
        continue

    # Convert to numpy arrays
    depth_image_0 = np.asanyarray(depth_frame_0.get_data())
    color_image_0 = np.asanyarray(color_frame_0.get_data())
    depth_image_1 = np.asanyarray(depth_frame_1.get_data())
    color_image_1 = np.asanyarray(color_frame_1.get_data())

    # Apply undistortion
    color_image_0 = cv2.remap(color_image_0, map1_0, map2_0, cv2.INTER_LINEAR)
    color_image_1 = cv2.remap(color_image_1, map1_1, map2_1, cv2.INTER_LINEAR)

    # Display color images
    cv2.imshow('Camera 0 Stream', color_image_0)
    cv2.imshow('Camera 1 Stream', color_image_1)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

finally:
pipeline_0.stop()
pipeline_1.stop()
cv2.destroyAllWindows()
'''
I want to make sure I get the same coordinates in python and real sense viewer. I manually measured, and the real sense viewer seems to provide the correct 3D measurements from each of the cameras.
I experimented with the parameters in the realsense viewer, do these parameters remain with the same value when accessing the cameras through python?
Please help me find the correct python parameters to ensure the depth information I have from the python code is correct and accurate. Let me know if you need further information, thank you.

Hi @Prefiro-Prathik  The RealSense Viewer tool applies a range of post-processing filters to the depth data by default, whilst a Python script applies no filters by default as they have to be deliberately programmed into the script.  So you could left-click on the blue icon next to 'Post-Processing' in the Stereo Module section of the Viewer's side-panel to turn the icon to red (all filters disabled).  You could then check the 3D values to see if they are the same of different, and therefore confirm whether the filters have an effect on the correctness of the values.
Was the configuration json that you are using exported from the RealSense Viewer?  If it was then it is worth bearing in mind that not all of the Viewer's settings are exported to a json, so there may be some settings where the Python program applies the default values instead of values that you customized in the Viewer.
I note that in your Python script you set the streams to 640x480 resolution and 30 FPS.  Are you using the same resolution and FPS settings in the Viewer, please?
In your definitions of depth_scale_0 and depth_scale_1, I would not recommend multiplying the depth scale (0.001) by 1000.  Instead, wait for the final distance value in meters to be calculated and then divide the result by 1000 to get the mm distance value (which you seem to be doing in your depth_value_m lines).

Hi @Prefiro-Prathik  Do you require further assistance with this case, please?  Thanks!

Case closed due to no further comments received.

authors: MartyG-RealSense Prefiro-Prathik


text:
[Bug] internlm qlora

Traceback (most recent call last):
File "/icooper/tools/miniconda3/envs/internLM/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 649, in iflatmap_unordered
yield queue.get(timeout=0.05)
File "", line 2, in get
File "/icooper/tools/miniconda3/envs/internLM/lib/python3.10/site-packages/multiprocess/managers.py", line 818, in _callmethod
kind, result = conn.recv()
File "/icooper/tools/miniconda3/envs/internLM/lib/python3.10/site-packages/multiprocess/connection.py", line 253, in recv
buf = self._recv_bytes()
File "/icooper/tools/miniconda3/envs/internLM/lib/python3.10/site-packages/multiprocess/connection.py", line 417, in _recv_bytes
buf = self._recv(4)
File "/icooper/tools/miniconda3/envs/internLM/lib/python3.10/site-packages/multiprocess/connection.py", line 382, in _recv
chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
multiprocess.context.TimeoutError
Exception ignored in tp_clear of: <class 'cell'>
SystemError: Negative size passed to PyUnicode_New
Exception ignored in: <built-in method _destroy of weakref.ReferenceType object at 0x7fb11ff47b00>
SystemError: <built-in method _destroy of weakref.ReferenceType object at 0x7fb11ff47b00> returned a result with an exception set

@rourouZ
Please provide more details about your dataset, configs and commands

命令
CUDA_VISIBLE_DEVICES=1 xtuner train {configPath}
数据集测试
xtuner check-custom-dataset 测试通过
config
# Copyright (c) OpenMMLab. All rights reserved.
import torch
from glob import glob
from pathlib import Path
from datasets import load_dataset

from mmengine.dataset import DefaultSampler
from mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,
                            LoggerHook, ParamSchedulerHook)
from mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR

from peft import LoraConfig
from torch.optim import AdamW
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          BitsAndBytesConfig)

from xtuner.dataset import ConcatDataset, process_hf_dataset
from xtuner.dataset.collate_fns import default_collate_fn
from xtuner.dataset.map_fns import (crime_kg_assitant_map_fn,
                                    law_reference_map_fn,
                                    emergency_reference_map_fn,
                                    template_map_fn_factory)
from xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,
                                 VarlenAttnArgsToMessageHubHook)
from xtuner.engine.runner import TrainLoop
from xtuner.model import SupervisedFinetune
from xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE

#######################################################################
#                          PART 1  Settings                           #
#######################################################################
# Model
pretrained_model_name_or_path = '/weights/Shanghai_AI_Laboratory/internlm2-chat-7b'
use_varlen_attn = False

# Data
data_path = "/dataset/internlm_train/test"
prompt_template = PROMPT_TEMPLATE.internlm2_chat
max_length = 2048
pack_to_max_length = True

# Scheduler & Optimizer
batch_size = 1  # per_device
accumulative_counts = 16
dataloader_num_workers = 0
max_epochs = 3
optim_type = AdamW
lr = 2e-4
betas = (0.9, 0.999)
weight_decay = 0
max_norm = 1  # grad clip
warmup_ratio = 0.03

# Save
save_steps = 10
save_total_limit = 2  # Maximum checkpoints to keep (-1 means unlimited)

# Evaluate the generation performance during the training
evaluation_freq = 100
# SYSTEM = SYSTEM_TEMPLATE.emergency
SYSTEM = ''

evaluation_inputs = ['你叫什么名字？', '乙醇的沸点是多少？']

#######################################################################
#                      PART 2  Model & Tokenizer                      #
#######################################################################
tokenizer = dict(
    type=AutoTokenizer.from_pretrained,
    pretrained_model_name_or_path=pretrained_model_name_or_path,
    trust_remote_code=True,
    padding_side='right')

model = dict(
    type=SupervisedFinetune,
    # use_varlen_attn=use_varlen_attn,
    llm=dict(
        type=AutoModelForCausalLM.from_pretrained,
        pretrained_model_name_or_path=pretrained_model_name_or_path,
        trust_remote_code=True,
        torch_dtype=torch.float16,
        quantization_config=dict(
            type=BitsAndBytesConfig,
            load_in_4bit=False,
            load_in_8bit=False,
            llm_int8_threshold=6.0,
            llm_int8_has_fp16_weight=False,
            bnb_4bit_compute_dtype=torch.float16,
            bnb_4bit_use_double_quant=True,
            bnb_4bit_quant_type='nf4')),
    lora=dict(
        type=LoraConfig,
        r=64,
        lora_alpha=16,
        lora_dropout=0.1,
        bias='none',
        task_type='CAUSAL_LM'))

#######################################################################
#                      PART 3  Dataset & Dataloader                   #
#######################################################################
all_file_list = glob(pathname=Path(data_path).joinpath("*.json").__str__())

data_assitant = dict(
    type=process_hf_dataset,
    dataset=dict(type=load_dataset, path=data_path),
    # dataset=dict(
    #     type=load_dataset,
    #     path='json',
    #     data_files=all_file_list),
    tokenizer=tokenizer,
    max_length=max_length,
    dataset_map_fn=None,
    template_map_fn=dict(
        type=template_map_fn_factory, template=prompt_template),
    remove_unused_columns=True,
    shuffle_before_pack=True,
    pack_to_max_length=pack_to_max_length,
    use_varlen_attn=use_varlen_attn)

# train_dataset = dict(
#     type=ConcatDataset, datasets=[data_assitant])

train_dataloader = dict(
    batch_size=batch_size,
    num_workers=dataloader_num_workers,
    dataset=data_assitant,
    sampler=dict(type=DefaultSampler, shuffle=True),
    collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))

#######################################################################
#                    PART 4  Scheduler & Optimizer                    #
#######################################################################
# optimizer
optim_wrapper = dict(
    type=AmpOptimWrapper,
    optimizer=dict(
        type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),
    clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),
    accumulative_counts=accumulative_counts,
    loss_scale='dynamic',
    dtype='float16')

# learning policy
# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md  # noqa: E501
param_scheduler = [
    dict(
        type=LinearLR,
        start_factor=1e-5,
        by_epoch=True,
        begin=0,
        end=warmup_ratio * max_epochs,
        convert_to_iter_based=True),
    dict(
        type=CosineAnnealingLR,
        eta_min=0.0,
        by_epoch=True,
        begin=warmup_ratio * max_epochs,
        end=max_epochs,
        convert_to_iter_based=True)
]

# train, val, test setting
train_cfg = dict(type=TrainLoop, max_epochs=max_epochs)

#######################################################################
#                           PART 5  Runtime                           #
#######################################################################
# Log the dialogue periodically during the training process, optional
custom_hooks = [
    dict(type=DatasetInfoHook, tokenizer=tokenizer),
    dict(
        type=EvaluateChatHook,
        tokenizer=tokenizer,
        every_n_iters=evaluation_freq,
        evaluation_inputs=evaluation_inputs,
        system=SYSTEM,
        prompt_template=prompt_template)
]

if use_varlen_attn:
    custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]

# configure default hooks
default_hooks = dict(
    # record the time of every iteration.
    timer=dict(type=IterTimerHook),
    # print log every 10 iterations.
    logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),
    # enable the parameter scheduler.
    param_scheduler=dict(type=ParamSchedulerHook),
    # save checkpoint per `save_steps`.
    checkpoint=dict(
        type=CheckpointHook,
        by_epoch=False,
        interval=save_steps,
        max_keep_ckpts=save_total_limit),
    # set sampler seed in distributed evrionment.
    sampler_seed=dict(type=DistSamplerSeedHook),
)

# configure environment
env_cfg = dict(
    # whether to enable cudnn benchmark
    cudnn_benchmark=False,
    # set multi process parameters
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
    # set distributed parameters
    dist_cfg=dict(backend='nccl'),
)

# set visualizer
visualizer = None

# set log level
log_level = 'INFO'

# load from which checkpoint
load_from = None

# whether to resume training from the loaded checkpoint
resume = False

# Defaults to use random seed and disable `deterministic`
randomness = dict(seed=None, deterministic=False)

# set log processor
log_processor = dict(by_epoch=False)

@rourouZ
Please set map_num_proc=1 for process_hf_dataset to disable the multi-process during the dataset preprocess
data_assitant = dict(
    type=process_hf_dataset,
+   map_num_proc=1,
    dataset=dict(type=load_dataset, path=data_path),
    # dataset=dict(
    #     type=load_dataset,
    #     path='json',
    #     data_files=all_file_list),
    tokenizer=tokenizer,
    max_length=max_length,
    dataset_map_fn=None,
    template_map_fn=dict(
        type=template_map_fn_factory, template=prompt_template),
    remove_unused_columns=True,
    shuffle_before_pack=True,
    pack_to_max_length=pack_to_max_length,
    use_varlen_attn=use_varlen_attn)

map_num_proc=1,

可以了，非常感谢

authors: LZHgrla rourouZ

repo: InternLM/xtuner


text:
http binding: Negotiation examples with '@id' are misleading and probably security-wise not best practice

During implementation I stumbled over the @id fields in the http binding
https://github.com/International-Data-Spaces-Association/ids-specification/blob/main/negotiation/contract.negotiation.binding.https.md
urn:uuid:dcbf434c-eacf-4582-9a02-f8dd50120fd3 is used in many places, e.g. the
GET on the negotation resource and also the POST on negotiation/request
to my understanding, that means the consumer sets an @id in its request and the provider would take this very same (consumer given id) under which the consumer can fetch the state of the request (from the provider). I think the provider should NOT take over a consumer given id in any case. The provider MUST generate its own resource id.

The @id is the correlation id that will be used for callback messages.

Yes, the callback needs a correlation to the user given id, correct. But never the provider should make a resource available under the consumer given id as I would read the example right now.
Any thoughts? Maybe just a copy/paste issue?
--
Matthias Binzer

Yes, as stated, the @id should be used as the correlation id for callback requests. This is necessary to implement reliability guarantees in implementations that choose to do so. The provider is not required to use the correlation id as its internal unique identifier.

The provider is not required to use the correlation id as its internal unique identifier.

Yes, this is how it should be and the text also describes it that way. But the examples look differently. There, the id from the request (of the consumer) is taken as the id on the provider side, which I think can be very misleading. I would propose to use different ids in 2.5.1 (request and response, including the location header) and 2.4.1
Additionally I think we should make a non-normative statement, probably not in http binding, but in the general part, that:

processId is NOT meant to be like a thread id, but only a reference between a request / response (previous state -> next state) reference.

That's at least how I would interpret it. Is this correct? I couldn't find the place (in negotiation) how the usage of processId is described. I can find multiple statements like:

A XXMessage must contain a processId.

But no details about what it should be exactly.

processId is NOT meant to be like a thread id, but only a reference between a request / response (previous state -> next state) reference.

That's at least how I would interpret it. Is this correct? I couldn't find the place (in negotiation) how the usage of processId is described. I can find multiple statements like:

ok, seems I misunderstood this part. It now sounds to me more like an id that stays the same for the entire 'process' / (thread).

The contract negotiation unique id.

It is also part of the example here:
https://github.com/International-Data-Spaces-Association/ids-specification/blob/main/negotiation/message/contract-negotiation.json
but unfortunately not part of the example here (that I used to understand the process...):
https://github.com/International-Data-Spaces-Association/ids-specification/blob/main/negotiation/contract.negotiation.binding.https.md#241-get
I think this caused the confusion. I only figured out because how processId is used in the transfer process.
I think we should:

improve the description
from: The contract negotiation unique id to:
The contract negotiation unique id that stays the same for the remaining negotiation process.
Optionally we could even extend the information to: It is generated by the consumer as the @id field in the request and will be taken over by the provider as the processId for further reference.
and remove it from the ContractNegotiationError section to a more general section
https://github.com/International-Data-Spaces-Association/ids-specification/blob/main/negotiation/contract.negotiation.binding.https.md#22-contract-negotiation-error
and update the example in the http binding to contain this very relevant piece of information (processId) https://github.com/International-Data-Spaces-Association/ids-specification/blob/main/negotiation/contract.negotiation.binding.https.md#241-get

and from the original issue:

update the http binding examples to NOT reuse the same message @id fields in 2.4.1 and 2.5.1

We need to discuss this again, because it turns out, EDC does use the processId also for the path param, e.g. /negotiations/{processId}/agreement where it expects the callbacks and NOT the @id from the ContractRequestmessage from the consumer.
I think whatever is correct, we need to explain this in more detail.

Meeting summary: needs a followup with: @jimmarino  @juliapampus @ndr_brt @matgnt

After a discussion (in a smaller group of @jimmarino @ndr-brt ) we agreed on the following:

processId will be changed into consumerId and providerId.
http Location header with an id will be removed
all provider endpoints using :id will use the providerId
Initial RequestMessages will contain only the consumerId
All follow-up POST messages will contain only 1 of the ids, depending where the message is sent to, e.g. messages to the consumer only contain the consumerId

This needs to go through a bigger group to be confirmed, maybe @juliapampus @sebbader-sap @mkollenstart if you could have a look at this.

@sebbader-sap asks for clearer names for consumerId and providerId

What about consumerPid and providerPid?

summary from the meeting today:
open questions:

use consumerPid instead of providerPid
how about http binding. is the field still required even though it is transferred via path? what if the 2 ids are different?
should we include both ids in all messages (when available, meaning not in the start messages...)

Decision Proposal
Simplify transfer and negotiation processes.
General

processId will be changed into consumerPid and providerPid.
http Location header with an id will be removed
all provider endpoints using :id will use the providerPid
all consumer endpoints using :id will use the consumerPid
initial messages will contain only the consumerPid (or providerPid)

Option 1
see comment

All follow-up messages will contain only 1 of the IDs, depending where the message is sent to, e.g., messages to the consumer only contain the consumerPid

Example:
POST https://connector.consumer.com/negotiations/:consumerPid/agreement

Authorization: ...

{
  "@context": "https://w3id.org/dspace/v0.8/context.json",
  "@type": "dspace:ContractAgreementMessage",
  "dspace:consumerPid": "urn:uuid:a343fcbf-99fc-4ce8-8e9b-148c97605aab",
  "dspace:agreement": {
    "@type": "odrl:Agreement",
    "@id": "e8dc8655-44c2-46ef-b701-4cffdc2faa44",
    "dspace:consumerId": "...",
    "dspace:providerId": "...",
    }
  }
}

Option 2
see comment

All follow-up messages contain the providerPid and consumerPid (except the initial messages)

Example:
POST https://connector.consumer.com/negotiations/:consumerPid/agreement

Authorization: ...

{
  "@context": "https://w3id.org/dspace/v0.8/context.json",
  "@type": "dspace:ContractAgreementMessage",
  "dspace:consumerPid": "urn:uuid:a343fcbf-99fc-4ce8-8e9b-148c97605aab",
  "dspace:providerPid": "urn:uuid:a343fcbf-99fc-4ce8-8e9b-148c57398-abc",
  "dspace:agreement": {
    "@type": "odrl:Agreement",
    "@id": "e8dc8655-44c2-46ef-b701-4cffdc2faa44",
    "dspace:consumerId": "...",
    "dspace:providerId": "...",
    }
  }
}

The JSON-LD messages in the examples still contain the consumerId & providerId instead of consumerPid and providerPid. And in option 1 the provider ID is also still in the JSON message, shouldn't that one only include the consumerPid?

decided on 31.08.2023 to implement option 2

Done with #151 and #152

authors: jimmarino juliapampus matgnt mkollenstart ssteinbuss


text:
fix: update docker image location in readme

This PR updates the docker image location to use the official intersectmbo published images.

superseded by #5771

authors: carbolymer stevanlohja

repo: IntersectMBO/cardano-node


text:
Improve protocolInfoCardano

This issue lists the different steps taken towards improving the protocolInfoCardano function

[x] #275
[x] #276
[x] #317
[x] #318
[ ] Rename ProtocolVersion to SupportedProtocolVersion or something similar, since the name has often caused confusion.
[x] Encode the invariant that protocol versions should be strictly increasing (except for the last era).
[x] #324
[x] https://github.com/input-output-hk/cardano-ledger/issues/3682

FTR: There have been some changes to protocolInfoCardano in #377 due to the new TransitionConfig concept from Ledger.

@amesgen, when you have time, we should discuss the remaining item in this issue and whether we can close it, as I don't see a sensible renaming.

As discussed out-of-band with @amesgen, with 1224, creating a separate type is no longer necessary.

authors: amesgen dnadales jorisdral

repo: IntersectMBO/ouroboros-consensus


text:
TrackCosts are retained more than expected for Darwin-x86-64

Summary
The testcase trackCostsRetaining https://github.com/IntersectMBO/plutus/blob/master/plutus-core/untyped-plutus-core/test/Evaluation/Builtins/Definition.hs#L458
fails only on Darwin-x86-64 with message:
TrackCosts: retaining:                                                             FAIL (0.51s)
untyped-plutus-core/test/Evaluation/Builtins/Definition.hs:482:
Too many elements picked up by GC
Expected at most: 5
But got: 6
The result was: [6829,0,0,0,0,3173]
Use -p '/TrackCosts: retaining/' to rerun this test only.

See also: https://ci.iog.io/build/5076829
@effectfully
Steps to reproduce the behavior
No response
Actual Result
No response
Expected Result
No response
Describe the approach you would take to fix this
No response
System info
OS: Darwin-x86-64
Plutus-1.32

Note that the particular failing test has been disabled for all ghc8.10:
https://github.com/IntersectMBO/plutus/blob/b7fbfd0b12e93bada7775b35e3c625a754940b67/plutus-core/untyped-plutus-core/test/Evaluation/Builtins/Definition.hs#L1141-L1152

repo: IntersectMBO/plutus


text:
🛑 alpha-as.no is down

In e472680, alpha-as.no (https://alpha-as.no) was down:

HTTP code: 0
Response time: 0 ms

Resolved: alpha-as.no is back up in ae09ecb after 2 days, 4 hours, 22 minutes.

repo: Intus-AS/Types-status


text:
🛑 simonsinterior.no is down

In b40f63e, simonsinterior.no (https://simonsinterior.no) was down:

HTTP code: 0
Response time: 0 ms

Resolved: simonsinterior.no is back up in 6cc99a4 after 11 hours, 33 minutes.


text:
🛑 xstadion.no is down

In 2557b6d, xstadion.no (https://xstadion.no) was down:

HTTP code: 0
Response time: 0 ms

Resolved: xstadion.no is back up in 2f8a24e after 32 minutes.


text:
Add option in UI to set the points for each question

Currently, each question gets one point, so for n points there will be n questions. This needs to be fixed so that user can choose points for each questions. This function needs to be added in the UI for all users (right now it works for admin only). It should be added as a preference for each assignment group and implemented such that if it is set for one assignment, it is set for the entire assignment group.

Actually that's a duplicate of #21


text:
Set .CodeMirror z-index to 0

When using easy-markdown-editor in context with other packages to build forms. The easy-markdown-editor package causes problems with the stacking order.
Setting the base z-index to 0 instead of 1 will solve this issue
Example
easy-markdown-editor used together with react-select
Before

After

Ok for me, I tried on my project:
Before

After

@Ionaru How are releases done? Now that this bug is fixed I would love to get this fix into react-simplemde-editor and eventually our component library

repo: Ionaru/easy-markdown-editor


text:
Important Message

Github forces everybody to use 2FA.
I will probably not enable this. I'm just a hobby programmer and this data grab to my mobile phone is not worth it.
I highly doubt that somebody will alter my code secretly.
Over the years all the donations combined were below 50$, so it is not worth the effort anyway.

So what are the plans if you have decided! migrate to codeberg etc?

I will probably move to Gitlab, I didn't know codeberg, will take a look at, thanks!

PipePipe (formerly AnimePipe) is also developed on codeberg
https://codeberg.org/NullPointerException/PipePipe
NewPipe code backups were also taken on codeberg in the past in fear of DMCA takedown, but i haven't checked recently.

Thanks for the tip for codeberg, opened an account, same name, already migrated some code. Even the connection to Android Studio simply worked. Had lots of problems to push code for Newpipe19 to Github this year.
I like BraveNewpipe with Rumble and Bitchute access.

Github forces everybody to use 2FA. I will probably not enable this. I'm just a hobby programmer and this data grab to my mobile phone is not worth it. I highly doubt that somebody will alter my code secretly.
Over the years all the donations combined were below 50$, so it is not worth the effort anyway.

I agree, 2FA intentions are very questionable by these big tech companies.  There has to be better way than giving out your phone number.

authors: Isayso ShareASmile slippyC

repo: Isayso/PlaylistEditorTV


text:
Consistently downcase the masterKubeConfig path

This PR is a short fix for a bug I encountered when trying to provision a node whose fqdn contains uppercase letters (eg.MY-OSE-SERVER.example.com).
The symptoms
The chef run failed with a broken origin-node service.
The interesting snippet was in the origin-node service logs :
May 21 12:19:15 MY-OSE-SERVER origin-node[32335]: I0521 12:19:15.969796   32335 start_node.go:309] Reading node configuration from /etc/origin/node/node-config.yaml
May 21 12:19:15 MY-OSE-SERVER origin-node[32335]: Invalid NodeConfig /etc/origin/node/node-config.yaml
May 21 12:19:15 MY-OSE-SERVER origin-node[32335]: masterKubeConfig: Invalid value: "/etc/origin/node/system:node:MY-OSE-SERVER.example.com.kubeconfig": could not read file: stat /etc/origin/node/system:node:MY-OSE-SERVER.example.com.kubeconfig: no such file or directory
May 21 12:19:15 MY-OSE-SERVER systemd[1]: origin-node.service: main process exited, code=exited, status=255/n/a

The investigation
Apparently, the cookbook downcases the path of the master kube config path when it creates it on the disk, but it does not downcase the path when referencing that file in /etc/origin/node/node-config.yaml.
The fix
The fix is just to downcase the masterKubeConfig entry in the template that is rendered into /etc/origin/node/node-config.yaml
Before this PR:
$ grep --color -ry system:node.*fqdn.*kubeconfig *
templates/default/openshift_node/node.yaml.v1.erb:masterKubeConfig: system:node:<%= node['fqdn'].downcase %>.kubeconfig
templates/default/node.yaml.erb:masterKubeConfig: system:node:<%= node['fqdn'] %>.kubeconfig

After this PR:
$ grep --color -ry system:node.*fqdn.*kubeconfig *
templates/default/openshift_node/node.yaml.v1.erb:masterKubeConfig: system:node:<%= node['fqdn'].downcase %>.kubeconfig
templates/default/node.yaml.erb:masterKubeConfig: system:node:<%= node['fqdn'].downcase %>.kubeconfig

Please do NOT merge, there are other broken behavior when the fqdn contains non downcase letters, for example origin-ndoe does not start because "Could not find an allocated subnet for node: MY-OSE-SERVER.example.com, Waiting..." ... I will force downcase fqdn and reinstall.


text:
Underworld is broken on some world sizes

There's a fuckton of lava in my underworld - world size is 15,000 by 5,000. I can barely get far down enough to SEE hellstone.

nvm i entirely forgot a setting i had turned on mb