text:
INDArray.get() single indexing on row vectors

Single indexing on row vectors fails:
myRowVector.get(NDArrayIndex.interval(idx, range + idx));
whereas myRowVector.get(new NDArrayIndex(0),NDArrayIndex.interval(idx, range + idx)); works

Would you specify example values?
I confirmed the following example works.
scala> import org.nd4j.linalg.indexing._

scala> val arr = Nd4j.create(Array(1f,2f,3f,4f))
arr: org.nd4j.linalg.api.ndarray.INDArray = [ 1.00, 2.00, 3.00, 4.00]

scala> val subarr = arr.get(NDArrayIndex.interval(0,2))
subarr: org.nd4j.linalg.api.ndarray.INDArray = [ 1.00, 2.00]

But row vector's get doesn't work,  if IntervalIndex has explicit stride field.
//This should return [1.00, 3.00].
scala> val subarr2 = arr.get(NDArrayIndex.interval(0,2,3))
subarr2: org.nd4j.linalg.api.ndarray.INDArray = [ 1.00, 2.00]

This was an issue that came from MultiLayerNetwork.setParams() and BaseLayer.setParams() doing .get() with single indexing on a row vector. I've just tested this with single indexing (i.e., just .get(interval(...)) as opposed to using .get(NDArrayIndex.point(0),interval(...))) an it seems to be functioning correctly at present.

I see, so it must be fine if it's working on your side. :)
Can I close this issue?

Seems to be resolved.

authors: AlexDBlack taisukeoe


text:
Update row/column in nd4s

I can update one value like this:
val m = Nd4j.create(2, 2)
m(1, 0) = 1.0

So I expected updating row/column would work like this
m(0, ->) = Array(1.0, 2.0)

But this throws an error
<console>:71: error: overloaded method value update with alternatives:
  (i1: org.nd4s.IndexRange,i2: org.nd4s.IndexRange,num: org.nd4j.linalg.api.complex.IComplexNumber)(implicit ev: =:=[org.nd4j.linalg.api.ndarray.INDArray,org.nd4j.linalg.api.complex.IComplexNDArray])org.nd4j.linalg.api.complex.IComplexNDArray <and>
  (i1: org.nd4s.IndexRange,i2: org.nd4s.IndexRange,num: Double)org.nd4j.linalg.api.ndarray.INDArray
 cannot be applied to (Int, org.nd4s.Implicits.->.type, Array[Double])
              m(0,->) = Array(1.0, 2.0)

Hi.  This PR adds the almost equivalent syntax.
https://github.com/deeplearning4j/nd4s/pull/62
At this moment, I would enforce r.h.s type as INDArray since Array doesn't have shape.
If it adds implicit conversion from Array to INDArray, it may make confusion in the following situation which depends on shape.
//If shape is 2x1, it cannot update all cells.
scala> m(->, ->) = Array(1.0,2.0,3.0,4.0).asNDArray(2,1)
res9: org.nd4j.linalg.api.ndarray.INDArray =
[[1.00,2.00]
 [0.00,0.00]]

//If shape is 2x2, it works.
scala> m(->, ->) = Array(1.0,2.0,3.0,4.0).asNDArray(2,2)
res10: org.nd4j.linalg.api.ndarray.INDArray =
[[1.00,2.00]
 [3.00,4.00]]

source: gharchive/issue

authors: jarutis taisukeoe

license_type: permissive

license_source: bigquery


text:
Haiku leaks tracers when used with JIT and custom JVP

Am I using Haiku incorrectly?
When I enable export JAX_CHECK_TRACER_LEAKS=1 and then
import haiku as hk
import jax

@jax.custom_jvp
def t(y):
    return y

def t_jvp(primals, tangents):
    assert False

t.defjvp(t_jvp)

@jax.jit
def s(y):
    return t(y)

def f():
    return s(0.0)

transformed = hk.transform_with_state(f)
transformed.init(None)

gives
Traceback (most recent call last):
  File "/home/neil/src/cmm/a.py", line 26, in <module>
    transformed.init(None)
  File "/home/neil/src/dm-haiku/haiku/_src/transform.py", line 281, in init_fn
    f(*args, **kwargs)
  File "/home/neil/src/cmm/a.py", line 22, in f
    return s(0.0)
  File "/home/neil/.pyenv/versions/3.9.1/lib/python3.9/contextlib.py", line 124, in __exit__
    next(self.gen)
jax._src.traceback_util.FilteredStackTrace: Exception: Leaked trace MainTrace(0,DynamicJaxprTrace)

The stack trace above excludes JAX-internal frames.
The following is the original exception that occurred, unmodified.

Seems to be a JAX problem actually.

Thanks for digging into this!

authors: NeilGirdhar tomhennigan

repo: deepmind/dm-haiku


text:
Fusion of RGB and flow models?

From the paper:
"experimentally we still found it valuable to have a two-stream configuration ... with one I3D network trained on RGB inputs, and another on flow inputs which carry optimized, smooth flow
information. We trained the two networks separately and averaged their predictions at test time".
Have you attempted to merge the RGB and flow models instead during training? Thanks!

I tried training both models jointly and the performance was similar, but
it takes 2x more memory and makes it necessary to use smaller batch sizes.
Joao
On Wed, Apr 10, 2019 at 6:22 AM Roman notifications@github.com wrote:

From the paper:
"experimentally we still found it valuable to have a two-stream
configuration ... with one I3D network trained on RGB inputs, and another
on flow inputs which carry optimized, smooth flow
information. We trained the two networks separately and averaged their
predictions at test time".
Have you attempted to merge the RGB and flow models instead during
training? Thanks!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/deepmind/kinetics-i3d/issues/60, or mute the thread
https://github.com/notifications/unsubscribe-auth/AO6qaoyOSUgNS99tA8eMhvyTUuAezVQ2ks5vfXT-gaJpZM4cmI7O
.


text:
Issues with Hanabi rewards and self-play.

Hello,
I was checking policy gradient algorithm on HLE.
Firstly, I think there is something wrong with how Hanabi handles rewards (or I am missing something). I ran this simple code (similar to PG for poker example) and looked how agents save rewards. I made discount =  0, so agent._dataset['returns'] is a list of immediate rewards.
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import tensorflow as tf
from open_spiel.python import policy
from open_spiel.python import rl_environment
from open_spiel.python.algorithms import policy_gradient

game = "hanabi"
num_players = 2
discount =0
env_configs = {"players": num_players, 'max_life_tokens' : 1, 'colors' : 2,
               'ranks' : 5, 'hand_size' : 2, 'max_information_tokens' : 3, 'discount' : discount}
env = rl_environment.Environment(game, **env_configs)
info_state_size = env.observation_spec()["info_state"][0]
num_actions = env.action_spec()["num_actions"]

with tf.Session() as sess:
    agents = [policy_gradient.PolicyGradient(sess, idx, info_state_size, num_actions,
                                             hidden_layers_sizes=(128,)) for idx in range(num_players)]

    sess.run(tf.global_variables_initializer())
    for ep in range(1):
        time_step = env.reset()
        while not time_step.last():
            player_id = time_step.observations["current_player"]
            print('Player %d' % player_id)
            fireworks_before = env._state.observation()[30:47]

            agent_output = agents[player_id].step(time_step)
            action_list = [agent_output.action]
            time_step = env.step(action_list)

            if not time_step.last():
                fireworks_after = env._state.observation()[30:47]
            else:
                fireworks_after = 'lost'

            print(fireworks_before, '-->', fireworks_after, '\n')
        for agent in agents:
            agent.step(time_step)
print('\n')
print('Agent 0 rewards history:')
print(agents[0]._dataset['returns'])
print('Agent 1 rewards history:')
print(agents[1]._dataset['returns'])

This is an example of weird output.
P0 get 0pt, then P1 gets 1pt and then P0 loses. As I understand, P0 should have rewards [1, -1] which is correct. However P1 should have [0] since he scored a point and right after P0 lost it.
Player0
Fireworks: R0 Y0  --> Fireworks: R0 Y0
Player 1
Fireworks: R0 Y0  --> Fireworks: R0 Y1
Player 0
Fireworks: R0 Y1  --> lost
Agent 0 rewards history:
[1.0, -1.0]
Agent 1 rewards history:
[-1.0]
Second question is concerning self-play. I would like to run PG agent in a self-play there it plays with itself (not similar agent.) All examples I found in open_spiel have several copies of agents playing with each other and learning separately.  As I understand, using one agent to play with itself in current implementation will not work since episode experience of different players will mix up.

Thanks for the report and the clear reproduction!
We agree the behaviour isn't right and we are discussing how best to fix it.
This may take us a couple of days, but we'll be as quick as we can.
On Mon, Sep 23, 2019 at 3:47 PM griprox notifications@github.com wrote:

Hello,
I was checking policy gradient algorithm on HLE.
Firstly, I think there is something wrong with how Hanabi handles rewards
(or I am missing something). I ran this simple code (similar to PG for
poker example) and looked how agents save rewards. I made discount = 0, so
agent._dataset['returns'] is a list of immediate rewards.
from future import absolute_importfrom future import divisionfrom future import print_functionimport tensorflow as tffrom open_spiel.python import policyfrom open_spiel.python import rl_environmentfrom open_spiel.python.algorithms import policy_gradient
game = "hanabi"
num_players = 2
discount =0
env_configs = {"players": num_players, 'max_life_tokens' : 1, 'colors' : 2,
'ranks' : 5, 'hand_size' : 2, 'max_information_tokens' : 3, 'discount' : discount}
env = rl_environment.Environment(game, **env_configs)
info_state_size = env.observation_spec()["info_state"][0]
num_actions = env.action_spec()["num_actions"]
with tf.Session() as sess:
agents = [policy_gradient.PolicyGradient(sess, idx, info_state_size, num_actions,
hidden_layers_sizes=(128,)) for idx in range(num_players)]
sess.run(tf.global_variables_initializer())
for ep in range(1):
    time_step = env.reset()
    while not time_step.last():
        player_id = time_step.observations["current_player"]
        print('Player %d' % player_id)
        fireworks_before = env._state.observation()[30:47]

        agent_output = agents[player_id].step(time_step)
        action_list = [agent_output.action]
        time_step = env.step(action_list)

        if not time_step.last():
            fireworks_after = env._state.observation()[30:47]
        else:
            fireworks_after = 'lost'

        print(fireworks_before, '-->', fireworks_after, '\n')
    for agent in agents:
        agent.step(time_step)print('\n')print('Agent 0 rewards history:')print(agents[0]._dataset['returns'])print('Agent 1 rewards history:')print(agents[1]._dataset['returns'])

This is an example of weird output.
P0 get 0pt, then P1 gets 1pt and then P0 loses. As I understand, P0 should
have rewards [1, -1] which is correct. However P1 should have [0] since he
scored a point and right after P0 lost it.
Player0
Fireworks: R0 Y0 --> Fireworks: R0 Y0
Player 1
Fireworks: R0 Y0 --> Fireworks: R0 Y1
Player 0
Fireworks: R0 Y1 --> lost
Agent 0 rewards history:
[1.0, -1.0]
Agent 1 rewards history:
[-1.0]
Second question is concerning self-play. I would like to run PG agent in a
self-play there it plays with itself (not similar agent.) All examples I
found in open_spiel have several copies of agents playing with each other
and learning separately. As I understand, using one agent to play with
itself in current implementation will not work since episode experience of
different players will mix up.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/deepmind/open_spiel/issues/68?email_source=notifications&email_token=AHAF7THK47GHPIR67IU35M3QLDJIZA5CNFSM4IZL2DZKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HNB2RTA,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHAF7TGYGMUBKEHRXPUZCDTQLDJIZANCNFSM4IZL2DZA
.

Thanks for the report and the clear reproduction! We agree the behaviour isn't right and we are discussing how best to fix it. This may take us a couple of days, but we'll be as quick as we can.
…
On Mon, Sep 23, 2019 at 3:47 PM griprox @.***> wrote: Hello, I was checking policy gradient algorithm on HLE. Firstly, I think there is something wrong with how Hanabi handles rewards (or I am missing something). I ran this simple code (similar to PG for poker example) and looked how agents save rewards. I made discount = 0, so agent._dataset['returns'] is a list of immediate rewards. from future import absolute_importfrom future import divisionfrom future import print_functionimport tensorflow as tffrom open_spiel.python import policyfrom open_spiel.python import rl_environmentfrom open_spiel.python.algorithms import policy_gradient game = "hanabi" num_players = 2 discount =0 env_configs = {"players": num_players, 'max_life_tokens' : 1, 'colors' : 2, 'ranks' : 5, 'hand_size' : 2, 'max_information_tokens' : 3, 'discount' : discount} env = rl_environment.Environment(game, **env_configs) info_state_size = env.observation_spec()["info_state"][0] num_actions = env.action_spec()["num_actions"] with tf.Session() as sess: agents = [policy_gradient.PolicyGradient(sess, idx, info_state_size, num_actions, hidden_layers_sizes=(128,)) for idx in range(num_players)] sess.run(tf.global_variables_initializer()) for ep in range(1): time_step = env.reset() while not time_step.last(): player_id = time_step.observations["current_player"] print('Player %d' % player_id) fireworks_before = env._state.observation()[30:47] agent_output = agents[player_id].step(time_step) action_list = [agent_output.action] time_step = env.step(action_list) if not time_step.last(): fireworks_after = env._state.observation()[30:47] else: fireworks_after = 'lost' print(fireworks_before, '-->', fireworks_after, '\n') for agent in agents: agent.step(time_step)print('\n')print('Agent 0 rewards history:')print(agents[0]._dataset['returns'])print('Agent 1 rewards history:')print(agents[1]._dataset['returns']) This is an example of weird output. P0 get 0pt, then P1 gets 1pt and then P0 loses. As I understand, P0 should have rewards [1, -1] which is correct. However P1 should have [0] since he scored a point and right after P0 lost it. Player0 Fireworks: R0 Y0 --> Fireworks: R0 Y0 Player 1 Fireworks: R0 Y0 --> Fireworks: R0 Y1 Player 0 Fireworks: R0 Y1 --> lost Agent 0 rewards history: [1.0, -1.0] Agent 1 rewards history: [-1.0] Second question is concerning self-play. I would like to run PG agent in a self-play there it plays with itself (not similar agent.) All examples I found in open_spiel have several copies of agents playing with each other and learning separately. As I understand, using one agent to play with itself in current implementation will not work since episode experience of different players will mix up. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#68?email_source=notifications&email_token=AHAF7THK47GHPIR67IU35M3QLDJIZA5CNFSM4IZL2DZKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HNB2RTA>, or mute the thread https://github.com/notifications/unsubscribe-auth/AHAF7TGYGMUBKEHRXPUZCDTQLDJIZANCNFSM4IZL2DZA .

Thank you!

Our chosen solution is to require agent.step() to be called for every
environment step on which there may be a reward, not just the ones at which
it is the agent's turn to play. This requires a slight modification to the
agent implementation, coming later today.

Player 0
Fireworks: R0 Y0  --> Fireworks: R0 Y0
Player 1
Fireworks: R0 Y0  --> Fireworks: R0 Y0
Player 0
Fireworks: R0 Y0  --> Fireworks: R0 Y1
Player 1
Fireworks: R0 Y1  --> Fireworks: R0 Y1
Player 0
Fireworks: R0 Y1  --> Fireworks: R0 Y1
Player 1
Fireworks: R0 Y1  --> Fireworks: R0 Y1
Player 0
Fireworks: R0 Y1  --> lost
Agent 0 rewards history:
[0.0, 0.0, 0.0, -1.0]
Agent 1 rewards history:
[1.0, 0.0, -1.0]
Still not working.

Our chosen solution is to require agent.step() to be called for every environment step on which there may be a reward, not just the ones at which it is the agent's turn to play. This requires a slight modification to the agent implementation, coming later today.

Hello, it seems that your solution is still not working.
After few tests I got following example:
Player 0
Fireworks: R0 Y0  --> Fireworks: R0 Y0
Player 1
Fireworks: R0 Y0  --> Fireworks: R0 Y0
Player 0
Fireworks: R0 Y0  --> Fireworks: R0 Y1
Player 1
Fireworks: R0 Y1  --> Fireworks: R0 Y1
Player 0
Fireworks: R0 Y1  --> Fireworks: R0 Y1
Player 1
Fireworks: R0 Y1  --> Fireworks: R0 Y1
Player 0
Fireworks: R0 Y1  --> lost
Agent 0 rewards history:
[0.0, 0.0, 0.0, -1.0]
Agent 1 rewards history:
[1.0, 0.0, -1.0]
In general, player who plays card does not save the reward into his buffer. Also, new implementation of PolicyGradient contains a bug,  line 256 is
self.player_id == time_step.current_player()):it raises error saying that time_step does not have .current_player() method. I guess the line should be self.player_id == time_step.observations['current_player']):`

Did you update your main loop to call step() for every player on every timestep?
e.g.
agent_output = [agent.step(time_step) for agent in agents]
time_step = env.step([agent_output[current_player].action])

Are you sure you have updated the whole repository? current_player has been added here:
https://github.com/deepmind/open_spiel/blob/71d76631feb2e62ee1b90421acf981ee11a28fce/open_spiel/python/rl_environment.py#L97

You are right, I did not update the main loop. However, now I did, and there is another problem.
I am running code from below and it looks that now agent saves reward each step it takes -- even on those when it did None action.
info_state_size = env.observation_spec()["info_state"][0]
num_actions = env.action_spec()["num_actions"]
with tf.Session() as sess:
    agents = [policy_gradient.PolicyGradient(sess,player_id=player_id,info_state_size=info_state_size,
                                             num_actions=num_actions,hidden_layers_sizes=[8])
              for player_id in range(num_players)]

    sess.run(tf.global_variables_initializer())
    time_step = env.reset()
    while not time_step.last():
        current_player = time_step.observations["current_player"]
        fireworks_before = env._state.observation()[30:47]
        agent_output = [agent.step(time_step) for agent in agents]
        time_step = env.step([agent_output[current_player].action])
        if not time_step.last():
            fireworks_after = env._state.observation()[30:47]
        else:
            fireworks_after = 'lost'
        print(fireworks_before, '-->', fireworks_after, '\n')
    for agent in agents:
        agent.step(time_step)

print('\n')
print('Agent 0 rewards history:')
print(agents[0]._dataset['returns'])
print(agents[0]._dataset['actions'])
print('Agent 1 rewards history:')
print(agents[1]._dataset['returns'])
print(agents[1]._dataset['actions'])

Fireworks: R0 Y0  --> Fireworks: R0 Y0
Fireworks: R0 Y0  --> Fireworks: R0 Y0
Fireworks: R0 Y0  --> Fireworks: R0 Y0
Fireworks: R0 Y0  --> Fireworks: R1 Y0
Fireworks: R1 Y0  --> Fireworks: R1 Y1
Fireworks: R1 Y1  --> lost
Agent 0 rewards history:
[0.0, 0.0, 0.0, 1.0, 1.0, -2.0]
[5, None, 4, None, 3, None]
Agent 1 rewards history:
[0.0, 0.0, 0.0, 1.0, 1.0, -2.0]
[None, 6, None, 2, None, 3]

Great! Glad it's now running okay.
The behaviour you describe is working as intended. Both agents see the same
(complete) reward history.
Does this cause a problem?

Well, it is kind of weird. Firstly, lengths of

It seems very weird. May be I misunderstand how you approach learning in this multiplayer in games in general. As far as I understand, there are two options:

Each player is essentially an individual network and learns only according to his own experience. I though that it is the case on open_spiel.
There is one network for all players, and each player just collects batch of experience. Then. all players' experience is used to make gradient step.
How I understood the original Hanabi paper (and actually github/HLE has an example of Rainbow agent which support my idea) used 2nd approach, but I think that 1st is also plausible.

Now, it seems that what agent saves in its experience now is not what it should.
For an agent all other players are basically part of the environment, so its interaction with it looks as follows:
-- Agent makes action (not None action, the real one)
-- Say, immediate reward after the action is r0
-- If episode is terminated, then agent just saves the reward
-- If episode is not terminated, agent waits and accumulates the rewards
-- Each next player i makes action, receives immediate reward r_i. Original 0 agent's reward r0 is then changed as r0 += r_i.
-- If episode is finished after player i made action, then original agent 0 saves accumulated reward
-- If episode does not finish until next agent 0 turn, then it accumulates reward from all other agents and saves it in the beginning of its next turn.
Currently, agent just saves all differences in the game's score and I do not really understand why. Also, learning does not work with None actions. To check it you shall change batch_size to 2, for example, in policy_gradient_test.
Correct me if I misunderstand something,

I replied a bit longer on the reddit thread that I won't copy here, but basically these are two separate paradigms (both are used in different contexts in MARL).

1. Each player is essentially an individual network and learns only according to his own experience. I though that it is the case on open_spiel.

2. There is one network for all players, and each player just collects batch of experience. Then. all players' experience is used to make gradient step.
   How I understood the original Hanabi paper (and actually github/HLE has an example of Rainbow agent which support my idea) used 2nd approach, but I think that 1st is also plausible.

These are two known paradigms in the field of multiagent RL (MARL). 1 is known as independent RL and 2 is known as self-play learning. (1) is maybe more common in the MARL literature more generally whereas the setup the Hanabi paper took (and AlphaZero and TD-Gammon) which is more common in games is (2).
The RL agents such as policy gradients and DQN, interact through the rl_environment (this is partly for historical reasons). When you do that, you normally assume the independent RL setup, because each player is a separate agent and they both think of their environment from their own personal perspective (treating everything else as the environment).

Currently, agent just saves all differences in the game's score and I do not really understand why. Also, learning does not work with None actions. To check it you shall change batch_size to 2, for example, in policy_gradient_test.

The definition of the reward was a choice that we made in the Hanabi Challenge paper and corresponding Hanabi Learning Environment. The reason is that the environment should give a reward signal when a card is scored (since the agent knows this as it happens).

Correct me if I misunderstand something,

You're not wrong. The example you listed out exactly highlights the difference in independent RL and self-play. It is certainly a different setting. To get self-play, you should be able to simply sample data from a single policy, and then go back and loop through all the players and train a single model based on the data from all players.

Hi @griprox , here is an alternative solution you might consider to get self-play working, which involves only a couple of changes to the main loop:

After creating the agents, sync their networks. You can use tf.assign for that.
In the beginning of the episode, sample a player position to be the learner (in 2-player game it's 0 or 1), this agent should .step as usual. The other agents should use .step with is_evaluation=True, so they don't update their networks or buffers.
Periodically sync the other agents' networks with the learner network.

Some considerations:

Syncing the networks too often might induce a harder learning / optimization task. This can cause a cyclical learning behavior (A beats B beats C beats A) or unstable learning process.
As opposed to Independent RL, agents in self-play will learn to "play in every player position" of the game. This might lead to a harder task (so you may need larger networks) or make it easier for the networks to learn as it could serve as a regularizer.
The method @lanctot presented is more data efficient, however slightly more complicated to implement. If you are going for large scale games I'd suggest using @lanctot's suggestion, otherwise the main_loop changes above will work just fine.

@fzvinicius @lanctot
Thank you very much for your responses! I am currently playing with self-play setup of my own implementation, but I hope to try independent agents using open_spiel implementation soon.

Hi @griprox did you have any success in training A2C agents on Hanabi either in your own implementation or using Open Spiel. If so, do you mind sharing the results?

Hi @lanctot , is there someone who has trained A2C under self-play setting on Hanabi using Open Spiel. If so, what are the kind of scores to expect and after how many steps. I have a version that's running but that doesn't seem to be learning anything at all.

Hi @akileshbadrinaaraayanan , my apologies for the delayed reply -- I got the notification while I was on vacation and then completely forgot about it.
As far as I know, nobody has been able to get this to work using vanilla A2C. But I only know of one other person from a conversation on a reddit thread who tried (unless that was you?) and as far as I remember they experienced similar trouble.
I think for actor-critic to work in Hanabi you probably need more than just vanilla A2C: a decent network architecture, probably recurrent, and population-based training. Take a look at the results in our Hanabi challenge paper, e.g. regarding ACHA: https://arxiv.org/pdf/1902.00506.pdf, the population based training made a huge difference (but even the basic architecture was recurrent and trained on batches as well, in addition it used V-TRACE and batched A2C from IMPALA). ACHA was out-performed later by a distributed recurrent DQN in the SAD paper: https://arxiv.org/pdf/1912.02288.pdf, but similarly, there seem to be a lot of enhancements (distributed replay, recurrent nets, auxiliary tasks, double Q-learning, dueling network, etc.)
Have you tried just standard DQN? When you say "not learning anything", do you mean like... 0ish points, 0-5 points, 5-10 points, or <= 15 points?

Ah, the person from the reddit thread was indeed @griprox from this thread. :)

authors: akileshbadrinaaraayanan fzvinicius griprox lanctot locked-deepmind


text:
fix: update OllamaChatGenerator with keep_alive parameter

Related Issues

tests failing: https://github.com/deepset-ai/haystack-experimental/actions/runs/11365200366/job/31612862722?pr=118
a new parameter was added in OllamaChatGenerator with https://github.com/deepset-ai/haystack-core-integrations/pull/1131

Proposed Changes:
Adapt the experimental component to take this new parameter into account
How did you test it?
CI, adapted tests
Checklist

I have read the contributors guidelines and the code of conduct
I have updated the related issue with new insights and changes
I added unit tests and updated the docstrings
I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
I documented my code
I ran pre-commit hooks and fixed any issue

Pull Request Test Coverage Report for Build 11365715369
Details

0 of 0   changed or added relevant lines in 0 files are covered.
5 unchanged lines in 1 file lost coverage.
Overall coverage remained the same at 89.024%

Files with Coverage Reduction
New Missed Lines
%

components/generators/ollama/chat/chat_generator.py
5
93.02%

Totals

Change from base Build 11251453098:
0.0%

Covered Lines:
1606

Relevant Lines:
1804

💛  - Coveralls

source: gharchive/pull-request

repo: deepset-ai/haystack-experimental

license_source: github-api


text:
Implement sendDocument endpoint

https://core.telegram.org/bots/api#senddocument

implemented

repo: def-studio/telegraph

license: MIT


text:
chore: add Made for UDS badge verification task and workflow

Description
Implements the Made for UDS badge verification task for local execution and via a GitHub workflow. The workflow will output the findings into a reports/ archive for download.
BREAKING CHANGES

removes the UI package's extraneous Service sub-manifest, which caused a warning during bundle or package deployment

CHANGES

adds the Made for UDS badge verification UDS task

tasks are run like this: uds run verify-uds-badge-<PACKAGE_NAME>

adds a nightly workflow for UDS badge verification, outputs to archive
adds ignore for tasks.yaml from other workflow triggers

Related Issue
Relates to #1050
Relies on: https://github.com/defenseunicorns/uds-common/pull/249
Checklist before merging

[x] Tests, documentation, ADR added or updated as needed
[x] Followed the Contributor Guide Steps

Blocked by https://github.com/defenseunicorns/uds-common/pull/249

There seems to be an issue with the stdin and read streams within the runner that doesn't allow Python nor bash read the intermediate-report.txt for cleaning. At the moment, the only way to get the clean report is locally. The intermediate-report.txt. is still visible in the saved runner artifacts.

See https://github.com/defenseunicorns/leapfrogai/issues/1169 for the issue tracking the test results.

repo: defenseunicorns/leapfrogai


text:
Figure out Publishing of Zarf Releases

To make Zarf easier to install and work with, we should be publishing releases to commonly used package managers like brew, yum, etc.
GoReleaser might be a good tool to help with this.
End goal of this task is to have releases of Zarf published to these package managers automagically by our build pipeline whenever a new release is created.

Closed by #498 and #507

repo: defenseunicorns/zarf


text:
Added custom api server

Added custom api server that only returns the generated output but not the prompt to simplify the output parsing. Updated README.md with instructions.
Updated gcs_eval.py with option to set NUM_BEAMS in 1 place.

Nice! Super convenient – thank you!!

authors: rishsriv wongjingping

repo: defog-ai/sql-eval


text:
Replace "egrep" with "grep -E"

Both egrep and fgrep have been marked obsolete and will issue a warning to stderr now if used.

egrep, fgrep: now obsolete
https://git.savannah.gnu.org/cgit/grep.git/commit/?id=a95156247098

But "grep -E" does the same and is required by POSIX:

grep - search a file for a pattern
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html

Signed-off-by: Christian Kujau lists@nerdbynature.de

Sorry for the noise, #868 is already open for this.

repo: dehydrated-io/dehydrated


text:
translation_BG

Übersetzung ins Bulgarische. Startseite und Menübaum sind beide angepasst.

Ich habe keine Ahnung warum du die menu.SVG Datei nicht sehen konntest. Bei mir war die vorhanden und ließ sich problemlos öffnen..
ich habe jetzt einfach das "file" in der SVG durch nichts ersetzt. Kannst du mir Rückmeldung geben ob das funktioniert hat oder ob da wieder irgendein Quatsch passiert ist?

ich habe jetzt einfach das "file" in der SVG durch nichts ersetzt.
Kannst du mir Rückmeldung geben ob das funktioniert hat

Nein, hat nicht funktioniert.
Du musst file:// ersetzen durch einen leeren String (nicht nur file).


text:
Build details

Hi, i would like to try to build kbord. As i can see, version described in readme is different from current version of the board. So i have a few questions:

Is current version of the board is working, or i should use version from specific commit?
I noticed that now usb type-c used instead of micro, is there any other changes in BOM?
I'm not a big fan of rgb, will the board work if i will just not solder them?

Hi disfenoid:
The new version of the board is fully working, I have been using one myself.
As for the BOM, USB connector is different, the four pushbuttons are different, and rotary encoders are removed. I think everything else is the same. I'll take a photo of the new board and update the BOM later today.
The board will work fine if you don't solder the RGB LEDs

Here are the photos of the current version:

I'll update the BOM tomorrow.

Great, thank you!

BOM is here: https://github.com/dekuNukem/kbord/blob/master/pcb/kbord_v2_bom.xlsx

BOM is here: https://github.com/dekuNukem/kbord/blob/master/pcb/kbord_v2_bom.xlsx

Hi, I am trying to build kbord but I got some questions about oled screen connector in 2nd version of your PCB. The 30pin wire bus of 0.96" oled screen have  got 1mm step but PCB pads have got 0.5mm step. So finally question:
How do you solve this problem?
Can you attach some photo of backside with connected screen?

@Ep1quell, I ordered this display and connector: aliexpress.com/item/4000375678671.html, but didn't try yet (still waiting for delivery).

@disfenoid , with your help I found one more link, here it is: https://s.click.aliexpress.com/e/bd6hKgeO
The main difference is driver IC. In first version of BOM deku said that he using oled with ssd1306 so I think that this will be better cause there is SH1106 in your link

This one looks like the OLED that I used:
https://www.aliexpress.com/item/32950730252.html
1.3 inch, 30pin FPC connecter, SSH1106 controller.

this listing comes with the FPC connector as well, and also detailed drawings:
https://www.aliexpress.com/item/4000350633981.html
make sure the 30-pin FPC connector is top-contact type,

do you have any details on the new buttons and usb connector?

you can find it here:
https://github.com/dekuNukem/kbord/blob/master/build_it_yourself.md
I significantly updated build details today, hope that helps!

authors: Ep1quell NeverMinde dekuNukem disfenoid kow

repo: dekuNukem/kbord


text:
"Not Empty" expansion suggestion: Not Destroy

Please add a toggleable (or disabled temporarily while holding shift) option to also get rid of the "destroy" option on items.

I'm not sure if I'm not using it correctly, but I'm using the potions in my bag and they are running out. If you could add how to apply it to the porions so they don't run out. I assumed that you just install it and restart the game and they would work, Thanks!

authors: NogglesJ vitorebom

repo: dekvall/runelite-external-plugins


text:
Consider removing memes from docs

First of all, thanks for the great tool! This solves most of my issues with alpinejs/htmx and finally there is a htmx-like tool with a typescript codebase :)
When I'm reading the docs I find it hard to concentrate on the text because the memes play in loops. Maybe you could only use them on the landiing pages/essays/examples but keep the docs a bit more focused? Or add an option to disable them, similar to a dark/light mode toggle.

In the middle of a revamp that does just that

@arnoson  https://data-star.dev is now meme free

great, thanks for the info!

authors: arnoson delaneyj

repo: delaneyj/datastar


text:
How does this compare to android-ddp-client?

How does Android-DDP compare to android-ddp-client? Do both libraries have the same features? I am considering making a switch from android-ddp-client to this library, but I want to make sure that its features are on par first.
See: https://github.com/kenyee/android-ddp-client
I know this isn't an "issue", I just couldn't find any other way to contact the developer.

Good question, thanks!
I've not really taken a closer look at the other library, unfortunately. But if you're a user of that library, you probably know best what features it has and how it works.
So please feel free to ask about any features or the workflow here, and I'll do my best to answer.
Generally, the "Usage" section of this project's README is probably the best point to start:
https://github.com/delight-im/Android-DDP#usage
We can then leave this thread open to collect all information that is relevant to this comparison.

I am considering making the switch from android-ddp-client to your library. It looks very clean and lightweight. I see that you are pretty active on this project and have made quite a few contributions recently, which is also a plus. Do you plan on continuing to make contributions to this library on a regular basis and providing support for the foreseeable future? Or is this just a side project on the back burner that isn't that high of a priority? I would hate to switch to this library only to have development cease and support dropped a day later.

You are welcome to making the switch!
As I said, this library probably has all the features that you need, but I can't say for sure. So please be careful and try things first without spending too much work on this right away :)
I absolutely do plan to continue working on this library and have no plans to stop support. But obviously, I cannot give any support guarantee. (Neither does the other library, however, I guess.) Sorry!
So I'll do my best to continue providing good support for this library and maybe the community can help as well if some problems occur. That's all I can offer ;)

authors: mwaclawek stephen-workpop

repo: delight-im/Android-DDP


text:
Old module name "dellemc_vxrail_clusterinfo" still referenced by documentation

The Examples section for the dellemc_vxrail_getclusterinfo module uses an old module name:  It looks like the name was updated from *_clusterinfo to *_getclusterinfo back in 1.4.0

The old name seems to only persist in the one example section.
I did a light test of all the module names against the documentation and there may be a few other modules that might need some cleanup.  There were a few that did not appear in any markdown (only in docs/samples yaml) and a few that did appear in markdown but didn't seem to make a valid appearance in the Examples section (as is the case with the dellemc_vxrail_getclusterinfo module because the old module name was still being used).
function module_names {  ls -1  plugins/modules/ | grep -oE '^dellemc[a-z0-9_]*';  }
function not_in_docs    {  ! `grep -qRl "${1}"  ${3:-docs} --include="*${2}"`; }
function not_in_markdown { not_in_docs $1 ".md"; }
function not_in_markdown_examples { not_in_docs "${1}:" ".md"; }
function not_in_samples { not_in_docs $1 ".ya?ml" "docs/samples"; }

for module in `module_names`; do
   if `not_in_docs $module`; then
       echo "[MISSING FROM DOCS]  $module"
   else
       if `not_in_markdown $module`; then
           echo "[MISSING FROM MARKDOWN]  $module"
       elif `not_in_markdown_examples $module`; then
           echo "[MISSING MARKDOWN EXAMPLES] $module"
       fi
       ## lots of output if we are checking the samples yaml
       #if  `not_in_samples $module`; then
       #    echo "[MISSING FROM DOCS/SAMPLES] $module"
       #fi
   fi
done

Results in the output:
[MISSING FROM MARKDOWN]  dellemc_vxrail_certificates_importcertsintotruststore
[MISSING FROM MARKDOWN]  dellemc_vxrail_cluster_expansion_cancel
[MISSING MARKDOWN EXAMPLES] dellemc_vxrail_cvs_compliance_report
[MISSING MARKDOWN EXAMPLES] dellemc_vxrail_getclusterinfo
[MISSING MARKDOWN EXAMPLES] dellemc_vxrail_idrac_adduser
[MISSING FROM MARKDOWN]  dellemc_vxrail_sequential_reboot_cancel
[MISSING FROM MARKDOWN]  dellemc_vxrail_sequential_reboot_retry
[MISSING MARKDOWN EXAMPLES] dellemc_vxrail_support_getaccount

https://github.com/dell/ansible-vxrail/pull/26

repo: dell/ansible-vxrail


text:
[ISSUE-688] Bump k8s lib version

Purpose
Resolves #689
Updated:

go 1.15 -> 1.16
k8s libs 0.18.19 -> 0.22.5
csi-spec 1.2.0 -> 1.5.0
controller-runtime 0.6.0 -> 0.9.0

Extra:

commented functional in scheduler pkg

PR checklist

[x] Add link to the issue
[x] Choose Project
[x] Choose PR label
[x] New unit tests added
[x] Modified code has meaningful comments
[x] All TODOs are linked with the issues
[ ] All comments are resolved

Testing
Provide test details

/ci

/ci

/ci

/ci

/ci

authors: rkiselenko safronovD

repo: dell/csi-baremetal


text:
Remove mode: StandAlone from reverseproxy config.yaml

Description
Remove mode: StandAlone from reverseproxy config.yaml
GitHub Issues
List the GitHub issues impacted by this PR:

GitHub Issue #

https://github.com/dell/csm/issues/1567

Checklist:

[X] I have performed a self-review of my own code to ensure there are no formatting, vetting, linting, or security issues
[x] I have verified that new and existing unit tests pass locally with my changes
[x] I have not allowed coverage numbers to degenerate
[x] I have maintained at least 90% code coverage
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] I have added tests that prove my fix is effective or that my feature works
[x] I have maintained backward compatibility

How Has This Been Tested?
-Executed cert-csi successfully
cert-csi-logs.txt.txt
E2E powermax scenario has passed:
[root@master-1-vxa4Qh4cj2ZFH e2e]# ./run-e2e-test.sh --pmax --cert-csi=/root/cert-csi
/root/kk/new-defect/csm-operator/tests/e2e
  W1122 10:04:14.081149  609344 test_context.go:538] Unable to find in-cluster config, using default host : https://127.0.0    .1:6443
  I1122 10:04:14.081329 609344 test_context.go:561] The --provider flag is not set. Continuing as if --provider=skeleton ha    d been used.
Running Suite: CSM Operator End-to-End Tests - /root/kk/new-defect/csm-operator/tests/e2e
=========================================================================================
Random Seed: 1732269845

Will run 1 of 1 specs
------------------------------
[BeforeSuite]
/root/kk/new-defect/csm-operator/tests/e2e/e2e_test.go:98
  STEP: Getting test environment variables @ 11/22/24 10:04:14.088
  STEP: [powermax] @ 11/22/24 10:04:14.088
  STEP: Reading values file @ 11/22/24 10:04:14.088
  STEP: Getting a k8s client @ 11/22/24 10:04:14.095
[BeforeSuite] PASSED [0.013 seconds]
------------------------------
[run-e2e-test] E2E Testing Running all test Given Test Scenarios
/root/kk/new-defect/csm-operator/tests/e2e/e2e_test.go:139
  STEP: Starting: Install PowerMax Driver(Standalone)  @ 11/22/24 10:04:14.099
  STEP: Returning false here @ 11/22/24 10:04:14.099
  STEP:      Executing  Given an environment with k8s or openshift, and CSM operator installed @ 11/22/24 10:04:14.099
  STEP:      Executing  Create storageclass with name [op-e2e-pmax] and template [testfiles/powermax-templates/powermax-sto    rageclass-template.yaml] for [pmax] @ 11/22/24 10:04:14.112
  STEP:      Executing  Set up creds with template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCr    eds] @ 11/22/24 10:04:14.816
  STEP:      Executing  Set up configMap with template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] na    me [powermax-reverseproxy-config] in namespace [powermax] for [pmaxAuthSidecar] @ 11/22/24 10:04:15.355
  STEP:      Executing  Apply custom resource [1] @ 11/22/24 10:04:15.876
  I1122 10:04:15.876544 609344 builder.go:121] Running '/usr/bin/kubectl --namespace=powermax apply --validate=true -f -'
  I1122 10:04:16.106782 609344 builder.go:146] stderr: ""
  I1122 10:04:16.106846 609344 builder.go:147] stdout: "containerstoragemodule.storage.dell.com/powermax created\n"
  STEP:      Executing  Validate custom resource [1] @ 11/22/24 10:04:16.106

err: expected custom resource status to be Succeeded. Got: Failed
  STEP:      Executing  Validate [powermax] driver from CR [1] is installed @ 11/22/24 10:05:06.154
  STEP:      Executing  Run custom test @ 11/22/24 10:05:26.167
  I1122 10:05:26.167096 609344 util.go:650] Running cert-csi [test vio --sc op-e2e-pmax --chainLength 1 --chainNumber 1]
[2024-11-22 10:05:26]  INFO Starting cert-csi; ver. 1.4.0
[2024-11-22 10:05:26]  INFO Using EVENT observer type
[2024-11-22 10:05:26]  INFO Using config from /root/.kube/config
[2024-11-22 10:05:26]  INFO Successfully loaded config. Host: https://10.247.22.239:6443
[2024-11-22 10:05:26]  INFO Created new KubeClient
[2024-11-22 10:05:26]  INFO Running 1 iteration(s)
[2024-11-22 10:05:26]  INFO     *** ITERATION NUMBER 1 ***
[2024-11-22 10:05:26]  INFO Starting VolumeIoSuite with op-e2e-pmax storage class
[2024-11-22 10:05:26]  INFO Successfully created namespace volumeio-test-aef1e93c
[2024-11-22 10:05:26]  INFO Using default number of volumes
[2024-11-22 10:05:26]  INFO Using default image: docker.io/centos:latest
[2024-11-22 10:05:26]  INFO Creating IO pod
[2024-11-22 10:05:26]  INFO Waiting for pod iowriter-test-gdthz to be READY
[2024-11-22 10:05:42]  INFO Executing command: [/bin/bash -c dd if=/dev/urandom bs=1M count=128 oflag=sync > /data0/writer-    0.data]
[2024-11-22 10:05:42]  INFO Executing command: [/bin/bash -c sha512sum /data0/writer-0.data > /data0/writer-0.sha512]
[2024-11-22 10:05:53]  INFO Waiting until no Volume Attachments with PV  left
[2024-11-22 10:05:53]  INFO VolumeAttachment deleted

[2024-11-22 10:42:14]  INFO Avg time of a run:   45.22s
[2024-11-22 10:42:14]  INFO Avg time of a del:   12.01s
[2024-11-22 10:42:14]  INFO Avg time of all:     60.10s
[2024-11-22 10:42:14]  INFO During this run 100.0% of suites succeeded
  STEP:      Executing  Enable forceRemoveDriver on CR [1] @ 11/22/24 10:42:14.614
  STEP:      Executing  Delete custom resource [1] @ 11/22/24 10:42:14.629
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax_reverse_proxy_config.yaml] for [pmaxReverse    Proxy] @ 11/22/24 10:42:14.643
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-storageclass-template.yaml] for [pmax] @ 11    /22/24 10:42:14.673
  STEP:      Executing  Restore template [testfiles/powermax-templates/powermax-secret-template.yaml] for [pmaxCreds] @ 11/    22/24 10:42:14.71
  STEP: Ending: Install PowerMax Driver(Standalone)
   @ 11/22/24 10:42:14.73
• [137.327 seconds]
------------------------------

Ran 1 of 1 Specs in 137.333 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS

created a new PR

repo: dell/csm-operator


text:
Adding the CI Workflow for doing Black Code Formatting check

Closes #4
This workflow checks the code and fails the pipeline if the Black Format Check conditions is not met. Like the image below:

@MrPowers
If required I can make a change to the workflow so that it automatically also formats the codebase according to the black formatting check and then commit the formatted code. Let me know if that's required.

@MrPowers @mshtelma  A review on this please 👆

LGTM.  Will merge this and then fix the files.  I think it's better to not auto-format.  I am too much of a control freak for auto-formatting, haha.

authors: MrPowers souvik-databricks

repo: delta-incubator/deltatorch


text:
PROTOCOL changes in release 0.6.2

PROTOCOL changes in release 0.6.0:

Query Table Version:

change from HEAD to GET.
with a /version suffix.
Support parameter startingTimestamp.

Read Data from a Table

Support parameter timestamp.
Support parameter startingVersion.
The response can be data change files and include historical metadata for query with startingVersion.

Read Change Data Feed from a Table:

Support parameter includeHistoricalMetadata
The response will include historical metadata if includeHistoricalMetadata is set to true.

Metadata now contains 3 optional fields: version/size/numFiles. version will be returned when querying table data with a version or timestamp parameter, or cdf query with includeHistoricalMetadata set to true, size/numFiles will be returned when available in the delta log.
File now contains 2 optional fields: version/timestamp, which will be returned when querying table with a version or timestamp parameter.

@wchau  Could you give another pass on this PR? Thanks!

repo: delta-io/delta-sharing


text:
[Question] py4j.protocol.Py4JJavaError: An error occurred while calling o38.applySchemaToPythonRDD

Bug
Getting Incompatible Class with the most recent spark and delta-spark versions for all of my unit tests.
I run my tests with:
python -m pytest tests/unit
The error:
E                   py4j.protocol.Py4JJavaError: An error occurred while calling o38.applySchemaToPythonRDD.
E                   : java.lang.IncompatibleClassChangeError: class org.apache.spark.sql.catalyst.plans.logical.DeltaDelete has interface org.apache.spark.sql.catalyst.plans.logical.UnaryNode as super class

Describe the problem
I am having a docker image running python 3.8 and my python application which runs fine with delta-spark 1.0.0 and spark 3.1.2.
I am trying to upgrade my docker image to python 3.9 with delta-spark 2.2.0 and spark 3.3.1. Unfortunately while all the tests of my application pass in the first image in the upgraded one I get the incompatible class error.
The changes between the 2 docker images are:

Python version
Delta spark package
Spark

Steps to reproduce
Unfortunately I cannot provide all the code for my unit tests but I have at least one version combination of spark and delta-spark working.
Observed results
answer = 'xro3487', gateway_client = <py4j.clientserver.JavaClient object at 0xffff64a47880>, target_id = 'o38', name = 'applySchemaToPythonRDD'

    def get_return_value(answer, gateway_client, target_id=None, name=None):
        """Converts an answer received from the Java gateway into a Python object.

        For example, string representation of integers are converted to Python
        integer, string representation of objects are converted to JavaObject
        instances, etc.

        :param answer: the string returned by the Java gateway
        :param gateway_client: the gateway client used to communicate with the Java
            Gateway. Only necessary if the answer is a reference (e.g., object,
            list, map)
        :param target_id: the name of the object from which the answer comes from
            (e.g., *object1* in `object1.hello()`). Optional.
        :param name: the name of the member from which the answer comes from
            (e.g., *hello* in `object1.hello()`). Optional.
        """
        if is_error(answer)[0]:
            if len(answer) > 1:
                type = answer[1]
                value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
                if answer[1] == REFERENCE_TYPE:
>                   raise Py4JJavaError(
                        "An error occurred while calling {0}{1}{2}.\n".
                        format(target_id, ".", name), value)
E                   py4j.protocol.Py4JJavaError: An error occurred while calling o38.applySchemaToPythonRDD.
E                   : java.lang.IncompatibleClassChangeError: class org.apache.spark.sql.catalyst.plans.logical.DeltaDelete has interface org.apache.spark.sql.catalyst.plans.logical.UnaryNode as super class
E                       at java.base/java.lang.ClassLoader.defineClass1(Native Method)
E                       at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
E                       at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
E                       at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:555)
E                       at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
E                       at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
E                       at java.base/java.security.AccessController.doPrivileged(Native Method)
E                       at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
E                       at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
E                       at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
E                       at org.apache.spark.sql.delta.DeltaAnalysis.apply(DeltaAnalysis.scala:64)
E                       at org.apache.spark.sql.delta.DeltaAnalysis.apply(DeltaAnalysis.scala:57)
E                       at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:211)
E                       at scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
E                       at scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
E                       at scala.collection.immutable.List.foldLeft(List.scala:91)
E                       at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:208)
E                       at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:200)
E                       at scala.collection.immutable.List.foreach(List.scala:431)
E                       at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:200)
E                       at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:227)
E                       at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:223)
E                       at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:172)
E                       at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:223)
E                       at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:187)
E                       at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:179)
E                       at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
E                       at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:179)
E                       at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:208)
E                       at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:330)
E                       at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:207)
E                       at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:76)
E                       at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
E                       at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:185)
E                       at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:510)
E                       at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:185)
E                       at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
E                       at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:184)
E                       at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:76)
E                       at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:74)
E                       at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:66)
E                       at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:91)
E                       at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
E                       at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:89)
E                       at org.apache.spark.sql.SparkSession.internalCreateDataFrame(SparkSession.scala:570)
E                       at org.apache.spark.sql.SparkSession.applySchemaToPythonRDD(SparkSession.scala:756)
E                       at org.apache.spark.sql.SparkSession.applySchemaToPythonRDD(SparkSession.scala:741)
E                       at jdk.internal.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
E                       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
E                       at java.base/java.lang.reflect.Method.invoke(Method.java:566)
E                       at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
E                       at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
E                       at py4j.Gateway.invoke(Gateway.java:282)
E                       at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
E                       at py4j.commands.CallCommand.execute(CallCommand.java:79)
E                       at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
E                       at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
E                       at java.base/java.lang.Thread.run(Thread.java:829)

/spark-3.3.1-bin-hadoop3/python/lib/py4j-0.10.9.5-src.zip/py4j/protocol.py:326: Py4JJavaError

Further Details
tests/unit/.../../abc.py:118:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.9/site-packages/../../test_case_runner.py:24: in run_test
    input_df, expected_df = self.test_case_builder.build_testcase_params(inputs, expected)
/usr/local/lib/python3.9/site-packages/../../test_case_parameter_builder.py:25: in build_testcase_params
    input_dfs = [self._convert_to_data_frame(input) for input in inputs]
/usr/local/lib/python3.9/site-packages/../../test_case_parameter_builder.py:25: in <listcomp>
    input_dfs = [self._convert_to_data_frame(input) for input in inputs]
/usr/local/lib/python3.9/site-packages/../../test_case_parameter_builder.py:38: in _convert_to_data_frame
    return self.spark_test_handler.rows_schema_to_df(test_data.data, test_data.schema)
/usr/local/lib/python3.9/site-packages/../spark_test_handler.py:55: in rows_schema_to_df
    data_frame = self.create_df(rows, schema)
/usr/local/lib/python3.9/site-packages/../spark_test_handler.py:51: in create_df
    return self.spark.createDataFrame(self.spark.sparkContext.parallelize(rows, 1), schema)
/spark-3.3.1-bin-hadoop3/python/pyspark/sql/session.py:894: in createDataFrame
    return self._create_dataframe(
/spark-3.3.1-bin-hadoop3/python/pyspark/sql/session.py:939: in _create_dataframe
    jdf = self._jsparkSession.applySchemaToPythonRDD(jrdd.rdd(), struct.json())
/spark-3.3.1-bin-hadoop3/python/lib/py4j-0.10.9.5-src.zip/py4j/java_gateway.py:1321: in __call__
    return_value = get_return_value(
/spark-3.3.1-bin-hadoop3/python/pyspark/sql/utils.py:190: in deco
    return f(*a, **kw)

The package omitted in site-packages in an internal package.
Expected results
The unit tests should be passing as the code has not changed. Running the same code with 2 different docker images results in failed results in the newest image.
Environment information

Python 3.9
Delta Spark 2.2.0
Spark version: 3.3.1
Pyspark 3.3.1

Willingness to contribute
The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?

[ ] Yes. I can contribute a fix for this bug independently.
[ ] Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
[X] No. I cannot contribute a bug fix at this time.

Good morning,
Apparently, the problem is that the delta-core library does not exist. I need to have it inside the the sparks/jars folder and have the JAVA_HOME set.
I also want to try to set put the jar inside the pyspark jar folder to see if it will work that way.
Although I see that in the documentation there is no reference to that in the Python section: https://docs.delta.io/latest/quick-start.html
Maybe there is room for improvement?

@zsxwing: I can understand that but there is a problem.
I have a python project which uses spark and delta tables. I want to run it in a docker image so I pick up the docker image from apache foundation which contains spark. When I try to run the tests for my project it fails because the delta-core library does not exist in the spark jars folder. Consequently as you see, I am not running python directly but my tests ran through python and for these tests I need pyspark and delta-core.
Moreover, the error handling should be better, nothing from the stack traces implies of a missing library which makes debugging the problem very hard.
In my opinion, I see 2 ways here:

The documentation is improved when you do not run pyspark directly with the needed libraries.
The error handling on the missing libraries is improved in code, so the errors become more obvious.

So I see room for improvement.

Thanks for the feedback. Yep, there is definitely room for improvement. I just read your description again and there is something I don't understand right.

java.lang.IncompatibleClassChangeError: class org.apache.spark.sql.catalyst.plans.logical.DeltaDelete has interface org.apache.spark.sql.catalyst.plans.logical.UnaryNode as super class

This error is because delta-lake version (<=1.0.x) and spark version (>=3.2.x) are not compatible. Is it possible that you have an old version delta-spark version somewhere, and putting a new version of delta-core to spark's jars folder happened to fix it?

Good morning @zsxwing,
I checked my docker image and there is nowhere another version of delta. The only version is the one I download. If I remove the jar file from the spark jars folder then I get these errors.
I can retry to be sure or I can share with you how I build the image.

If I remove the jar file from the spark jars folder then I get these errors.

What's your command to start Spark? I'm looking for the --packages parameter.

Hi @zsxwing,
I do not run any pyspark command I wait for it to be invoked from python when the unit tests are called. What i only do is to run the tests which need pyspark from my code repository with:
python -m pytest tests/unit

Hi,
I'm experincing some diff issue , in my case spark application container is restarting .... when i process an it is running fine for first event and if we process 2nd event it is restarting nad throwing below exception , pls help me if any body have an idea on this.

rdd = spark_inst.readCSV(sc_tuple[1], process_cur, schema_sent=schema).rdd.repartition(num_partition) 2023-10-16T07:08:52.091016207Z File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 214, in rdd 2023-10-16T07:08:52.091020214Z File "/opt/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in call 2023-10-16T07:08:52.091022227Z File "/opt/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 179, in deco
Mon, Oct 16 2023 12:38:52 pm | File "/opt/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
Mon, Oct 16 2023 12:38:52 pm | py4j.protocol.Py4JJavaError: An error occurred while calling o424.javaToPython. 2023-10-16T07:08:52.099571794Z : java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
Mon, Oct 16 2023 12:38:52 pm | This stopped SparkContext was created at:
Mon, Oct 16 2023 12:38:52 pm |
Mon, Oct 16 2023 12:38:52 pm | The currently active SparkContext was created at:
Mon, Oct 16 2023 12:38:52 pm |
Mon, Oct 16 2023 12:38:52 pm | org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58) 2023-10-16T07:08:52.099615975Z java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 2023-10-16T07:08:52.099617906Z java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
Mon, Oct 16 2023 12:38:52 pm | java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
Mon, Oct 16 2023 12:38:52 pm | java.base/java.lang.reflect.Constructor.newInstance(Unknown Source) 2023-10-16T07:08:52.099623338Z py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) 2023-10-16T07:08:52.099625123Z py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) 2023-10-16T07:08:52.099626901Z py4j.Gateway.invoke(Gateway.java:238)
Mon, Oct 16 2023 12:38:52 pm | py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
Mon, Oct 16 2023 12:38:52 pm | py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
Mon, Oct 16 2023 12:38:52 pm | py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) 2023-10-16T07:08:52.099634062Z py4j.ClientServerConnection.run(ClientServerConnection.java:106)
Mon, Oct 16 2023 12:38:52 pm | java.base/java.lang.Thread.run(Unknown Source)

Hi. Could you figure out a solution to this issue?

authors: aimtsou imenas manikumarn559 zsxwing

repo: delta-io/delta


text:
Added documentation on how to manage a pull request in CI/CD playbook

Status
Ready/In Progress/In Hold (Reason for hold)
Related Issues
https://jira-hq.paloaltonetworks.local/browse/CIAC-4598
Description
Adding a documentation on how to manage a pull request in CI/CD playbook, how to create and update a pull request in this option.
Screenshots
Paste here any images that will help the reviewer

@Shellyber @bakatzir Doc review completed.

Thanks for all the details in the documentation! Very helpful

@Shellyber Updated doc as discussed.

authors: RotemAmit ShirleyDenkberg nericksen

repo: demisto/content-docs


text:
Update playbook-contributions.md

A few changes to correct grammar

Status
Ready/In Progress/In Hold (Reason for hold)
Related Issues
fixes: link to the issue
Description
A few sentences describing the overall goals of the pull request's commits.
Screenshots
Paste here any images that will help the reviewer

changes have been confirmed and need to be merged.


text:
Cannot get report by ID with latest report data via HTTP, just the form UI constructs of the report without data

Hello,
I have tried a few variations of this as I cannot leverage python for this activity. When I attempt this as specified, I never get the report contents returned in the query. How do I get the reports contents returned to me in an API HTTP call without python?
I am attempting to call the Rest API to pull a reports contents, any report will be fine so long as I can see and parse its contents.
When I call https://(xsoar-servername/reports using the following https://github.com/demisto/demisto-py/blob/master/docs/DefaultApi.md#download_latest_report, I seem to see all the form UI elements returned but not the report itself, just the name of the report. I can also drill down visually into areas with records like, decoder, recipients and sections, each with more records to drill down into like layout, query and automation. However, nowhere can I visually get to the report contents itself. I can even see the headers I have chosen for the report but nowhere is actual report.
I need to perform this request all within an HTTP request and cannot use cURL. If I can be sure "the latest" report always has the same file ID, I'm more than happy to bypass the reports API and go right to the download file API, however I am not sure what the report would be under that context.
TL;DR BTW, this is Microsoft PowerBI shown. Notice I am already authorized to the source URI and navigated into the data after converting it to a table. Typically, there is a record somewhere in a URL like this, which contains the actual report contents, from which I would break a table back out into and navigate, transform appropriately. When looking at PAN XSOAR, there doesn't appear to be a means to get to the report itself, even when specifying the ID of the report, it still doesn't contain the actual contents.

@RUXperienced This API in question uses the /report/:id/latest endpoint, which should return the full file.
Please see if you're able to run this API directly.
If you have further issues, please contact customer support, and they'll further assist.

authors: DeanArbel RUXperienced

repo: demisto/demisto-py


text:
fixed search by providing period

Preview: https://democracy-tailwind-44xmnd2gn-appinteractive.vercel.app/vergangen/

🫡

authors: appinteractive visionmakery

repo: demokratie-live/desktop


text:
Write source maps for .mjs

Closes demurgos/turbo-gulp#84

Codecov Report

Merging #86 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #86   +/-   ##
=======================================
  Coverage   66.88%   66.88%
=======================================
  Files          38       38
  Lines         912      912
  Branches       99       99
=======================================
  Hits          610      610
  Misses        232      232
  Partials       70       70

Impacted Files
Coverage Δ

src/lib/target-tasks/build-typescript.ts
62.74% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2f375c2...8d4a56e. Read the comment docs.

authors: codecov-io demurgos

repo: demurgos/turbo-gulp


text:
feat: Block reference support in HTML

This pull request adds support for block references in HTML. It includes support for:

Single block anchor references ![[note#^anchor]]
Range block anchor references ![[note#^anchor1:#^anchor2]]
Same file block references ![[#^anchor]] and ![[#header]]

Range block anchor references are allowed to be mixed with headers too, so it's possible to do ![[#header:#^anchor]] or ![[#^anchor:#header]] to reference only a part of a section.
It's also possible to reference single items in lists, or parts of lists. Consider the following list
* Aut id architecto quia.
* Sapiente sed accusamus eum.
  * Ullam optio est quia. ^one
* Reprehenderit doloribus.
  * Sint minus fuga omnis non. ^two
  * Iure neque alias dolorem. ^three

The reference ![[#^one]] would only reference the list element it's next to. ![[#^two:#^three]] would reference both elements, and anything in between.
An interesting case is something like ![[#^one:#^two]] where the referenced items are nested in different levels of the list. In this case, the reference refers to anything after the start anchor and before the end anchor, and also the parent of the start anchor up to a common ancestor with the end. In other words, the output looks like:
* Sapiente sed accusamus eum.
  * Ullam optio est quia. ^one
* Reprehenderit doloribus.
  * Sint minus fuga omnis non. ^two

This allows you to refer to regions of lists across different depths while preserving the required context for the items you selected within the region, and avoids ending up with a broken list.

@kevinslin Added the tests you suggested, and caught a bug in the process.

@kevinslin Done, thanks!

repo: dendronhq/dendron


text:
Sponsored Brands Version 4 Update

Updating the Sponsored Brands part of the API to the recent version 4.
New changes :
campaigns_v4 : creating, editing, deleting and listing campaigns in accordance with version 4.
ad_groups_v4 :  creating, editing, deleting and listing ad groups in accordance with version 4.
ads_v4 :  creating, editing, deleting and listing ads in accordance with version 4.
Documentation to each method and function is available.
For more information, check the amazon ads documentation : https://advertising.amazon.com/API/docs/en-us/sponsored-brands/3-0/openapi/prod#/Campaigns

Hi Hamza, thank you very much for your contribution.
I made some minor changes to follow the method convention of the api:
As the class for Sponsored Brands V4 endpoints it contains itself the V4, I removed the "_v4" from the class methods as is redundant:
So your method create_campaigns_v4 is just create_campaigns as you always could call:
result = sponsored_brands.CampaignsV4(debug=True).list_campaigns()
or
result = sponsored_brands.Campaigns(debug=True).list_campaigns()
I also added a decorator (deprecated) to show the clients if there is a new method in V4 to let them know if they want migrate to the new endpoint:
So if you call:
result = sponsored_brands.Campaigns(debug=True).list_campaigns() you will get a warning:
WARNING:This function Campaigns.list_campaigns() is deprecated >> Use instead /sb/v4/campaigns/list - CampaignsV4.list_campaigns()
Other minor changes are the method naming convention I use that if the endpoint attacks only one entity I use singular (example: get_ad_group(self, adGroupId, **kwargs)) and if you could attack one or more entities I used plural (example: list_campaigns(self, version: int = 4, **kwargs)).
I saw in the v4 endpoint for delete_campaigns amazon documentation:
The operation is currently not supported subject to potential change.
So I added a deprecation warning that we could remove later when the endpoint is stable, as apparently it could allow to remove several campaigns the method name is delete_campaigns instead of the method delete_campaign in the previous version that only allow to delete/archive one campaign by campaignId.
I also added the new docs to the python amazon ad api documentation
Thanks again for your help and hope that minor changes do not dissapoint you.
Please upgrade to version v0.3.9 and test if everything is fine for you.
Kindest Regards,
Daniel.

Hi Daniel,
Thanks for checking the changes so quickly! Glad to be of help. Thanks also for adding the changes I didn't think of.
I will test the functionalities and if something isn't right I will get back to you.
Kind Regards,
Hamza

authors: HamzaChx denisneuf

repo: denisneuf/python-amazon-ad-api


text:
Subscribe, unsubscribe, resubscribe

I am getting odd errors when my server:

creates a subscription to a specific key subscribeClient.subscribe(key).then(async (subscription:any) => {
listens for messages
breaks out of the for await (const { channel, message } of subscription.receive()) { loop
? do I unsubscribe here? It doesn't seem to help
subscribes again to the same key

this subscription usually doesn't return

There is something odd going on with subscribe and unsubscribe (or just exiting the subscription for await loop)
What is the correct expected pattern to subscribe->unsubscribe->subscribe again to the same key?
Thank you!

It might be related to https://github.com/denodrivers/redis/issues/390

Thanks for reporting the issue @dionjwa!

? do I unsubscribe here? It doesn't seem to help

No, it is not unsubscribed if it just exits the loop.

What is the correct expected pattern to subscribe->unsubscribe->subscribe again to the same key?

The following code seemed to work properly. If possible, could you share the code that is causing the problem? I would like to look into it.
import { connect } from "./redis.ts";
import { assertEquals } from "./vendor/https/deno.land/std/assert/mod.ts";

const opts = {
  hostname: "127.0.0.1",
  port: 6379
};
const redis = await connect(opts);
const pub = await connect(opts);
const channel = "foobar";
const sub = await redis.subscribe(channel);
const messages = [];
try {
  const promise = (async () => {
    for await (const m of sub.receive()) {
      messages.push(m);
      break;
    }
  })();
  await pub.publish(channel, "message");

  await promise;
  await sub.unsubscribe(channel);
  await sub.subscribe(channel);

  const promise2 = (async () => {
    for await (const m of sub.receive()) {
      messages.push(m);
      break;
    }
  })();
  await pub.publish(channel, "message 2");

  await promise2;
} finally {
  sub.close();
  redis.close();
  pub.close();
}
assertEquals(messages, [
  { channel, message: "message" },
  { channel, message: "message 2" }
]);

P.S. I believe #390 has been resolved in v0.32.1. Maybe this problem has also been resolved.

Yes, I am not noticing this issue anymore, this issue can be closed.

repo: denodrivers/redis