text:
Added automatic requirements installation from docstring

This makes it so that the requirements in the python docstring get installed before executing the pipeline. Fixes #151 .
The process goes like:

load python file
parse using ast
get the docstring from ast
find the requirements section in the docstring using regex (re)
create a process using subprocess.Popen and pip install the dependencies

Everything gets done before the script executes for the first time.
To test this:

first run pip uninstall numpy
then add this test.py to your pipelines directory or upload using swagger UI (http://localhost:9099/docs)

"""
title: Test pipline
requirements: numpy
"""

from typing import List, Union, Generator, Iterator
import numpy

class Pipeline:
    def __init__(self):
        print(numpy)

    def pipe(
        self, user_message: str, model_id: str, messages: List[dict], body: dict
    ) -> Union[str, Generator, Iterator]:
        return "hello world"

Looks nice. How does it behave in case a dependency isn't found or some other error?
Sidenote: there is the uv pip drop-in alternative which is (without exaggeration) x10 faster, if there would be an option to use it instead, that would be awesome. Though make it configurable might be a bit extra work.

Looks nice. How does it behave in case a dependency isn't found or some other error?

it doesnt care. notice line 130 one could change the _, _ = ... to stdout, stderr = ... and get the output or get the return code from the process but i figured that we might as well try to run the script anyway and see if it fails (right now it just runs the script with missing dependencies aswell).

Sidenote: there is the uv pip drop-in alternative which is (without exaggeration) x10 faster, if there would be an option to use it instead, that would be awesome. Though make it configurable might be a bit extra work.

i never used it. we could add uv  and in the requirements.txt and then just use uv pip are there any drawbacks?

also i could remove line 84-105 and line 120-124 in start.sh as it isnt needed anymore once the pull request gets merged

authors: DarkIceXD Marenz

repo: open-webui/pipelines


text:
Typo: uuid to be updated to new uuid format in example

In OAS, /v1/list-end-points-of-link, responses :: 200 :: example :: the uuid has been present in old uuid format which needs to be updated to new uuid format.
Here, in line 3069, the operation-uuid of RO should be ro-2-0-1-op-c-im-alt-1-0-0-000 corresponds to operation-server alt-2-0-1-op-s-bm-001

Changed as proposed.

source: gharchive/issue

authors: IswaryaaS openBackhaul

repo: openBackhaul/ApplicationLayerTopology

license_type: permissive

license_source: github-api


text:
PostTermination - Deep check of CC for being at initial stage checks against old release

The errors are either IP addresses not matching (which looks like the fake-ip script issue) or release numbers still being checked
for 1.0.0 despite all of them already on 2.0.1
Provides expected properties in capability class for 'eatl-2-0-1-http-s-000' |
expected '[{"local-id":"0","release-number":"2.0.1","release-date":"17.12.2021","changes":"Initial version."},{"local-id":"1","release-number":"2.0.1","release-date":"17.12.2022","changes":"Update on ApplicationPattern 2.0.1."}]'
to equal '[{"local-id":"0","release-number":"1.0.0","release-date":"17.12.2021","changes":"Initial version."},{"local-id":"1","release-number":"2.0.1","release-date":"17.12.2022","changes":"Update on ApplicationPattern 2.0.1."}]'
Provides expected properties in configuration class for 'eatl-2-0-1-http-c-es-1-0-0-000' | expected '2.0.1' to equal '1.0.0'
Provides expected properties in configuration class for 'eatl-2-0-1-http-c-or-1-0-0-000' | expected '2.0.1' to equal '1.0.0'
Provides expected properties in configuration class for 'eatl-2-0-1-http-c-tar-1-0-0-000' | expected '2.0.1' to equal '1.0.0'
Provides expected properties in configuration class for 'eatl-2-0-1-http-c-ol-1-0-0-000' | expected '2.0.1' to equal '1.0.0'
Provides expected properties in configuration class for 'eatl-2-0-1-http-c-aa-1-0-0-000' | expected '2.0.1' to equal '1.0.0'
Provides expected properties in configuration class for 'eatl-2-0-1-http-c-alt-1-0-0-000' | expected '2.0.1' to equal '1.0.0'
Provides expected properties in configuration class for 'eatl-2-0-1-http-c-okm-1-0-0-000' | expected '2.0.1' to equal '1.0.0'

Shall be fixed in https://github.com/openBackhaul/ExecutionAndTraceLog/pull/322. Kindly reopen if the issue reoccurs

authors: DanaSunal IswaryaaS

repo: openBackhaul/ExecutionAndTraceLog


text:
The query with limit does not return the lowest tag points

Describe the bug（Bug 描述）
The query with limit does not return the lowest tag points
To Reproduce（Bug 复现步骤）
1.insert such datas:

2.query "select * from mst1 limit 1"
3.result point is not the lowest.

Expected behavior（期望结果）
The query with limit should return the sorted tag points of the equal minimum time points.
Screenshots（屏幕截图）
No response
Logs（完整的错误日志）
No response
Additional context（其他的一些补充内容）
No response

In standard SQL, a SELECT statement without ORDER BY may return the rows in any order.
Did openGemini design and document that the rows are ordered by the timestamp in ascending order,
even when there is no "ORDER BY ts" clause?

In standard SQL, a SELECT statement without ORDER BY may return the rows in any order. Did openGemini design and document that the rows are ordered by the timestamp in ascending order, even when there is no "ORDER BY ts" clause?
Yes, queries without “ORDER BY” clause in openGemini return data sorted in ascending time by default.

authors: lihanxue zedware

repo: openGemini/openGemini


text:
Beautify Code

https://github.com/openGemini/opengemini-client-python/blob/c96d4eb1f6f43460d88aef27ccae8b7681f8abe5/opengemini_client/client_impl.py#L24-L30
Suggest compare with AuthType.PASSWORD and AuthType.TOKEN. Not 0 or 1. To improve readability

Fix by #51

repo: openGemini/opengemini-client-python


text:
Sentry Relay: Configure HTTP/3 with nginx

According to nginx quic demo, the latest nginx version supports QUIC and HTTP/3. As we are already running the latest nginx mainline version, we can make use of it 😁.
Let's add HTTP/3 support for our nginx Ansible role.

Thank you for bringing this up! I learned a lot about the new protocols :student:
See codeocean-terraform!128

After resolving some issues with the unattended-upgrades (to get the latest version of nginx, that fixes an important bug related to the HTTP/3 TLS handshake), we have HTTP/3 working reliably 🥳.

repo: openHPI/poseidon

license: MIT


text:
Clayton County, GA

http://weba.co.clayton.ga.us/arcgis/rest/services/TaxAssessor/Parcels_Address/MapServer/0

Are the parcels preferable to the address points added in #2592 ?

Nope, the address points we have are preferable. Let's close this for now.

authors: chdean iandees justinelliotmeyers

repo: openaddresses/openaddresses


text:
SubProcVecEnv raises  ConnectionResetError: [Errno 104] Connection reset by peer

I'm trying to run the following code and test PPO with Sonic the hedghehog, running it in parralel with SubProcVecEnv
Unfortunately I run in the following error:
Traceback (most recent call last):

  File "<ipython-input-2-9712559f6750>", line 3, in <module>
    env = SubprocVecEnv([foo(game,state) for _ in range(num_cpu)])

  File "/home/chryskan/anaconda3/lib/python3.7/site-packages/stable_baselines/common/vec_env/subproc_vec_env.py", line 96, in __init__
    observation_space, action_space = self.remotes[0].recv()

  File "/home/chryskan/anaconda3/lib/python3.7/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()

  File "/home/chryskan/anaconda3/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)

  File "/home/chryskan/anaconda3/lib/python3.7/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)

ConnectionResetError: [Errno 104] Connection reset by peer

This is the code:
from stable_baselines.common.policies import MlpPolicy, CnnPolicy
from stable_baselines.common import make_vec_env
from stable_baselines import PPO2
from my_wrappers import wrap_sonic, make_sonic
from stable_baselines.common.env_checker import check_env
from stable_baselines.common.vec_env import SubprocVecEnv`

def foo(game, state):
        def bar():
            return wrap_sonic(make_sonic(game, state), episode_life=False,
                  frame_stack=True, scale=True,
                  reward_scale = True, sonic_actions=True,
                  max_x_reward=True)
        return bar

game , state = "SonicTheHedgehog-Genesis", "GreenHillZone.Act1"
num_cpu = 4
env = SubprocVecEnv([foo(game,state) for _ in range(num_cpu)])

model = PPO2(CnnPolicy, env, verbose=1)
model.learn(total_timesteps=1000_000)

And the my_wrappers module
import numpy as np
from collections import deque
import gym
from gym import spaces
import cv2
from retro_contest.local import make

height, width = 84, 84

class ActionsDiscretizer(gym.ActionWrapper):
    def __init__(self, env):
        super(ActionsDiscretizer, self).__init__(env)
        buttons = ["B", "A", "MODE", "START", "UP", "DOWN", "LEFT",
                   "RIGHT", "C", "Y", "X", "Z"]
        actions = [['LEFT'], ['RIGHT'], ['LEFT', 'DOWN'],
                   ['RIGHT', 'DOWN'], ['DOWN'],
                   ['DOWN', 'B'], ['B']]
        self._actions = []

        for action in actions:
            arr = np.array([False] * 12)
            for button in action:
                arr[buttons.index(button)] = True
            self._actions.append(arr)
        self.action_space = gym.spaces.Discrete(len(self._actions))

    def action(self, a):
        return self._actions[a].copy()

class RewardScaler(gym.RewardWrapper):
    """
    Bring rewards to a reasonable scale for PPO.
    This is incredibly important and effects performance
    drastically.
    """
    def reward(self, reward):
        return reward * 0.01

class AllowBacktracking(gym.Wrapper):
    """
    Use deltas in max(X) as the reward, rather than deltas
    in X. This way, agents are not discouraged too heavily
    from exploring backwards if there is no way to advance
    head-on in the level.
    """
    def __init__(self, env):
        super(AllowBacktracking, self).__init__(env)
        self._cur_x = 0
        self._max_x = 0

    def reset(self, **kwargs):
        self._cur_x = 0
        self._max_x = 0
        return self.env.reset(**kwargs)

    def step(self, action):
        obs, rew, done, info = self.env.step(action)
        self._cur_x += rew
        rew = max(0, self._cur_x - self._max_x)
        self._max_x = max(self._max_x, self._cur_x)
        return obs, rew, done, info

class NoopResetEnvSonic(gym.Wrapper):
    def __init__(self, env, noop_max=30):
        """Sample initial states by taking random number of no-ops on reset.
        No-op is assumed to be action 0.
        """
        gym.Wrapper.__init__(self,env)
        self.noop_max = noop_max
        self.override_num_noops = None
        self.noop_action = np.zeros(env.action_space.n)

    def reset(self, **kwargs):
         """ Do no-op action for a number of steps in [1, noop_max]."""
         self.env.reset(**kwargs)
         if self.override_num_noops is not None:
             noops = self.override_num_noops
         else:
            noops = self.unwrapped.np_random.randint(1, self.noop_max+1)
         assert noops > 0
         obs = None
         for _ in range(noops):
             obs, _, done, _ = self.env.step(self.noop_action)
             if done:
                 obs = self.env.reset(**kwargs)
         return obs

class MaxAndSkipEnv(gym.Wrapper):
    def __init__(self, env, skip=4):
        """Return only every `skip`-th frame"""
        gym.Wrapper.__init__(self, env)
        # most recent raw observations (for max pooling across time steps)
        self._obs_buffer = np.zeros((2,)+env.observation_space.shape, dtype='uint8')
        self._skip = skip

    def step(self, action):
        """Repeat action, sum reward, and max over last observations."""
        total_reward = 0.0
        done = None
        for i in range(self._skip):
            obs, reward, done, info = self.env.step(action)
            if i==self._skip - 2: self._obs_buffer[0] = obs
            if i==self._skip - 1: self._obs_buffer[1] = obs
            total_reward += reward
            if done:
                break
        # Note that the observation on the done=True frame
        # doesn't matter
        max_frame = self._obs_buffer.max(axis=0)

        return max_frame, total_reward, done, info

class FireResetEnv(gym.Wrapper):
    def __init__(self, env):
        """Take action on reset for environments that are fixed until firing."""
        gym.Wrapper.__init__(self, env)
        assert env.unwrapped.get_action_meanings()[1] == 'FIRE'
        assert len(env.unwrapped.get_action_meanings()) >= 3

    def reset(self, **kwargs):
        self.env.reset(**kwargs)
        obs, _, done, _ = self.env.step(1)
        if done:
            self.env.reset(**kwargs)
        obs, _, done, _ = self.env.step(2)
        if done:
            self.env.reset(**kwargs)
        return obs

class EpisodicLifeEnv(gym.Wrapper):
    def __init__(self, env):
        """Make end-of-life == end-of-episode, but only reset on true game over.
        Done by DeepMind for the DQN and co. since it helps value estimation.
        """
        gym.Wrapper.__init__(self, env)
        self.lives = 0
        self.was_real_done  = True

    def step(self, action):
        obs, reward, done, info = self.env.step(action)
        self.was_real_done = done
        # check current lives, make loss of life terminal,
        # then update lives to handle bonus lives
        lives = self.env.unwrapped.ale.lives()
        if lives < self.lives and lives > 0:
            # for Qbert somtimes we stay in lives == 0 condtion for a few frames
            # so its important to keep lives > 0, so that we only reset once
            # the environment advertises done.
            done = True
        self.lives = lives
        return obs, reward, done, info

    def reset(self, **kwargs):
        """Reset only when lives are exhausted.
        This way all states are still reachable even though lives are episodic,
        and the learner need not know about any of this behind-the-scenes.
        """
        if self.was_real_done:
            obs = self.env.reset(**kwargs)
        else:
            # no-op step to advance from terminal/lost life state
            obs ,_, _, _ = self.env.step(0)
        self.lives = self.env.unwrapped.ale.lives()
        return obs

class ClipRewardEnv(gym.RewardWrapper):
    def reward(self, reward):
        """Bin reward to {+1, 0, -1} by its sign."""
        return np.sign(reward)

class WarpFrame(gym.ObservationWrapper):
    def __init__(self, env):
        """Warp frames to 84x84 as done in the Nature paper and later work."""
        gym.ObservationWrapper.__init__(self, env)
        self.width = width
        self.height = height
        self.observation_space = spaces.Box(low=0, high=255, shape=(self.height, self.width, 1), dtype=np.uint8)

    def observation(self, frame):
        frame = cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)
        frame = cv2.resize(frame, (self.width, self.height), interpolation=cv2.INTER_AREA)
        return frame[:, :, None]

class FrameStack(gym.Wrapper):
    def __init__(self, env, k):
        """Stack k last frames.
        Returns lazy array, which is much more memory efficient.
        See Also
        --------
        baselines.common.atari_wrappers.LazyFrames
        """
        gym.Wrapper.__init__(self, env)
        self.k = k
        self.frames = deque([], maxlen=k)
        shp = env.observation_space.shape
        self.observation_space = spaces.Box(low=0, high=255, shape=(shp[0], shp[1], shp[2] * k), dtype=np.uint8)

    def reset(self):
        ob = self.env.reset()
        for _ in range(self.k):
            self.frames.append(ob)
        return self._get_ob()

    def step(self, action):
        ob, reward, done, info = self.env.step(action)
        self.frames.append(ob)
        return self._get_ob(), reward, done, info

    def _get_ob(self):
        assert len(self.frames) == self.k
        return np.reshape(self.frames, newshape=self.observation_space.shape)

class ScaledFloatFrame(gym.ObservationWrapper):
    def observation(self, observation):
        # careful! This undoes the memory optimization, use
        # with smaller replay buffers only.
        return np.array(observation).astype(np.float32) / 255.0

class LazyFrames(object):
    def __init__(self, frames):
        """This object ensures that common frames between the observations are only stored once.
        It exists purely to optimize memory usage which can be huge for DQN's 1M frames replay
        buffers.
        This object should only be converted to numpy array before being passed to the model.
        You'd not believe how complex the previous solution was."""
        self._frames = frames

    def __array__(self, dtype=None):
        out = np.concatenate(self._frames, axis=2)
        if dtype is not None:
            out = out.astype(dtype)
        return out

def make_sonic(game, state):
    env = make(game=game, state=state)
    #env = retrowrapper.RetroWrapper(game=game, state=state)
    env = NoopResetEnvSonic(env, noop_max=30)
    env = MaxAndSkipEnv(env, skip=4)
    return env

def wrap_sonic(env, episode_life=True, clip_rewards=False,
                  frame_stack=False, scale=False, pytorch_img=False,
                  reward_scale = True, sonic_actions=True, max_x_reward=True):
    if episode_life:
        env = EpisodicLifeEnv(env)

    key = 'get_action_meanings'
    if hasattr(env.unwrapped, key):
        meanings = env.unwrapped.get_action_meanings()
    else:
        meanings = [env.unwrapped.get_action_meaning(np.zeros(a))
                        for a in range(env.action_space.n)]
    if 'FIRE' in meanings:
        env = FireResetEnv(env)
    env = WarpFrame(env)
    if scale:
        env = ScaledFloatFrame(env)
    if max_x_reward:
        env=AllowBacktracking(env)
    if sonic_actions:
        env=ActionsDiscretizer(env)
    if clip_rewards:
        env = ClipRewardEnv(env)
    if reward_scale:
        env = RewardScaler(env)
    if frame_stack:
        env = FrameStack(env, 4)
    return env

I was facing similar issues, but I solved it by removing verbose. Try removing verbose =1 , from model = PPO2(CnnPolicy, env, verbose=1).

still, struggling in this issues, does anyone help me out?

Same problem. Does this have an answer now?

Same

Facing the same problem. Is anyone having a solution to this ?

/root/miniconda3/lib/python3.7/multiprocessing/connection.py:379: ConnectionResetError
----------------------------------------------------------------------------- Captured stderr call ------------------------------------------------------------------------------
Process SpawnProcess-2:
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/root/miniconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/root/virtualenv/venv/baselines/baselines/common/vec_env/subproc_vec_env.py", line 15, in worker
    envs = [env_fn_wrapper() for env_fn_wrapper in env_fn_wrappers.x]
  File "/root/virtualenv/venv/baselines/baselines/common/vec_env/subproc_vec_env.py", line 15, in <listcomp>
    envs = [env_fn_wrapper() for env_fn_wrapper in env_fn_wrappers.x]
  File "/root/virtualenv/venv/baselines/baselines/common/tests/test_env_after_learn.py", line 15, in make_env
    env = gym.make('CartPole-v1' if algo == 'acktr' else 'PongNoFrameskip-v4')
  File "/root/miniconda3/lib/python3.7/site-packages/gym/envs/registration.py", line 156, in make
    return registry.make(id, **kwargs)
  File "/root/miniconda3/lib/python3.7/site-packages/gym/envs/registration.py", line 101, in make
    env = spec.make(**kwargs)
  File "/root/miniconda3/lib/python3.7/site-packages/gym/envs/registration.py", line 73, in make
    env = cls(**_kwargs)
  File "/root/miniconda3/lib/python3.7/site-packages/gym/envs/atari/atari_env.py", line 49, in __init__
    self.game_path = atari_py.get_game_path(game)
  File "/root/miniconda3/lib/python3.7/site-packages/atari_py/games.py", line 20, in get_game_path
    raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,))
Exception: ROM is missing for pong, see https://github.com/openai/atari-py#roms for instructions
__________________________________________________________________________ test_env_after_learn[deepq] __________________________________________________________________________

algo = 'deepq'

    @pytest.mark.parametrize('algo', algos)
    def test_env_after_learn(algo):
        def make_env():
            # acktr requires too much RAM, fails on travis
            env = gym.make('CartPole-v1' if algo == 'acktr' else 'PongNoFrameskip-v4')
            return env

        make_session(make_default=True, graph=tf.Graph())
>       env = SubprocVecEnv([make_env])

baselines/common/tests/test_env_after_learn.py:19:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
baselines/common/vec_env/subproc_vec_env.py:71: in __init__
    observation_space, action_space, self.spec = self.remotes[0].recv().x
/root/miniconda3/lib/python3.7/multiprocessing/connection.py:250: in recv
    buf = self._recv_bytes()
/root/miniconda3/lib/python3.7/multiprocessing/connection.py:407: in _recv_bytes
    buf = self._recv(4)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <multiprocessing.connection.Connection object at 0x7f4be43769b0>, size = 4, read = <built-in function read>

    def _recv(self, size, read=_read):
        buf = io.BytesIO()
        handle = self._handle
        remaining = size
        while remaining > 0:
>           chunk = read(handle, remaining)
E           ConnectionResetError: [Errno 104] Connection reset by peer

/root/miniconda3/lib/python3.7/multiprocessing/connection.py:379: ConnectionResetError
----------------------------------------------------------------------------- Captured stderr call

I noticed this error:
 raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,))

When I solved this error, ConnectionResetError: [Errno 104] Connection reset by peer disappeared.But there is one error left
===================================================== 46 failed, 49 passed, 31 skipped, 1111 warnings in 110.05s (0:01:50) ======================================================

The error becomes:
===================================================== 46 failed, 49 passed, 31 skipped, 1111 warnings in 110.05s (0:01:50) ======================================================

baselines/common/tests/test_doc_examples.py:20: in <lambda>
    venv = DummyVecEnv([lambda: cmd_util.make_mujoco_env('Reacher-v2', seed=0)])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

env_id = 'Reacher-v2', seed = 0, reward_scale = 1.0

    def make_mujoco_env(env_id, seed, reward_scale=1.0):
        """
        Create a wrapped, monitored gym.Env for MuJoCo.
        """
>       rank = MPI.COMM_WORLD.Get_rank()
E       AttributeError: 'NoneType' object has no attribute 'COMM_WORLD'

baselines/common/cmd_util.py:112: AttributeError

My environment(for your information):
┌──(root💀agi)-[~]
└─# uname -a
Linux agi 5.14.0-kali4-amd64 #1 SMP Debian 5.14.16-1kali1 (2021-11-05) x86_64 GNU/Linux
┌──(venv)─(root💀agi)-[~/virtualenv/venv/baselines]
└─# pip list                                                                                130 ⨯
Package                Version    Location

absl-py                1.0.0
asn1crypto             0.24.0
astor                  0.8.1
atari-py               0.2.9
attrs                  21.2.0
baselines              0.1.6      /root/virtualenv/venv/baselines
box2d-py               2.3.8
cached-property        1.5.2
certifi                2021.10.8
cffi                   1.12.2
chardet                3.0.4
click                  8.0.3
cloudpickle            1.2.2
conda                  4.10.3
conda-package-handling 1.7.3
cryptography           2.6.1
cycler                 0.11.0
Cython                 0.29.26
filelock               3.4.0
fonttools              4.28.5
future                 0.18.2
gast                   0.5.3
glfw                   2.5.0
google-pasta           0.2.0
grpcio                 1.43.0
gym                    0.15.7
h5py                   3.6.0
idna                   2.8
imageio                2.13.4
importlib-metadata     4.10.0
iniconfig              1.1.1
joblib                 1.1.0
Keras-Applications     1.0.8
Keras-Preprocessing    1.1.2
kiwisolver             1.3.2
lockfile               0.12.2
Markdown               3.3.6
matplotlib             3.5.1
mujoco-py              1.50.1.68
numpy                  1.21.5
opencv-python          4.5.4.60
packaging              21.3
pandas                 1.3.5
Pillow                 8.4.0
pip                    19.0.3
pluggy                 1.0.0
protobuf               3.19.1
py                     1.11.0
pycosat                0.6.3
pycparser              2.19
pyglet                 1.5.0
pyOpenSSL              19.0.0
pyparsing              3.0.6
PySocks                1.6.8
pytest                 6.2.5
python-dateutil        2.8.2
pytz                   2021.3
requests               2.21.0
ruamel-yaml            0.15.46
scipy                  1.7.3
setuptools             41.0.0
six                    1.12.0
tensorboard            1.14.0
tensorflow             1.14.0
tensorflow-estimator   1.14.0
termcolor              1.1.0
toml                   0.10.2
tqdm                   4.62.3
typing-extensions      4.0.1
urllib3                1.24.1
Werkzeug               2.0.2
wheel                  0.33.1
wrapt                  1.13.3
youtube-dl             2021.12.17
zipp                   3.6.0

same error...

same error

same issue and i know it's related to my customed env bc change to another env the error won't occur

for the " 'NoneType' object has no attribute 'COMM_WORLD' " error, solve it using 'pip install mpi4py==4.0.1'.
Just a lack of dependency module error.

authors: Abhimanyu8713 HongminWu MakisKans Mickeyyyang Open-AGI PostResPerditas Vaillus ZhaoRunyi bamasa cmal harsh-ux

repo: openai/baselines

license: mit

license_source: bigquery


text:
How to pretrain the model?

could u share the pretrain code?

same question

repo: openai/finetune-transformer-lm


text:
CMake cannot copy output executable due to missing executables

Issue summary
I am getting this error when I install gym-retro from pip:
log1.txt
System information

Windows 10 x64
Python 3.6.3
0.6.0

I think something is wrong with your CMake, but it's...very hard to tell. Please upload the error log too.

Ran pip install setup.py after extracting tarball. This is the error log:
CMakeError.log

Welp, I got it working on Ubuntu. If I find a proper solution then I will post it here.

authors: BhayanakMoth endrift

repo: openai/retro


text:
Configurable flags for the backend compiler

Hi! I see that openai/triton requires a working toolchain at run-time, including a CUDAToolkit and libpython installations for the host platform. Currently, triton attempts to guess the correct compiler flags on its own: https://github.com/openai/triton/blob/deb2c71fb4f912a5298003fa3fc789885b726607/python/triton/common/build.py#L77-L82
This includes inferring the library locations: https://github.com/openai/triton/blob/deb2c71fb4f912a5298003fa3fc789885b726607/python/triton/common/build.py#L19-L22
What this means, in practice, is that openai/triton is taking on a job that is usually performed by tools like CMake, and that certain care is to be taken when deploying openai/triton. The current flag inference logic is platform-specific and, of course, it isn't expected to be universal either. But we probably should work out a solution on how to make it configurable, so that e.g. distributions can set up their environments to meet triton's expectations.
Some concrete examples of issues that arise:

On NixOS the libcuda.so user-space driver is deployed in a special location, /run/opengl-driver/lib, and whereis wouldn't produce any reasonable output because /lib and /usr/lib do not exit. In https://github.com/NixOS/nixpkgs/pull/222273 we end up patching triton/compiler.py to pass the correct -L flag to the compiler: https://github.com/NixOS/nixpkgs/blob/e4474334415ac41efb5fda33d4cc8f312397ef05/pkgs/development/python-modules/openai-triton/default.nix#L128-L147. We also have to work around triton trying to vendor a copy of ptxas
In pytorch/pytorch there is a number of confused issues about broken -lcuda and  #include <Python.h>

https://github.com/pytorch/pytorch/issues/97008
https://github.com/pytorch/pytorch/issues/92745
https://github.com/pytorch/pytorch/issues/97146
https://github.com/pytorch/pytorch/issues/97501

An off-the-shelf way of making libpython and cuda flags configurable would be pkg-config, although I'd feel weird and conflicted about setting up pkg-config at run-time side by side with pytorch. I also note that this situation is somewhat similar to that of torch.utils.cpp_extension, which also attempts to guess build flags at run-time and usually gets them wrong

Extra: a configurable path to ptxas is desirable. Maybe even just accepting ptxas from $PATH?

Update: I also notice that pytorch seems to have tried vendoring ptxas at a point, and ceased doing so since triton became their dependency. This is probably good: if pytorch just asks some function from triton to report the location of ptxas, then by making ptxas optional and configurable in triton, we'd have addressed the issue in pytorch as well
The ptxas discussion can be moved to https://github.com/openai/triton/issues/1618

FWIW, I met the related issue when I use triton (2.2.0) in a conda environment. The cuda toolkit is installed in the conda env (rather than in the system), so the compiler can't find the cuda library default linking path. Below is the stacktrace
/usr/bin/ld: skipping incompatible /usr/lib/i386-linux-gnu/libcuda.so when searching for -lcuda
/usr/bin/ld: skipping incompatible /usr/lib/i386-linux-gnu/libcuda.so when searching for -lcuda
/usr/bin/ld: cannot find -lcuda
collect2: error: ld returned 1 exit status
Traceback (most recent call last):
  File "/home/jacobz/Downloads/01-vector-add.py", line 87, in <module>
    output_triton = add(x, y)
  File "/home/jacobz/Downloads/01-vector-add.py", line 73, in add
    add_kernel[grid](x, y, output, n_elements, BLOCK_SIZE=1024)
  File "/home/jacobz/.conda/envs/lmdeploy-build/lib/python3.10/site-packages/triton/runtime/jit.py", line 532, in run
    self.cache[device][key] = compile(
  File "/home/jacobz/.conda/envs/lmdeploy-build/lib/python3.10/site-packages/triton/compiler/compiler.py", line 614, in compile
    so_path = make_stub(name, signature, constants, ids, enable_warp_specialization=enable_warp_specialization)
  File "/home/jacobz/.conda/envs/lmdeploy-build/lib/python3.10/site-packages/triton/compiler/make_launcher.py", line 37, in make_stub
    so = _build(name, src_path, tmpdir)
  File "/home/jacobz/.conda/envs/lmdeploy-build/lib/python3.10/site-packages/triton/common/build.py", line 107, in _build
    ret = subprocess.check_call(cc_cmd)
  File "/home/jacobz/.conda/envs/lmdeploy-build/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/gcc', '/tmp/tmpvdufqg5s/main.c', '-O3', '-I/home/jacobz/.conda/envs/lmdeploy-build/lib/python3.10/site-packages/triton/common/../third_party/cuda/include', '-I/home/jacobz/.conda/envs/lmdeploy-build/include/python3.10', '-I/tmp/tmpvdufqg5s', '-shared', '-fPIC', '-L/home/jacobz/.conda/envs/lmdeploy-build/targets/x86_64-linux/lib/stubs', '-lcuda', '-o', '/tmp/tmpvdufqg5s/add_kernel.cpython-310-x86_64-linux-gnu.so', '-L/usr/lib/x86_64-linux-gnu', '-L/usr/lib/i386-linux-gnu', '-L/usr/lib/i386-linux-gnu']' returned non-zero exit status 1.

I need to pass the cuda library to the compiler, and what you need is to add the cuda library path (for me it's <conda env root>/lib/stubs) to the compiler flags here: https://github.com/triton-lang/triton/blob/v2.1.0/python/triton/common/build.py#L89

authors: SomeoneSerge cmpute

repo: openai/triton


text:
[AutoPR @azure-rest/iot-device-update] Merge deviceupdate

Create to sync https://github.com/openapi-env-test/azure-rest-api-specs/pull/3066

API change check for @azure/iot-device-update
API changes are not detected in this pull request for @azure/iot-device-update

API change check for @azure-rest/iot-device-update
API changes have been detected in @azure-rest/iot-device-update. You can review API changes here

source: gharchive/pull-request

repo: openapi-env-test/azure-sdk-for-js


text:
Drag and Drop wordlist not attributing Type

When you drag and drop a wordlist in BB. It automatically detect it and set it's Type.
Per example ; MailPass
While in openbullet, the Type is always Default.
Is that a normal behavior ?

If you looked carefully you would've seen this https://github.com/openbullet/openbullet/commit/32f623a3b804ef6150474842f87f9ef7af2d243e commit fixes the issue, so either recompile or get the 1.0.1 version which already has the fix inside it.

repo: openbullet/openbullet


text:
Youtube documentation is out of date

Describe the bug
The current Youtube docs say that a URL should be printed in the logs.  This no longer happens, instead a browser window is opened.  This presents an issue for those configuring on their servers.  The docs should be amended to say that the first run should occur on a local machine, and that the configs can be copied to the server after that

Thanks Greg :)
There's also a project dedicated to documentation. Just wanted to make sure this issue is correctly registered

@ypatios The markdown doc under docs/guides/admin/ is what you want to update.  The docs project is the code powering docs.opencast.org, which deploys the docs from this project's docs/guides directory.

authors: gregorydlogan ypatios

repo: opencast/opencast


text:
Choose visibility in resource upload form

When uploading resources, the visibility of those are automatically set to visible. A checkbox for choosing the visibility of the resource manually before uploading would be helpful for many users. This checkboxes (advcheckbox) default value could be set with admin_settings to be either default_visible or default_hidden.

I'm somewhat confused. Can you explain which upload form you are talking about? Event visibility is always controlled via the event's ACL which you can set.

Based on your other request not being about opencast, I assume this one is about one of the Opencast plugins as well. That's why I'm closing this. In case this is actually about Opencast, don't hesitate to re-open the issue and explain which UI you mean.

This is probably Moodle again and the change in design to letting Moodle handle access internally opens up the host of mechanisms of Moodle to handle this.

authors: colradectug lkiesow mtneug


text:
MH-13431 Remove unfinished feature "Bulk Messaging"

Opencast contains quite a bit of code to the unfinished feature "Bulk messaging". The idea behind that feature is that Opencast allows users to send e-mails to some technical contact of events.
The feature would include:

Management of e-mail templates
Management of e-mail signatures
Persistence of SMTP configurations (per tenant)
Bulk action "Send message" to actually send messages

The goal of this task is to removed that unfinished feature from the Opencast codebase.

Just checking: This is not used in any way by the email workflow operation, is it?

This is neither related to the email workflow operation nor the email template service used by that workflow operation.

Oh, just realised that the security configuration has rules related to the e-mail templates that can also be removed... will update the PR tomorrow.

@lkiesow PR updated.

authors: lkiesow staubesv


text:
Task1: Installed Kali Linux and shared the screenshot successful

Issue < #2 >
Installed Kali Linux on VMware on windows
Drive link in the text file in kanakaraju26/week1 folder

Hey @Kanakaraju26
PR Template is not followed. Kindly follow the template & edit it again for consideration as a valid pull request.
Thanks for contributing in OpenCode'24 ✨✨!

Hey @Kanakaraju26
PR Template is not followed. Kindly follow the template & edit it again for consideration as a valid pull request.
Thanks for contributing in OpenCode'24 ✨✨!

Hey @Kanakaraju26
PR Template is not followed. Kindly follow the template & edit it again for consideration as a valid pull request.
Thanks for contributing in OpenCode'24 ✨✨!

Hey @Kanakaraju26
PR Template is not followed. Kindly follow the template & edit it again for consideration as a valid pull request.
Thanks for contributing in OpenCode'24 ✨✨!

Hey @Kanakaraju26
PR Template is not followed. Kindly follow the template & edit it again for consideration as a valid pull request.
Thanks for contributing in OpenCode'24 ✨✨!


text:
Send an email to 'MEMBERS' for new expenses filed

Email needs to include

info about the expense
possibly the receipt (or a link to view the receipt)
Approve and Reject button (note it doesn't need a pay button, since that's the host's job).

See https://github.com/OpenCollective/opencollective-api/pull/551

This is a request from WWCode, let's put it in the to do list

this happened.

authors: asood123 piamancini xdamman

repo: opencollective/opencollective


text:
Add transaction details on expense page

Initially suggested by @kewitz
We should display transactions information on the new expense page (once it's paid).
The best place to do that would be the activity, so maybe we should attach the public transaction information directly to the activity.
I suggest displaying something similar to what we have on the collective page:

Transaction details
$55.00 - $0.97 (-1.76% payment processor fee)

It should display:

Total amount
Host fees (if any)
Payment processor fees  (if any)

Good idea!
(By the way, the 'expense paid' action should also include the person who paid it - that's missing from your screenshot).

(By the way, the 'expense paid' action should also include the person who paid it - that's missing from your screenshot).

@kewitz To achieve that :arrow_up: we would need to store the person who triggered the payment (aka. the one who marked it as processing) in the COLLECTIVE_EXPENSE_PAID activity. Do you see any issue if we decide to do that?

@Betree Is it OK to use the lastUpdatedBy id?

@kewitz given that the expense cannot be edited while it's processing, it seems like a safe thing to do

+1 to this... it's not critical but it would be good for transparency

+1 to this... it's not critical but it would be good for transparency

Already implemented:

Already implemented:

@kewitz I don't think it was, from the screenshot you shared where is the fees breakdown?

@kewitz I don't think it was, from the screenshot you shared where is the fees breakdown?

I'm so sorry @Betree, I saw the image and assumed it was missing the author. :man_facepalming:

I'm so sorry @Betree, I saw the image and assumed it was missing the author. :man_facepalming:

I'm so sorry @Betree, I saw the image and assumed it was missing the author. man_facepalming

I see ^^ I didn't even notice that it was missing an avatar in my screenshot :smile:

I'm so sorry @Betree, I saw the image and assumed it was missing the author. man_facepalming

I see ^^ I didn't even notice that it was missing an avatar in my screenshot :smile:

I still think we need this, it would have helped today for the use case presented in https://github.com/opencollective/opencollective/issues/4691.
To propose some specs:

[ ] On server/graphql/v2/object/Activity.ts, expose a new transaction field (based on activity.TransactionId)
[ ] In expense's ThreadActivity, whenever a COLLECTIVE_EXPENSE_PAID activity is detected, show the details from activity.transaction
[ ] Details should look more or less like this (do not show a fee if it's $0):

Total paid: $50
Payment processor fees: $0.42
Host fee: $0.13

As a pre-requisite for this, we need to associate the TransactionId to collective.expense.paid activities. It's currently missing.

I would love to see this issue prioritized, it seems like a very important piece of information that is not properly surfaced. The current flow of going to the transactions list while you're already on the expense page feels like it's hidden.

authors: Betree alanna kewitz


text:
Display total contributed on Funds

In #3589 we removed the total contributed number at the top because it was misleading for Collectives. However, now it's gone from Sponsor profiles where it's the most important number, e.g. https://opencollective.com/salesforce - you can't see anywhere how much they have contributed, and that's the key thing this profile page exists to show.
I'm not sure what we should do here exactly. Maybe a sponsor profile type with a design emphasising this number? Or a user-selectable setting to display or not display it?
There is also now a big blank space where this number used to be that looks odd.

It's because Salesforce is not an ORGANIZATION anymore, it's now a FUND. The number is still displayed for organizations and individuals that are not hosts.
We could add a setting to display that, but before adding more complexity we should maybe define the expectations for each account type.

We need a solution to this. I'm about to go out with the contributor leaderboard and if someone clicks through they won't be able to see how much that sponsor has contributed.

We nee to make sure that metrics is exposed in the API and up to date. It was Collective.stats.totalAmountSpent in v1. Should this include only Financial Contributions or should this also contain Expenses Paid (FUNDs are often contributing by paying Expenses).

Then it can be simply displayed like this:

that's a good first step -

Should this include only Financial Contributions or should this also contain Expenses Paid (FUNDs are often contributing by paying Expenses).

Expenses were removed from the computation of this number last week, based on this feedback: https://github.com/opencollective/opencollective-api/pull/4962. It again surfaces the fact that collectives have different expectations for what should be displayed here, based on how they're operating.

It's live ! https://opencollective.com/chrome

authors: Betree alanna piamancini znarf


text:
Add option to delete BG image and set to default on Collective

There's np ay to set the BG image back to the default.

done

authors: nicobarretoOC piamancini


text:
conformance: remove check for zero layers in image manifest

As per image-spec, since this is a SHOULD (hence recommended) ...
https://raw.githubusercontent.com/opencontainers/image-spec/main/manifest.md
===
layers array of objects
Each item in the array MUST be a descriptor. For portability, layers SHOULD have at least one entry.
===

I really hope to see zero layers come back as a valid manifest. Is it possible to keep these cases to see in the future which registries support these without failing conformance? This matrix should be automatable - https://github.com/opencontainers/image-spec/issues/1025

Perhaps, a way out could be to just enforce all MUSTs and warn on all SHOULDs, so we know without breaking the spirit of the conformance tests.
IINM, this particular test was enforcing that registries that don't accept zero layer manifests are not compliant, whereas the image-spec language has no such language.

A simpler variant of what we want to achieve.

@sudo-bmitch pls check https://github.com/opencontainers/distribution-spec/compare/main...rchincha:distribution-spec:artifacts
if this is preferable.
Not sure why this PR is not getting reflected the two commits.

@sudo-bmitch et al, the two commits can be squashed if we agree with the warning approach.

Borrowed the Warn() from https://github.com/opencontainers/distribution-spec/pull/375, also what appears to be a bugfix

zot/main passes with these changes
••••••••••••••••HTML report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/report.html
JUnit report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/junit.xml
Ran 63 of 68 Specs in 0.204 seconds
SUCCESS! -- 63 Passed | 0 Failed | 0 Pending | 5 Skipped

zot/main instrumented to cause warn passes but with warning

••
WARNING: image manifest with no layers is not supported
/local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:366                                                                    ••••••••

••••••••••••••••HTML report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/report.html
JUnit report was created: /local/rchincha/go/src/github.com/opencontainers/distribution-spec/conformance/junit.xml
Ran 63 of 68 Specs in 0.186 seconds
SUCCESS! -- 63 Passed | 0 Failed | 0 Pending | 5 Skipped

authors: rchincha sajayantony

repo: opencontainers/distribution-spec


text:
add curvebs volume target blocksize

issues https://github.com/opencurve/curveadm/issues/150
Signed-off-by: mfordjody mfordjody@gmail.com
[root@client-host curveadm]# go run main.go target add -h
Usage:  curveadm target add USER:VOLUME [OPTIONS]

Add a target of CurveBS

Options:
      --blocksize string   Specify target volume block size (default "512byte")
  -c, --conf string        Specify client configuration file (default "client.yaml")
      --create             Create volume iff not exist
  -h, --help               Print usage
      --host string        Specify target host (default "localhost")
      --size string        Specify volume size (default "10GB")

[root@client-host curveadm]# go run main.go target add curve:/etc --host server-host1 --create --blocksize 512
---
Error-Code: 201009
Error-Description: volume block size must end with "byte" suffix
Error-Clue: blocksize: 512
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno2#201009
  * Log: /root/.curveadm/logs/curveadm-2022-11-19_04-48-02.log
  * WeChat: opencurve_bot
exit status 1

I won't solve the container error
go run main.go target add curve:/etc --host server-host1 --create --blocksize 512byte
Add Target: [ERROR]
  + host=server-host1  volume=/etc [0/1] [ERROR]
---
Error-Code: 630009
Error-Description: run a command in container failed (docker exec CONTAINER COMMAND)
Error-Clue: Process exited with status 1
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno6#630009
  * Log: /root/.curveadm/logs/curveadm-2022-11-19_04-48-26.log
  * WeChat: opencurve_bot
exit status 1

I have to enter the container to debug the code

It should be that you made an error executing the command in the container, and you can enter the container to execute the corresponding command. Or check whether the incoming command is wrong. The way to execute the command is: docker exec -t xxx xxx.

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test --host server-hosts --create --blocksize 4096byte
Add Target: [OK]
  + host=server-hosts  volume=/test [1/1] [OK]

Add target (curve:/test) to server-hosts success ^_^
curve@client-hosts:~/curveadm/cmd/curveadm$ curveadm target list --host server-hosts
List Targets: [OK]
  + host=192.168.10.136 [1/1] [OK]

Tid  Host          Target Name                                                       Store                  Portal
---  ----          -----------                                                       -----                  ------
1    server-hosts  iqn.2022-11.com.opencurve:curve.5a82e0b942bc472fe1de2059b971b7bd  cbd:pool//test_curve_  192.168.10.136:3260

issues https://github.com/opencurve/curveadm/issues/150
Problem solved
curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add -h
Usage:  curveadm target add USER:VOLUME [OPTIONS]

Add a target of CurveBS

Options:
      --blocksize string   Specify volume block size (default "512byte")
  -c, --conf string        Specify client configuration file (default "client.yaml")
      --create             Create volume if not exist
  -h, --help               Print usage
      --host string        Specify target host (default "localhost")
      --size string        Specify volume size (default "10GB")

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test --host server-hosts --create --blocksize 4096byte
Add Target: [OK]
  + host=server-hosts  volume=/test [1/1] [OK]

Add target (curve:/test) to server-hosts success ^_^
curve@client-hosts:~/curveadm/cmd/curveadm$ curveadm target list --host server-hosts
List Targets: [OK]
  + host=192.168.10.136 [1/1] [OK]

Tid  Host          Target Name                                                       Store                  Portal
---  ----          -----------                                                       -----                  ------
1    server-hosts  iqn.2022-11.com.opencurve:curve.5a82e0b942bc472fe1de2059b971b7bd  cbd:pool//test_curve_  192.168.10.136:3260

Merge multiple commits into one.
Can this value be read from a configuration file? Add the default value in the conf

blocksize 不是在curve_ops_tool create的时候指定的，curve_ops_tool没有指定blocksize的地方。
The blocksize is not specified when the curve_ops_tool create, and the curve_ops_tool does not have the blocksize param.
$ curve_ops_tool create --example
Example:
curve_ops_tool create -mdsAddr=127.0.0.1:6666 -fileName=/test -userName=test -password=123 -fileLength=20

blocksize是在这里指定的，在这个命令的后面添加一个--blocksize $g_blocksize
blocksize is specified here, add a --blocksize $g_blocksize after this command
tgtadm --lld iscsi \
    --mode logicalunit \
    --op new \
    --tid ${g_tid} \
    --lun 1 \
    --bstype curve \
    --backing-store ${g_image}

为什么要这样修改，因为curvebs在一些版本需要保证下发的请求是4KB对齐的请求，如果用户下发一个非4KB对齐的请求到curvebs，就会出错。通过nbd挂载的设备可以保证，nbd给到curvebs的请求是4KB对齐的，而现在使用target的方式挂载的设备没有对这个做一个保证。在target 添加lun的时候，指定blocksize，可以保证所有的请求都是指定blocksize下来的。
希望这个值默认是4KB。添加target的时候，如果不指定这个命令，就按照4096字节作为blocksize。如果指定blocksize，就按照指定的blocksize。
Why do we want to modify it like this, because in some versions of curvebs, you need to ensure that the issued request is a 4KB-aligned request. If the user sends a non-4KB-aligned request to curvebs, an error will occur. The device mounted through nbd can guarantee that the request from nbd to curvebs is 4KB aligned, but the device mounted using the target method does not make a guarantee for this. When adding a lun to the target, specify the blocksize to ensure that all requests come from the specified blocksize.
Hopefully this value defaults to 4KB. When adding a target, if this command is not specified, 4096 bytes will be used as the blocksize. If blocksize is specified, follow the specified blocksize.
使用的地方参考文档
Reference documents
https://github.com/opencurve/curveadm/wiki/curve-tgt-deployment#第-5-步添加-target
$ curveadm target add <user>:<volume-name> --host target-host --create --size 10GB

如果未指定blocksize，fdisk -l 看到的可能是这样的
If blocksize is not specified, fdisk -l may see something like this

target 添加lun的时候指定blocksize，看到的是这样的
Target specifies the blocksize when adding lun, what you see is this

BTW: there are many commits in this PR, maybe we can rebase them into one :)

curve@client-hosts:~/1/curveadm/cmd/curveadm$ go run main.go target add curve:/test --host server-hosts --create --size 30GB --blocksize 512B
Add Target: [ERROR]
  + host=server-hosts  volume=/test [0/1] [ERROR]
---
Error-Code: 630009
Error-Description: run a command in container failed (docker exec CONTAINER COMMAND)
Error-Clue: tgtadm: failed to send request hdr to tgt daemon, Transport endpoint is not connected
            tgtadm target new failed
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno6#630009
  * Log: /home/curve/.curveadm/logs/curveadm-2022-11-24_10-21-01.log
  * WeChat: opencurve_bot
exit status 1

curve@client-hosts:~/1/curveadm/cmd/curveadm$ go run main.go target add curve:/test --host server-hosts --create --size 30GB --blocksize 1kB
---
Error-Code: 221011
Error-Description: volume block size be a multiple of 512B,like 1.024kB, 2.048kB,3.072kB...
Error-Clue: blocksize: 1.0 kB
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno2#221011
  * Log: /home/curve/.curveadm/logs/curveadm-2022-11-24_10-21-09.log
  * WeChat: opencurve_bot
exit status 1

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test --host server-hosts --create --size 30GB --blocksize 512
---
Error-Code: 201009
Error-Description: volume block size must end with "B" suffix
Error-Clue: blocksize: 512
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno2#201009
  * Log: /home/curve/.curveadm/logs/curveadm-2022-11-24_10-25-10.log
  * WeChat: opencurve_bot
exit status 1

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test --host server-hosts --create --size 30GB --blocksize 1kB
---
Error-Code: 221011
Error-Description: volume block size be a multiple of 512B,like 1.024kB, 2.048kB,3.072kB...
Error-Clue: blocksize: 1.0 kB
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno2#221011
  * Log: /home/curve/.curveadm/logs/curveadm-2022-11-24_10-21-09.log
  * WeChat: opencurve_bot
exit status 1

You can use the package go-humanize to parse string.

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test --host server-hosts --create --size 10GB
Add Target: [OK]
  + host=server-hosts  volume=/test [1/1] [OK]

Add target (curve:/test) to server-hosts success ^_^

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test1 --host server-hosts --create --size 10GB --blocksize 512
Add Target: [OK]
  + host=server-hosts  volume=/test1 [1/1] [OK]

Add target (curve:/test1) to server-hosts success ^_^

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test1 --host server-hosts --create --size 10GB --blocksize 1k
---
Error-Code: 221011
Error-Description: volume block size be a multiple of 512B,like 1.024kB, 2.048kB,3.072kB...
Error-Clue: blocksize: 1.0 kB
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno2#221011
  * Log: /home/curve/.curveadm/logs/curveadm-2022-11-24_11-23-51.log
  * WeChat: opencurve_bot
exit status 1

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test2 --host server-hosts --create --size 10GB --blocksize 4.096k
Add Target: [OK]
  + host=server-hosts  volume=/test2 [1/1] [OK]

Add target (curve:/test2) to server-hosts success ^_^

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target list --host server-hosts
List Targets: [OK]
  + host=192.168.10.143 [1/1] [OK]

Tid  Host          Target Name                                                       Store                   Portal
---  ----          -----------                                                       -----                   ------
1    server-hosts  iqn.2022-11.com.opencurve:curve.5a82e0b942bc472fe1de2059b971b7bd  cbd:pool//test_curve_   192.168.10.143:3260
2    server-hosts  iqn.2022-11.com.opencurve:curve.d97372e236558863e4ffc63dcfa7eb27  cbd:pool//test1_curve_  192.168.10.143:3260
3    server-hosts  iqn.2022-11.com.opencurve:curve.41f7dfa48ebe871dd74b2db0d88f1243  cbd:pool//test2_curve_  192.168.10.143:3260

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add -h
Usage:  curveadm target add USER:VOLUME [OPTIONS]

Add a target of CurveBS

Options:
      --blocksize string   Specify volume blocksize (default "512B")
  -c, --conf string        Specify client configuration file (default "client.yaml")
      --create             Create volume iff not exist
  -h, --help               Print usage
      --host string        Specify target host (default "localhost")
      --size string        Specify volume size (default "10GiB")

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test --host server-hosts --create --size 10GB
---
Error-Code: 221003
Error-Description: volume size must end with "GiB" suffix
Error-Clue: size: 10GB
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno2#221003
  * Log: /home/curve/.curveadm/logs/curveadm-2022-11-24_17-54-56.log
  * WeChat: opencurve_bot
exit status 1

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test --host server-hosts --create --size 10GiB
Add Target: [OK]
  + host=server-hosts  volume=/test [1/1] [OK]
Add target (curve:/test) to server-hosts success ^_^

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test1 --host server-hosts --create --size 10GiB --blocksize 512
---
Error-Code: 201009
Error-Description: volume block size must end with "B" suffix
Error-Clue: blocksize: 512
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno2#201009
  * Log: /home/curve/.curveadm/logs/curveadm-2022-11-24_17-50-10.log
  * WeChat: opencurve_bot
exit status 1

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test1 --host server-hosts --create --size 10GiB --blocksize 512B
Add Target: [OK]
  + host=server-hosts  volume=/test1 [1/1] [OK]

Add target (curve:/test1) to server-hosts success ^_^

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test2 --host server-hosts --create --size 10GiB --blocksize 1kiB
Add Target: [OK]
  + host=server-hosts  volume=/test2 [1/1] [OK]

Add target (curve:/test2) to server-hosts success ^_^

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test2 --host server-hosts --create --size 10GiB --blocksize 1kiB
Add Target: [ERROR]
  + host=server-hosts  volume=/test2 [0/1] [ERROR]
---
Error-Code: 630009
Error-Description: run a command in container failed (docker exec CONTAINER COMMAND)
Error-Clue: tgtadm: this target already exists
            tgtadm target new failed
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno6#630009
  * Log: /home/curve/.curveadm/logs/curveadm-2022-11-24_17-50-44.log
  * WeChat: opencurve_bot
exit status 1

curve@client-hosts:~/curveadm/cmd/curveadm$ cat /home/curve/.curveadm/logs/curveadm-2022-11-24_17-50-44.log
2022-11-24 17:50:45     [ERR]:  (log.go:109):   Execute command
    remoteAddr: curve@192.168.10.143:22
    command: sudo docker exec  curvebs-target-daemon /bin/bash /curvebs/tools/sbin/target.sh curve /test2 true 10 1024
    output: tgtadm: this target already exists
tgtadm target new failed
    error: Process exited with status 1

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target list --host server-hosts
List Targets: [OK]
  + host=192.168.10.143 [1/1] [OK]

Tid  Host          Target Name                                                       Store                   Portal
---  ----          -----------                                                       -----                   ------
1    server-hosts  iqn.2022-11.com.opencurve:curve.5a82e0b942bc472fe1de2059b971b7bd  cbd:pool//test_curve_   192.168.10.143:3260
2    server-hosts  iqn.2022-11.com.opencurve:curve.d97372e236558863e4ffc63dcfa7eb27  cbd:pool//test1_curve_  192.168.10.143:3260
3    server-hosts  iqn.2022-11.com.opencurve:curve.41f7dfa48ebe871dd74b2db0d88f1243  cbd:pool//test2_curve_  192.168.10.143:3260

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test --host server-hosts --create --size 30GiB --blocksize 4kiB
Add Target: [OK]
  + host=server-hosts  volume=/test [1/1] [OK]

Add target (curve:/test) to server-hosts success ^_^

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go target add curve:/test --host server-hosts --create --size 30GiB --blocksize 4kiB
Add Target: [ERROR]
  + host=server-hosts  volume=/test [0/1] [ERROR]
---
Error-Code: 630009
Error-Description: run a command in container failed (docker exec CONTAINER COMMAND)
Error-Clue: tgtadm: this target already exists
            tgtadm target new failed
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno6#630009
  * Log: /home/curve/.curveadm/logs/curveadm-2022-11-24_17-57-32.log
  * WeChat: opencurve_bot
exit status 1

curve@client-hosts:~/curveadm/cmd/curveadm$ cat /home/curve/.curveadm/logs/curveadm-2022-11-24_17-57-32.log
2022-11-24 17:57:34     [ERR]:  (log.go:109):   Execute command
    remoteAddr: curve@192.168.10.143:22
    command: sudo docker exec  curvebs-target-daemon /bin/bash /curvebs/tools/sbin/target.sh curve /test true 30 4096
    output: tgtadm: this target already exists
tgtadm target new failed
    error: Process exited with status 1

done

看起来有两个地方都增加了blocksize的改动，一个map的时候，一个是add target的时候。
map的地方，map命令使用curve-nbd map，这个接收的参数不包含blocksize，而且这个命令没有测试结果。
另一个是add target的地方，看起主要逻辑是正确的，遗漏了一点，blocksize默认应该是4096B。
请确认一下，map.go中的逻辑是否正确。
It seems that there are two places where the changes of blocksize have been added, one for map and one for add target.
For map, the map command uses curve-nbd map, the received parameter does not include blocksize, and this command has no test results.
The other is add target. It seems that the main logic is correct. One thing is missing. The default blocksize should be 4096B.
Please confirm whether the logic in map.go is correct.

看起来有两个地方都增加了blocksize的改动，一个map的时候，一个是add target的时候。
map的地方，map命令使用curve-nbd map，这个接收的参数不包含blocksize，而且这个命令没有测试结果。
另一个是add target的地方，看起主要逻辑是正确的，遗漏了一点，blocksize默认应该是4096B。
请确认一下，map.go中的逻辑是否正确。
It seems that there are two places where the changes of blocksize have been added, one for map and one for add target.
For map, the map command uses curve-nbd map, the received parameter does not include blocksize, and this command has no test results.
The other is add target. It seems that the main logic is correct. One thing is missing. The default blocksize should be 4096B.
Please confirm whether the logic in map.go is correct.

这个默认值应该自定义，让用户自己选择。保证他是512整数倍即可。
This default value should be customized for users to choose. Make sure he is an integral multiple of 512

我将 add 里面默认值修改成 4096B，现在他默认就是 4096B 了
I changed the default value of add to 4096B, and now it is 4096B by default

root@client-hosts:/home/curve/curveadm/cmd/curveadm# go run main.go target add curve:/test1 --host server-hosts --create --size 30GiB
Add Target: [OK]
  + host=server-hosts  volume=/test1 [1/1] [OK]

Add target (curve:/test1) to server-hosts success ^_^

root@client-hosts:/home/curve/curveadm/cmd/curveadm# go run main.go target add curve:/test1 --host server-hosts --create --size 30GiB
Add Target: [ERROR]
  + host=server-hosts  volume=/test1 [0/1] [ERROR]
---
Error-Code: 630009
Error-Description: run a command in container failed (docker exec CONTAINER COMMAND)
Error-Clue: tgtadm: this target already exists
            tgtadm target new failed
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno6#630009
  * Log: /root/.curveadm/logs/curveadm-2022-11-28_22-48-41.log
  * WeChat: opencurve_bot
exit status 1

root@client-hosts:/home/curve/curveadm/cmd/curveadm# cat /root/.curveadm/logs/curveadm-2022-11-28_22-48-41.log
2022-11-28 22:48:42     [ERR]:  (log.go:109):   Execute command
    remoteAddr: curve@192.168.10.143:22
    command: sudo docker exec  curvebs-target-daemon /bin/bash /curvebs/tools/sbin/target.sh curve /test1 true 30 4096
    output: tgtadm: this target already exists
tgtadm target new failed
    error: Process exited with status 1

root@client-hosts:/home/curve/curveadm/cmd/curveadm# go run main.go target add curve:/test2 --host server-hosts --create --size 20GiB --blocksize 1kiB
Add Target: [OK]
  + host=server-hosts  volume=/test2 [1/1] [OK]

Add target (curve:/test2) to server-hosts success ^_^

root@client-hosts:/home/curve/curveadm/cmd/curveadm# go run main.go target add curve:/test2 --host server-hosts --create --size 20GiB --blocksize 1kiB
Add Target: [ERROR]
  + host=server-hosts  volume=/test2 [0/1] [ERROR]
---
Error-Code: 630009
Error-Description: run a command in container failed (docker exec CONTAINER COMMAND)
Error-Clue: tgtadm: this target already exists
            tgtadm target new failed
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno6#630009
  * Log: /root/.curveadm/logs/curveadm-2022-11-28_22-50-12.log
  * WeChat: opencurve_bot
exit status 1

root@client-hosts:/home/curve/curveadm/cmd/curveadm# cat /root/.curveadm/logs/curveadm-2022-11-28_22-50-12.log
2022-11-28 22:50:13     [ERR]:  (log.go:109):   Execute command
    remoteAddr: curve@192.168.10.143:22
    command: sudo docker exec  curvebs-target-daemon /bin/bash /curvebs/tools/sbin/target.sh curve /test2 true 20 1024
    output: tgtadm: this target already exists
tgtadm target new failed
    error: Process exited with status 1

test map command
https://github.com/opencurve/curveadm/wiki/curvebs-client-deployment#第-4-步映射-curvebs-卷

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go map curve:/test --host server-hosts -c client.yaml --create --size 10GiB
Check Kernel Module: [OK]
  + host=server-hosts  module=nbd [1/1] [OK]

Start NEBD Service: [OK]
  + hostname=192.168.10.143  image=opencurvedocker/curvebs:v1.2 [1/1] [OK]

Create Volume: [OK]
  + hostname=192.168.10.143  image=opencurvedocker/curvebs:v1.2 [1/1] [OK]

Map Volume: [OK]
  + hostname=192.168.10.143  volume=curve:/test [1/1] [OK]

Map curve:/test to server-hosts nbd device success ^_^

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go map curve:/test --host server-hosts -c client.yaml --create --size 10GiB
Check Kernel Module: [OK]
  + host=server-hosts  module=nbd [1/1] [OK]

Start NEBD Service: [ERROR]
  + hostname=192.168.10.143  image=opencurvedocker/curvebs:v1.2 [0/1] [ERROR]
---
Error-Code: 420000
Error-Description: volume already mapped
Error-Clue: volume: curve:/test
How to Solve:
  * Website: https://github.com/opencurve/curveadm/wiki/errno4#420000
  * Log: /home/curve/.curveadm/logs/curveadm-2022-11-29_16-01-15.log
  * WeChat: opencurve_bot
exit status 1

curve@client-hosts:~/curveadm/cmd/curveadm$ cat /home/curve/.curveadm/logs/curveadm-2022-11-29_16-01-15.log

curve@client-hosts:~/curveadm/cmd/curveadm$ go run main.go map curve:/test1 --host server-hosts -c client.yaml --create --size 10GiB --blocksize 4kiB
Check Kernel Module: [OK]
  + host=server-hosts  module=nbd [1/1] [OK]

Start NEBD Service: [OK]
  + hostname=192.168.10.143  image=opencurvedocker/curvebs:v1.2 [1/1] [OK]

Create Volume: [OK]
  + hostname=192.168.10.143  image=opencurvedocker/curvebs:v1.2 [1/1] [OK]

Map Volume: [OK]
  + hostname=192.168.10.143  volume=curve:/test1 [1/1] [OK]

Map curve:/test1 to server-hosts nbd device success ^_^

curve@client-hosts:~/curveadm/cmd/curveadm$ curveadm client status
Get Client Status: [OK]

Id            Kind     Host          Container Id  Status             Aux Info
--            ----     ----          ------------  ------             --------
d056f4d680dd  curvebs  server-hosts  2a7d0b491f2b  Up 25 seconds      {"user":"curve","volume":"/test1"}
b96b111a2066  curvebs  server-hosts  356ed3034de8  Up About a minute  {"user":"curve","volume":"/test"}

repo: opencurve/curveadm


text:
Want to have an option to change the type of shape (e.g from polyline to polygon)

My actions before raising this issue

[ ] I am annotating some work and I only use polygon type of shape. However, sometimes I accidentally choose polylines and realize it after I finished the annotation. Here I faced the problem that it is not possible to change the type of annotation,  thus you can only reannotate your current job, which I found ineffective time usage. It would be great if there would be an option to change the type of shape with one click.

Expected Behaviour
Having an option to change the type of shape of annotated instances.
Also it would be great to have an option to change all same typed shapes to desired shape. In the info section we have information about the quantity of instances, and going further if we could interchange them that would be awesome.
Current Behaviour
There isn't any)
Possible Solution
Just write few lines of code, with usage of already given points, and change the structure of annotation to desired one
Steps to Reproduce (for bugs)

Upload some work
Annotate with any shape
try to change the type of shape

Context
Currently I have more than 100 instances that are polylines, but I need polygons.
Your Environment
don't think it is necessary here (I downloaded cvat 4 months ago)

@Abulegenov
In your case, you can try dumping annotations in one of formats (let's say CVAT format), create a copy (just for backup) then open the copy, rename all polyline to polygon and upload this copy annotation file back.
I do not think that implementing this conversion is UI is a good solution:

You anyway need to go throught each object and change its annotation type
If try to generalize, in the most of cases, it is unclear how to convert different annotation types between each other (polygon -> rectangle, points -> polyline, polygon, ellipse -> cuboid, mask -> polygon, etc.. )

@Abulegenov , thanks for the proposal. I don't think we are going to implement the idea in the nearest future. I will close the issue.

authors: Abulegenov bsekachev nmanovic

repo: opencv/cvat


text:
[Snyk] Upgrade dotenv-webpack from 7.1.1 to 8.0.1

This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to upgrade dotenv-webpack from 7.1.1 to 8.0.1.

:information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project.

Warning: This is a major version upgrade, and may be a breaking change.

The recommended version is 2 versions ahead of your current version.
The recommended version was released 2 months ago, on 2022-08-15.

Release notes

Package name: dotenv-webpack

8.0.1 - 2022-08-158.0.1 (2022-08-15)
Bug Fixes

bump terser from 5.14.1 to 5.14.2 (#488) (97a864d)

8.0.0 - 2022-07-088.0.0 (2022-07-08)
Bug Fixes

Extending custom path support (#487) (f5d79ee)

BREAKING CHANGES

The path, defaults and safe props all look at the path by default. Please be advised.

7.1.1 - 2022-06-277.1.1 (2022-06-27)
Bug Fixes

generating a new lockfile (41d7281)
security patch (8883b2b)

    from dotenv-webpack GitHub release notes

Commit messages

Package name: dotenv-webpack

97a864d fix: bump terser from 5.14.1 to 5.14.2 (#488)
f5d79ee fix: Extending custom path support (#487)
1adf0f3 Update README.md

Compare

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.
For more information:
🧐 View latest project report
🛠 Adjust upgrade PR settings
🔕 Ignore this dependency or unsubscribe from future upgrade PRs

/check

authors: azhavoro nmanovic


text:
Update to Airflow 2.0

Features like OAuth/LDAP Group Sync likely won't get back-ported to Airflow 1.10.X [1], so if we care about providing the ability for customers to deploy Airflow properly in a multi-tenant environment we should probably take a look at upgrading (or at least providing a compatible branch) for the 2.0 Airflow community image.
I've done some testing and besides changing the build process, the only glaring incompatibilities between the Puckel image (ours) and the new community image is how some of the service environment variables are injected into the container at runtime. We should be able to easily shim around this using environment variable templating.
Another thing I've noticed in testing 2.0 is that
airflow worker

Has changed to the more specific:
airflow celery worker

So we'll need to address that in the worker code as well.

1: https://airflow.apache.org/docs/apache-airflow/stable/upgrading-to-2.html#support-for-airflow-1-10-x-releases

Any support from opendatahub to upgrade or install airflow 2.x


text:
Update package version

@label-u/components 1.0.0-alpha.30 (2023-03-13)
Bug Fixes

frontend: fix attribute edit modal (30d78cf)

:tada: This PR is included in version 1.0.0-alpha.31 :tada:
The release is available on npm package (@alpha dist-tag)
Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.0.0-alpha.20 :tada:
The release is available on npm package (@alpha dist-tag)
Your semantic-release bot :package::rocket:

repo: opendatalab/labelU-Kit


text:
Add proper message in case of long initialization. Issue:#133

Issue #, if available:
#133
Description of changes:
Add proper message in case of long initialization. Initialization is considered as overtime if current initialization > SHINGLE_SIZE * detectionInterval + enabledTime.
No full shingle error:

No data in current time window error:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Have asked Ashwin to take a look. no comments from him.


text:
Make Pre-Filtering (Custom Scoring) Faster

Hi,
I want to use elastic search to perform a similarity search through my vectors (cosine similarity). I would like to have the search done in a subset of the documents.
I have seen that this can be done through custom scoring. The problem is the performance drop compared to the indexed knn. I guess it is due to the custom scoring is brute-forcing instead of searching through the hnsw indexed vectors.
Is there any way I can perform the prefilter search without a major performance drop?
Thanks

I have one thought for making it faster but I don't know how can I implement it:
We can index the normalized versions of the vectors and normalize the query before performing the search. This way for a normal search we won't lose performance and when we want to search with a custom scoring approach, we can search with just a dot product (as I am searching with cosine similarity space and it is same when the vectors are normalized) which is faster than cosine similarity.
If this is a good idea how can I implement it? And if it's not may I know why?

HI @H-Shafiei
Your approach would be faster. Basically, it would be the cosineOptimized function without these lines: one, two, and three. It would not be too difficult to create a dot product space for our custom scoring approach. We can treat this as a feature request.
That being said, performance would speed up, but a brute force search will take the same time complexity, regardless of space. What is your current latency with cosine approach and what is your target latency?

Closing this thread as its one month older. Feel free to reopen incase of the issues.

Closing this thread as its one month older. Feel free to reopen incase of the issues.

authors: H-Shafiei jmazanec15 vamshin

repo: opendistro-for-elasticsearch/k-NN


text:
Upgrade Opendistro to Previous versions - Dependency Issues

Hello,
I'm trying to upgrade Openditro from version 1.0.2 to version 1.7.0 with CENTOS 7. I'm following the docs as below but i'm getting dependency issues.
yum install opendistroforelasticsearch-1.7.0
Error: Package: opendistro-knn-1.7.0.0-1.noarch (opendistroforelasticsearch-artifacts-repo)
Requires: elasticsearch-oss = 7.6.1
Removing: elasticsearch-oss-7.0.1-1.x86_64 (@elasticsearch-7.x)
elasticsearch-oss = 7.0.1-1
Updated By: elasticsearch-oss-7.8.0-1.x86_64 (elasticsearch-7.x)
elasticsearch-oss = 7.8.0-1
Available: elasticsearch-oss-6.3.0-1.noarch (elasticsearch-6.x)
elasticsearch-oss = 6.3.0-1
Available: elasticsearch-oss-6.3.1-1.noarch (elasticsearch-6.x)
elasticsearch-oss = 6.3.1-1
Available: elasticsearch-oss-6.3.2-1.noarch (elasticsearch-6.x)
Is there any way to upgrade Opendistro to previous versions? I've tested to upgrade to the last version and it was succesfull.
Thanks in advance.
Gabriel.

Hi @gferrette sorry for the inconvenience here.
Seems like there is an issue with our docs on upgrading process, sorry about that.
Here is an option I test it myself to upgrade from 1.0.2 to 1.7.0.
Note that I use a wildcard due to 1.7.0 have plugin version all match to 1.7.0.0, not necessarily true for all releases:
https://opendistro.github.io/for-elasticsearch-docs/docs/install/plugins/
sudo yum install yum-utils sudo yumdownloader elasticsearch-oss-7.6.1 sudo yumdownloader opendistroforelasticsearch-1.7.0 sudo yumdownloader opendistro-*1.7.0.0 sudo yum install `ls | grep rpm`
I am seeing:
``
Installing:
opendistro-anomaly-detector   1.7.0.0-1
opendistro-index-management   1.7.0.0-1
opendistro-knn   1.7.0.0-1
Updating:
elasticsearch-oss   7.6.1-1
opendistro-alerting   1.7.0.0-1
opendistro-job-scheduler   1.7.0.0-1
opendistro-performance-analyzer   1.7.0.0-1
opendistro-security   1.7.0.0-0
opendistro-sql   1.7.0.0-1
opendistroforelasticsearch   1.7.0-1
``
NOTE: I have no data in my test and I recommended you to test this on another server with dummy data to check if data is correct after the upgrade.
Please let me know if you have any other questions.
I hope this could help you with your upgrade.
Thanks,
Peter

Hello @peterzhuamazon !
I've tested this procedure that you have suggest on my test environment and it worked like a charm!
On saved objects now i see"Advanced Settings [7.0.1]" and "Advanced Settings [7.6.1]" after upgrade.
Do you know if i can delete "Advanced Settings [7.0.1]"?

Thank you very much for your quick answer!
Gabriel.

Hi @gferrette I am not sure about this.
Where did you see this happening?
Could you show me the entire page?

Hello @peterzhuamazon !
Sure, follow below the entire page, i saw that on Management->Saved Objects:

Hi @gferrette could you show me the contents of both advanced settings?
If the contents of these saved objects do not affect normal usages, then you can just keep it there.

Hello @peterzhuamazon !
Both advanced settings have the same content as example bellow, if I change something on one "Advanced settings" it is reflecting on the other one.

Hi @gferrette you can just leave it as it be, seems like it wont impact normal usages.
If you really want to remove it:

We suggest you test the deletion on your test machine and see if there is any issues
Make sure you export the saved objects as a backup before doing so:
https://www.elastic.co/guide/en/kibana/current/managing-saved-objects.html#:~:text=Kibana are overwritten.-,Exportedit,and export objects by type

Thanks,
Peter

Hello @peterzhuamazon !
Thanks a lot for this help.
I'm going to make those tests.
Att,
Gabriel.

authors: gferrette peterzhuamazon


text:
Docker doesn't start when overriding command

Describe the bug
I m using the image amazon/opendistro-for-elasticsearch in my Gitlab CI to run integration test.
So I need to enable "discovery.type=single-node"
This can be done only by settings command: [ "elasticsearch", "-Ediscovery.type=single-node" ]
If you set 'discovery.type=single-node' as environnement variable, bash will failed
The problem is now, elastiscsearch won't start. We get this error:
java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
Likely root cause: ElasticsearchException[opendistro_security.ssl.transport.keystore_filepath or opendistro_security.ssl.transport.pemkey_filepath must be set if transport ssl is requested.]

To Reproduce
Steps to reproduce the behavior:

docker run --rm amazon/opendistro-for-elasticsearch:latest elasticsearch -Ediscovery.type=single-node

Expected behavior
I expect to be able to start elasticsearch with some parameters without using environnement variable
**Configuration **

Distribution: Docker
Host Machine: Gitlab CI, MacOS

Relevant information
I fixed the issue by editing the docker-entrypoint.sh to run install_demo_configuration.sh earlier. I can open a PR if you agree with this idea

Hi @fxalgrain this line works for me, can you try:
docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" amazon/opendistro-for-elasticsearch:latest

Hi @peterzhuamazon
First of all, thanks you to have checked on your side.
I agree with you. If we set environnement variables like this, it works as expected. But in the GitlabCI (I don't check yet on others), you could not start the container using this command line.
You can do two things in the config file:

you can define global environnement variables
you can define the docker command.

Unfortunately, the first option will not work because "export discovery.type=single-node" is not valid in bash.
So we have to define the docker command, and this result in the error I described.

Hi @fxalgrain I am not experienced with GitLabCI so I am not sure what do you mean by could not start the container using this command line.
-e "discovery.type=single-node" is part of the docker command parameter, and if you can define the docker command like your said, you should be able to use it.
Anyway this is something quite rare to us, could you just use a docker-compose file to define env var and just call docker-compose instead of docker run?
I quickly created a docker compose file and seems working well with single-node cluster:
version: '3'
services:
  odfe-node1:
    image: amazon/opendistro-for-elasticsearch:1.11.0
    container_name: odfe-node1
    environment:
      - cluster.name=odfe-cluster
      - node.name=odfe-node1
      - discovery.type=single-node
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
        hard: 65536

    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer

Thanks.

Hi @fxalgrain there is no responses from you after 7 days, we will close this issue for now.
Please feel free to reopen it if needed.
Thanks.

Hi @fxalgrain there is no responses from you after 7 days, we will close this issue for now.
Please feel free to reopen it if needed.
Thanks.

Hi Peter,
To summarize, the goal is to start an elasticsearch with the option discovery.type=single-node in a Gitlab CI context.
I reopen the ticket to give you more details about the issue, as you have not experiences with Gitlab CI.
First of all, I agree with you that in docker CLI or in docker-compose  context, it's working fine.
But to run jobs on Gitlab CI, you have to define a .gitlab-ci.yml.
The first idea is to mimic the docker-compose settings and to declare the env variable discovery.type=single-node
But, If you define this, you will get a Bash error. (because Gitlab runner translate it into export discovery.type=single-node)
To set the discovery type, the remaining option is to override the docker command.
So in .gitlab-ci.yml, you have to define command: [ "elasticsearch", "-Ediscovery.type=single-node" ]
But now, you will get an error from elasticsearch :
java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
Likely root cause: ElasticsearchException[opendistro_security.ssl.transport.keystore_filepath or opendistro_security.ssl.transport.pemkey_filepath must be set if transport ssl is requested.]

You can reproduce this on your local environnement :
docker run --rm amazon/opendistro-for-elasticsearch:latest elasticsearch -Ediscovery.type=single-node
I analyse the docker-entrypoint.sh init script. And we get this error because 'install_demo_configuration.sh' is not run when command is overrided.  When "$1" != "eswrapper", we execute the request command and so the script never get to bottom lines.
I think we can fix this issue (and may be some other issue) by moving down the  "$1" != "eswrapper" code block.

Hi Peter,
To summarize, the goal is to start an elasticsearch with the option discovery.type=single-node in a Gitlab CI context.
I reopen the ticket to give you more details about the issue, as you have not experiences with Gitlab CI.
First of all, I agree with you that in docker CLI or in docker-compose  context, it's working fine.
But to run jobs on Gitlab CI, you have to define a .gitlab-ci.yml.
The first idea is to mimic the docker-compose settings and to declare the env variable discovery.type=single-node
But, If you define this, you will get a Bash error. (because Gitlab runner translate it into export discovery.type=single-node)
To set the discovery type, the remaining option is to override the docker command.
So in .gitlab-ci.yml, you have to define command: [ "elasticsearch", "-Ediscovery.type=single-node" ]
But now, you will get an error from elasticsearch :
java.lang.IllegalStateException: failed to load plugin class [com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin]
Likely root cause: ElasticsearchException[opendistro_security.ssl.transport.keystore_filepath or opendistro_security.ssl.transport.pemkey_filepath must be set if transport ssl is requested.]

You can reproduce this on your local environnement :
docker run --rm amazon/opendistro-for-elasticsearch:latest elasticsearch -Ediscovery.type=single-node
I analyse the docker-entrypoint.sh init script. And we get this error because 'install_demo_configuration.sh' is not run when command is overrided.  When "$1" != "eswrapper", we execute the request command and so the script never get to bottom lines.
I think we can fix this issue (and may be some other issue) by moving down the  "$1" != "eswrapper" code block.

Please find the docker-entrypoint.sh I used to fix the issue on our side
docker-entrypoint.sh.zip

Please find the docker-entrypoint.sh I used to fix the issue on our side
docker-entrypoint.sh.zip

@fxalgrain I am having exactly the same issue after upgrading to Elasticsearch 8.2.2.
Have you found an alternative solution, other than building your own docker image?

@dawi I didn't inspect again the issue since last year.
If the docker-entrypoint.sh has not change since, it's still the only way to fix it

GitLab allows to configure the entrypoint as well.
Therefore, the following service definition works for me:
services:
  - name: elastic/elasticsearch:8.2.2
    alias: elasticsearch
    entrypoint:
      - "bin/elasticsearch"
      - "-Ediscovery.type=single-node"
      - "-Expack.security.enabled=false"

I don't know if this also applies to opensearch or advanced configuration use-cases.

GitLab allows to configure the entrypoint as well.
Therefore, the following service definition works for me:
services:
  - name: elastic/elasticsearch:8.2.2
    alias: elasticsearch
    entrypoint:
      - "bin/elasticsearch"
      - "-Ediscovery.type=single-node"
      - "-Expack.security.enabled=false"

I don't know if this also applies to opensearch or advanced configuration use-cases.

authors: dawi fxalgrain peterzhuamazon


text:
Missing DEB packages for 1.13.3 version

Describe the bug
Same story as here but for DEB packages.
To Reproduce
Steps to reproduce the behavior:

wget -qO - https://d3g5vo6xdbdb9a.cloudfront.net/GPG-KEY-opendistroforelasticsearch | sudo apt-key add -
echo "deb https://d3g5vo6xdbdb9a.cloudfront.net/apt stable main" | sudo tee -a   /etc/apt/sources.list.d/opendistroforelasticsearch.list
sudo apt list -a opendistro-*
sudo apt list -a opendistroforelasticsearch

Expected behavior
Available deb and yum packages in the repository for latest release opendistroforelasticsearch and opendistro- plugins
Relevant information
opendistroforelasticsearch/stable 1.13.2-1 amd64
opendistroforelasticsearch/stable 1.13.1-1 amd64
opendistroforelasticsearch/stable 1.13.0-1 amd64
...

Please refer to this forum thread for more information on RPM packages for 1.13.x https://discuss.opendistrocommunity.dev/t/log4j2-vulnerability-in-opensearch/7935/15?u=bbarani

authors: bbarani rafzei


text:
Polars MVP documentation

Include examples of using Polars for sums, means, groupby, and quantiles

(depending on time, this may move to the next, smaller release)

In the API docs, rust/src/measurements/code/make_private_lazyframe.rst already has sums and groupby, adding mean wouldn't be hard, and with a data set this trivial, median might be sufficient to demonstrate quantiles?

closing in favor of multiple notebook issues

authors: mccalluc raprasad

repo: opendp/opendp


text:
InitContainers - Handle builtin kernel modules

Describe the bug
Some components, by default, use init containers to check for nvme kernel modules.
The problem is these modules might actually get compiled builtin, and these checks are failing in this case.
To Reproduce
Install on nodes with builtin nvme-tcp kernel modules.
Expected behavior
InitContainer should not fail.
** OS info (please complete the following information):**

Distro: Talos
Example: https://discuss.kubernetes.io/t/talos-linux-cluster-missing-module/27519

Thanks for all your hard work on this!   I just ran into this one. The link suggests disabling init_containers.    While this is being worked on, how can I disable init_containers?

Hey, I think you need to disable like so for example, if you use mayastor chart directly:
helm upgrade mayastor mayastor/mayastor -n mayastor --reuse-values --set "csi.node.initContainers.enabled=false"

@CloudFocused I work with @tiagolobocastro on the futures for Mayastor. If you're open to talking about your experiences, and what you're doing, let me know (you can reply here or DM me). Ed

You are right @tiagolobocastro -  That did it. Thanks!

Is there a solution for this using the openebs helm chart?

Scoping this for v4.2

authors: CloudFocused avishnu edrob999 johnpanos tiagolobocastro

repo: openebs/mayastor


text:
Update bluegreen.md

Minor grammar changes

Thank you for notifying the change. Since there are some conflicts in the Page, Changes are done manually. We appreciate your effort.

authors: dharmendra-lingaiah ranjithwingrider

repo: openebs/openebs-docs


text:
failed to migrate striped SPC with multiple disks into CSPC

Is this a BUG REPORT or FEATURE REQUEST?
Choose one: BUG REPORT or FEATURE REQUEST

What happened:

SPC to CSPC migration job failed saying:

I0815 11:49:29.820555       1 pool.go:458] Updating bdc bdc-0b39d003-4a73-4049-aada-63f20d6526c6 with cspc labels & finalizer.
I0815 11:49:29.832968       1 pool.go:458] Updating bdc bdc-315d9da7-ca8b-4aa0-9725-1cca1c4eaac2 with cspc labels & finalizer.
I0815 11:49:29.841167       1 pool.go:458] Updating bdc bdc-3401607b-0d52-461d-ba86-a368b5f489d0 with cspc labels & finalizer.
I0815 11:49:29.851333       1 pool.go:458] Updating bdc bdc-5254387a-194e-414b-9f6f-6af8b6ee48a4 with cspc labels & finalizer.
I0815 11:49:29.863005       1 pool.go:458] Updating bdc bdc-90f9ed6b-f2f7-4d00-9235-ac286ac6b6ae with cspc labels & finalizer.
I0815 11:49:29.871193       1 pool.go:458] Updating bdc bdc-a62796fa-7658-40b7-bcb8-6b640f65af95 with cspc labels & finalizer.
I0815 11:49:29.880070       1 pool.go:458] Updating bdc bdc-db878430-65dc-4194-b68d-1622d3324c97 with cspc labels & finalizer.
I0815 11:49:29.888270       1 pool.go:458] Updating bdc bdc-fe163f74-49bc-4516-ac6c-5ab23435f2d5 with cspc labels & finalizer.
I0815 11:49:29.897097       1 pool.go:153] Creating equivalent cspc cstor-disk-gitlab for spc cstor-disk-gitlab
E0815 11:49:30.039931       1 cstor_pool.go:86] admission webhook "admission-webhook.cstor.openebs.io" denied the request: invalid cspc specification: invalid pool spec: stripe dataRaidGroups should have exactly one raidGroup
F0815 11:49:30.040071       1 cstor_pool.go:49] Failed to migrate cStor SPC : cstor-disk-gitlab

What you expected to happen:

Migration should have completed successfully.

How to reproduce it (as minimally and precisely as possible):

Provision SPC with the following spec:

apiVersion: openebs.io/v1alpha1
kind: StoragePoolClaim
metadata:
  name: cstor-disk-gitlab
spec:
  name: cstor-disk-gitlab
  type: disk
  maxPools: 4
  poolSpec:
    poolType: striped
    thickProvisioning: false
  blockDevices:
    blockDeviceList:
    - blockdevice-<node1>
    - blockdevice-<node1>
    - blockdevice-<node2>
    - blockdevice-<node2>
    - blockdevice-<node3>
    - blockdevice-<node3>
    - blockdevice-<node4>
    - blockdevice-<node4>

Then try migrating the above provisioned SPC to CSPC.

Anything else we need to know?:

The issue got fixed with the PR https://github.com/openebs/upgrade/pull/38

repo: openebs/openebs


text:
Fix for Issue #638

What this PR does / why we need it:

Refactors the storage classes for OpenEBS and reduces the number of classes to 2.
Updates the storage class changes to reflect in all the demo applications.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Issue #638
Special notes for your reviewer:
This PR is also dependent on the fix for Issue #645 , to pickup the properties from the storage classes.
Signed-off-by: yudaykiran udaykiran.y@gmail.com

@yudaykiran
Just looking at the changes, this is touching quite a lot of files. And can cause merge conflicts with some of the open PRs. Can we split the change as follows:

Only replace the openebs-basic to openebs-standard (change only the demo yaml files that were using openebs-basic)
Add a new class called openebs-standalone (for the no-replication case). Don't use this anywhere yet.

I am going to close this PR, can you resend a new PR with the above changes.

Sure, will raise a new PR with the suggested changes.

authors: kmova yudaykiran


text:
Test django-splash on Python 3.11

This repository is a depedency of edx-platform and needs to be upgraded to Python 3.11 before
the Readwood release is cut (mid-April).

[ ] Requirements are compiled with Python 3.8
[ ] Tests are run on Python 3.8 and 3.11
[ ] (Optional) Tests are also run with 3.12 and passing or 3.12 issues are ticketed.
[ ] Classifiers in setup.py setup.cfg or pyproject.toml indicate Python 3.11 support
[ ] A new version is release to PyPI
[ ] A PR is merged to edx-platform to use the new version

I wrote this in the django-splash PR, but django-splash should be archived and instead of upgraded.

authors: feanil ormsbee

repo: openedx/django-splash


text:
[BD-46] feat: added variants for the Select Menu component

Description
Include a description of your changes here, along with a link to any relevant Jira tickets and/or Github issues.
Deploy Preview
Include a direct link to your changes in this PR's deploy preview here (e.g., a specific component page).
Merge Checklist

[ ] If your update includes visual changes, have they been reviewed by a designer? Send them a link to the Netlify deploy preview, if applicable.
[ ] Does your change adhere to the documented style conventions?
[ ] Do any prop types have missing descriptions in the Props API tables in the documentation site (check deploy preview)?
[ ] Were your changes tested using all available themes (see theme switcher in the header of the deploy preview, under the "Settings" icon)?
[ ] Were your changes tested in the example app?
[ ] Is there adequate test coverage for your changes?
[ ] Consider whether this change needs to reviewed/QA'ed for accessibility (a11y). If so, please add wittjeff and adamstankiewicz as reviewers on this PR.

Post-merge Checklist

[ ] Verify your changes were released to NPM at the expected version.
[ ] If you'd like, share your contribution in #show-and-tell.
[ ] 🎉 🙌 Celebrate! Thanks for your contribution.

Thanks for the pull request, @PKulkoRaccoonGang!
When this pull request is ready, tag your edX technical lead.

@PKulkoRaccoonGang 🎉 Your pull request was merged!  Please take a moment  to answer a two question survey so we can improve your experience in the future.

:tada: This PR is included in version 20.39.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 21.0.0-alpha.31 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

repo: openedx/paragon


text:
configmap openelb-vip-configmap was not found

Describe the bug

openelb-system/openelb-vip-configmap: configmap openelb-vip-configmap was not found
To Reproduce

Expected behaviour

don't traverse configmap openelb-vip-configmap in all namespaces.
openelb-system/openelb-vip-configmap don't created.
Output

err unexpected error searching configmap openelb-system/openelb-vip-configmap: configmap openelb-vip-configmap was not found
Version Info

Version of Kubernetes:
v1.21.5
Version of OpenELB:

kube-keepalived-vip:0.35
openelb:v0.5.1

https://github.com/openelb/openelb/pull/304

Hi,
As Hurricane spoken, I've been hit by the same issue. The openelb-system/openelb-vip-configmap did not been created automatically. And https://github.com/openelb/openelb/pull/304 seems be another issue, which is not meet this problem.
Really seeking for your help.

I've been hit by the same issue.

Hi, @cliff-wsy can you give us some information to help us investigate this problem?

I've been hit by the same issue.

Hi, @cliff-wsy can you give us some information to help us investigate this problem?

Hi, @chaunceyjiang
Apologize to reply delay.
Version infor as below:
Kubernetes: v1.21.5
OpenELB: v0.5.1
Kube-Keepalived-vip: 0.35
Description:
I applied yaml file on kubernetes cluster successfully with no error. However, logs from openelb-keepalived-vip showed configmap about openelb-vip-configmap was not found. As mentioned, there was no such configmap in any namespaces. And I had to create configmap manually. Then error message was gone.
And there are another two issues.
One as daemonset "openelb-keepalived-vip" was created automatically. How can I define the nodeselector or toleration rules for it? I can not see any deployment yaml for it.
The second was about how to define image repository about Kube-keepalived-vip to local harbor?
Really appreciate for your kindly help.

You can edit daemonset openelb-keepalived-vip directly.
See this for a custom mirror: https://github.com/openelb/openelb/pull/285

You can edit daemonset openelb-keepalived-vip directly.
See this for a custom mirror: Support config for container image #285

That could help. Thanks in advance!

authors: chaunceyjiang cliff-wsy hurricane1988 renyunkang

repo: openelb/openelb


text:
Add CNCF logo

via https://github.com/cncf/toc/issues/743

/cc @FeynmanZhou

repo: openelb/porter-website