text:
ppc not building anymore

http://mr-0xc1:8080/job/h2o4gpu-ppc64le-cuda9/job/master/24/consoleFull
http://mr-0xc1:8080/job/h2o4gpu-ppc64le-cuda9/job/master/25/consoleFull
The two logs:
consoles.zip
For ppc h2o4gpu was fine on build 24 but not 25, but only diff was things that explicitly aren't run for ppc:
git diff fd44ea30f9ec32da462d031092c16173a159030c ad2d4a92ec282ef29bc3924cddb179ca38fa0ece

diff --git a/Dockerfile-build b/Dockerfile-build
index 9e688ea..b3988f8 100644
--- a/Dockerfile-build
+++ b/Dockerfile-build
@@ -181,9 +181,9 @@ RUN \
     cd boost_1_58_0 && \
     export PYTHONPATH=/opt/h2oai/h2o4gpu/python/ && \
     ./bootstrap.sh --prefix=/opt/boost/ --with-python=python3 && \
-    export CPPFLAGS="-I/opt/h2oai/h2o4gpu/python/include/python3.6m/" && \
+    export CPPFLAGS="-I/opt/h2oai/h2o4gpu/python/include/python3.6m/ -fPIC" && \
     export C_INCLUDE_PATH="/opt/h2oai/h2o4gpu/python/include/python3.6m/" ; export CPLUS_INCLUDE_PATH="/opt/h2oai/h2o4gpu/python/include/python3.6m/" && \
-    ./b2 -a -d0 install --prefix=/opt/boost/ --with=all -j 20 && \
+    ./b2 link=static -a -d0 install --prefix=/opt/boost/ --with=all -j 20 cxxflags="-fPIC -I /opt/h2oai/h2o4gpu/python/include/python3.6m/" && \
     cd /usr/include ; rm -rf boost ; ln -s /opt/boost/include/boost . && \
     cd /usr/lib64/ ; rm -rf libboost* ; cp -a /opt/boost/lib/* . && \
     cd / ; \
diff --git a/Makefile b/Makefile
index 9208dff..ca7161a 100644
--- a/Makefile
+++ b/Makefile
@@ -103,7 +103,7 @@ lightgbm:
        @echo "sudo apt-get install libboost-dev libboost-system-dev libboost-filesystem-dev cmake"
        if [ `arch` != "ppc64le" ]; then \
        rm -rf LightGBM ; result=`git clone --recursive https://github.com/Microsoft/LightGBM` && \
-       cd LightGBM && (rm -rf build || true) && mkdir -p build ; cd build && cmake .. -DUSE_GPU=1 -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ -DOpenCL_LIBRARY=$(CUDA_HOME)/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=$(CUDA_HOME)/include/ && \
+       cd LightGBM && (rm -rf build || true) && mkdir -p build ; cd build && cmake .. -DUSE_GPU=1 -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ -DOpenCL_LIBRARY=$(CUDA_HOME)/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=$(CUDA_HOME)/include/ -DBOOST_ROOT=/opt/boost -DBoost_USE_STATIC_LIBS=1 && \
        make OPENCL_HEADERS=$(CUDA_HOME)/targets/x86_64-linux/include LIBOPENCL=$(CUDA_HOME)/targets/x86_64-linux/lib -j && cd .. && \
        cd python-package &&  sed -i 's/self\.gpu \= 0/self.gpu = 1/g' setup.py && cd .. && \
        cd python-package &&  sed -i 's/self\.precompile \= 0/self.precompile = 1/g' setup.py && cd .. && \
(END)

All the above is inside an arch != "ppc64le" so literally nothing changed between commits.  Must be external change, like versioning, but didn't find it so far.

The error:
03:18:38 h2o4gpu/cluster/_k_means_elkan.c: In function
â<U+0080><U+0098>__pyx_pf_7h2o4gpu_7cluster_14_k_means_elkan_k_means_elkanâ<U+0080><U+0099>:
03:18:38 h2o4gpu/cluster/_k_means_elkan.c:3890:7: error: too many arguments to
function â<U+0080><U+0098>__Pyx_PyObject_to_MemoryviewSlice_dsds_floatâ<U+0080><U+0099>
03:18:38        __pyx_t_8 = __Pyx_PyObject_to_MemoryviewSlice_dsds_float
(__pyx_v_arg, 0);
03:18:38        ^
03:18:38 h2o4gpu/cluster/_k_means_elkan.c:2169:41: note: declared here
03:18:38  static CYTHON_INLINE __Pyx_memviewslice
__Pyx_PyObject_to_MemoryviewSlice_dsds_float(PyObject *);
03:18:38                                          ^
03:18:38 h2o4gpu/cluster/_k_means_elkan.c:3912:7: error: too many arguments to
function â<U+0080><U+0098>__Pyx_PyObject_to_MemoryviewSlice_dsds_doubleâ<U+0080><U+0099>
03:18:38        __pyx_t_8 = __Pyx_PyObject_to_MemoryviewSlice_dsds_double
(__pyx_v_arg, 0);
03:18:38        ^
03:18:38 h2o4gpu/cluster/_k_means_elkan.c:2172:41: note: declared here
03:18:38  static CYTHON_INLINE __Pyx_memviewslice
__Pyx_PyObject_to_MemoryviewSlice_dsds_double(PyObject *);
03:18:38                                          ^
03:18:38 error: Command "gcc -pthread -B /opt/h2oai/h2o4gpu/python/
compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -
Wstrict-prototypes -fPIC -I/opt/h2oai/h2o4gpu/python/lib/python3.6/site-
packages/numpy/core/include -I/opt/h2oai/h2o4gpu/python/lib/python3.6/site-
packages/numpy/core/include -I/opt/h2oai/h2o4gpu/python/include/python3.6m -
c h2o4gpu/cluster/_k_means_elkan.c -o build/temp.linux-ppc64le-3.6/h2o4gpu/
cluster/_k_means_elkan.o -MMD -MF build/temp.linux-ppc64le-3.6/h2o4gpu/cluster/
_k_means_elkan.o.d" failed with exit status 1

Tom suggested:
 # Convert .py files to .c and compile to .so
        ext_modules=cythonize(extensions,
                              compiler_directives={
                                  'always_allow_keywords': True
                              }),

But that failed to build:
https://github.com/h2oai/h2o4gpu/pull/644

So the above seems to happen on the px01 machine, on the GPU enabled Power box we have I noticed a different issue http://mr-0xc1:8080/job/h2o4gpu-ppc64le-cuda9/job/master/29/consoleFull
15:18:47     Did not find libzmq via pkg-config.
15:18:47     {'libraries': ['zmq'], 'include_dirs': [], 'library_dirs': [], 'runtime_library_dirs': [], 'extra_link_args': []}
15:18:47     gcc -pthread -B /opt/h2oai/h2o4gpu/python/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -c build/temp.linux-ppc64le-3.6/scratch/check_sys_un.c -o build/temp.linux-ppc64le-3.6/scratch/check_sys_un.o
15:18:47     gcc -pthread -B /opt/h2oai/h2o4gpu/python/compiler_compat -Wl,--sysroot=/ build/temp.linux-ppc64le-3.6/scratch/check_sys_un.o -o build/temp.linux-ppc64le-3.6/scratch/check_sys_un
15:18:47     ************************************************
15:18:47     Configure: Autodetecting ZMQ settings...
15:18:47         Custom ZMQ dir:
15:18:47     creating build/temp.linux-ppc64le-3.6/scratch/tmp
15:18:47     cc -c /tmp/timer_create98s9tok6.c -o build/temp.linux-ppc64le-3.6/scratch/tmp/timer_create98s9tok6.o
15:18:47     cc build/temp.linux-ppc64le-3.6/scratch/tmp/timer_create98s9tok6.o -o build/temp.linux-ppc64le-3.6/scratch/a.out
15:18:47     build/temp.linux-ppc64le-3.6/scratch/tmp/timer_create98s9tok6.o: In function `main':
15:18:47     timer_create98s9tok6.c:(.text+0x28): undefined reference to `timer_create'
15:18:47     collect2: error: ld returned 1 exit status
15:18:47     gcc -pthread -B /opt/h2oai/h2o4gpu/python/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Izmq/utils -Izmq/backend/cython -Izmq/devices -c build/temp.linux-ppc64le-3.6/scratch/vers.c -o build/temp.linux-ppc64le-3.6/scratch/vers.o
15:18:47     build/temp.linux-ppc64le-3.6/scratch/vers.c:4:17: fatal error: zmq.h: No such file or directory
15:18:47      #include "zmq.h"
15:18:47                      ^
15:18:47     compilation terminated.
15:18:47
15:18:47     error: command 'gcc' failed with exit status 1
15:18:47
15:18:47     Failed with default libzmq, trying again with /usr/local
15:18:47     {'libraries': ['zmq'], 'include_dirs': ['/usr/local/include'], 'library_dirs': ['/usr/local/lib'], 'runtime_library_dirs': ['/usr/local/lib'], 'extra_link_args': []}
15:18:47     ************************************************
15:18:47     Configure: Autodetecting ZMQ settings...
15:18:47         Custom ZMQ dir:       /usr/local
15:18:47     cc -c /tmp/timer_createdhg0ij3q.c -o build/temp.linux-ppc64le-3.6/scratch/tmp/timer_createdhg0ij3q.o
15:18:47     Assembler messages:
15:18:47     Fatal error: can't create build/temp.linux-ppc64le-3.6/scratch/tmp/timer_createdhg0ij3q.o: No such file or directory
15:18:47     gcc -pthread -B /opt/h2oai/h2o4gpu/python/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/include -Izmq/utils -Izmq/backend/cython -Izmq/devices -c build/temp.linux-ppc64le-3.6/scratch/vers.c -o build/temp.linux-ppc64le-3.6/scratch/vers.o
15:18:47     build/temp.linux-ppc64le-3.6/scratch/vers.c:4:17: fatal error: zmq.h: No such file or directory
15:18:47      #include "zmq.h"
15:18:47                      ^
15:18:47     compilation terminated.
15:18:47
15:18:47     error: command 'gcc' failed with exit status 1
15:18:47
15:18:47     ************************************************
15:18:47     Warning: Couldn't find an acceptable libzmq on the system.
15:18:47
15:18:47     If you expected pyzmq to link against an installed libzmq, please check to make sure:
15:18:47
15:18:47         * You have a C compiler installed
15:18:47         * A development version of Python is installed (including headers)
15:18:47         * A development version of ZMQ >= 3.2 is installed (including headers)
15:18:47         * If ZMQ is not in a default location, supply the argument --zmq=<path>
15:18:47         * If you did recently install ZMQ to a default location,
15:18:47           try rebuilding the ld cache with `sudo ldconfig`
15:18:47           or specify zmq's location with `--zmq=/usr/local`
15:18:47
15:18:47     You can skip all this detection/waiting nonsense if you know
15:18:47     you want pyzmq to bundle libzmq as an extension by passing:
15:18:47
15:18:47         `--zmq=bundled`
15:18:47
15:18:47     I will now try to build libzmq as a Python extension
15:18:47     unless you interrupt me (^C) in the next 10 seconds...
15:18:47
15:18:47
    10...
     9...
     8...
     7...
     6...
     5...
     4...
     3...
     2...
     1...
15:18:47     ************************************************
15:18:47     Using bundled libzmq
15:18:47     already have bundled/zeromq
15:18:47     already have platform.hpp
15:18:47     ************************************************
15:18:47     checking for timer_create
15:18:47     creating build/temp.linux-ppc64le-3.6/tmp
15:18:47     cc -c /tmp/timer_createry_12djx.c -o build/temp.linux-ppc64le-3.6/tmp/timer_createry_12djx.o
15:18:47     cc build/temp.linux-ppc64le-3.6/tmp/timer_createry_12djx.o -o build/temp.linux-ppc64le-3.6/a.out
15:18:47     build/temp.linux-ppc64le-3.6/tmp/timer_createry_12djx.o: In function `main':
15:18:47     timer_createry_12djx.c:(.text+0x28): undefined reference to `timer_create'
15:18:47     collect2: error: ld returned 1 exit status
15:18:47     no timer_create, linking librt
15:18:47     ************************************************
15:18:47     building 'zmq.libzmq' extension
15:18:47     creating build/temp.linux-ppc64le-3.6/buildutils
15:18:47     creating build/temp.linux-ppc64le-3.6/bundled
15:18:47     creating build/temp.linux-ppc64le-3.6/bundled/zeromq
15:18:47     creating build/temp.linux-ppc64le-3.6/bundled/zeromq/src
15:18:47     gcc -pthread -B /opt/h2oai/h2o4gpu/python/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DZMQ_HAVE_CURVE=1 -DZMQ_USE_TWEETNACL=1 -DZMQ_USE_EPOLL=1 -Ibundled/zeromq/include -Ibundled -I/opt/h2oai/h2o4gpu/python/include/python3.6m -c buildutils/initlibzmq.c -o build/temp.linux-ppc64le-3.6/buildutils/initlibzmq.o
15:18:47     gcc -pthread -B /opt/h2oai/h2o4gpu/python/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DZMQ_HAVE_CURVE=1 -DZMQ_USE_TWEETNACL=1 -DZMQ_USE_EPOLL=1 -Ibundled/zeromq/include -Ibundled -I/opt/h2oai/h2o4gpu/python/include/python3.6m -c bundled/zeromq/src/socket_base.cpp -o build/temp.linux-ppc64le-3.6/bundled/zeromq/src/socket_base.o
15:18:47     cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ [enabled by default]
15:18:47     In file included from bundled/zeromq/src/options.hpp:38:0,
15:18:47                      from bundled/zeromq/src/own.hpp:37,
15:18:47                      from bundled/zeromq/src/socket_base.hpp:37,
15:18:47                      from bundled/zeromq/src/socket_base.cpp:50:
15:18:47     bundled/zeromq/src/atomic_ptr.hpp: In member function 'int zmq::atomic_value_t::load() const':
15:18:47     bundled/zeromq/src/atomic_ptr.hpp:271:9: error: invalid initialization of reference of type 'zmq::mutex_t&' from expression of type 'const zmq::mutex_t'
15:18:47              );
15:18:47              ^
15:18:47     bundled/zeromq/src/atomic_ptr.hpp:116:14: error: in passing argument 4 of 'void* zmq::atomic_cas(void* volatile*, void*, void*, zmq::mutex_t&)'
15:18:47      inline void *atomic_cas (void *volatile *ptr_,
15:18:47                   ^
15:18:47     bundled/zeromq/src/atomic_ptr.hpp:273:5: warning: control reaches end of non-void function [-Wreturn-type]
15:18:47          }
15:18:47          ^
15:18:47     error: command 'gcc' failed with exit status 1
15:18:47
15:18:47     ----------------------------------------
15:18:47 Command "/opt/h2oai/h2o4gpu/python/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-to3fqj3u/pyzmq/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-uhcrftk0-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-to3fqj3u/pyzmq/
15:18:47 You are using pip version 9.0.1, however version 10.0.1 is available.
15:18:47 You should consider upgrading via the 'pip install --upgrade pip' command.
15:18:47 The command '/bin/sh -c pip install -r requirements_buildonly.txt' returned a non-zero code: 1

That may be more recent, as it seems to be due to xgboost pip install line I added in Makefile2.  It works fine with x86/centos, but maybe some things there are not needed for xgboost as we use it.

I resolved the pyzmq build error on ppc64le by installing libzmq3-dev (apt-get) instead of libzmq-dev before pip install -U pyzmq.

Great, thanks!  Was that a new issue or did the build fully pass?

I think a new version of pyzmq was pushed to PyPI which looks for libzmq3-dev to be installed.  When it is not found, it attempts to compile libzmq as a C extension, which fails on power8.

@CreatureDev , I still see ppc builds failing for PR-671 due to llvmlite. Have verified that the same code works fine for x86_64.
Any thoughts ?
17:51:05   Complete output from command /opt/h2oai/h2o4gpu/python/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-pvcub999/llvmlite/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpqmr2_x8xpip-wheel- --python-tag cp36:
17:51:05   running bdist_wheel
17:51:05   /opt/h2oai/h2o4gpu/python/bin/python /tmp/pip-build-pvcub999/llvmlite/ffi/build.py
17:51:05   LLVM version... 4.0.1
17:51:05
17:51:05   Traceback (most recent call last):
17:51:05     File "/tmp/pip-build-pvcub999/llvmlite/ffi/build.py", line 141, in <module>
17:51:05       main()
17:51:05     File "/tmp/pip-build-pvcub999/llvmlite/ffi/build.py", line 131, in main
17:51:05       main_posix('linux', '.so')
17:51:05     File "/tmp/pip-build-pvcub999/llvmlite/ffi/build.py", line 105, in main_posix
17:51:05       raise RuntimeError(msg)
17:51:05   RuntimeError: Building llvmlite requires LLVM 5.0.x. Be sure to set LLVM_CONFIG to the right executable path.
17:51:05   Read the documentation at http://llvmlite.pydata.org/ for more information about building llvmlite.
17:51:05
17:51:05   error: command '/opt/h2oai/h2o4gpu/python/bin/python' failed with exit status 1
17:51:05
17:51:05   ----------------------------------------
17:51:05   Failed building wheel for llvmlite

authors: hemenkapadia mdymczyk pseudotensor trstovall


text:
TypeError: Chroma.init() got an unexpected keyword argument 'anonymized_telemetry'

Running this command python generate.py --base_model=h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-7b --langchain_mode=MyData --score_model=None
I'm getting the title'd error while trying to parse a single PDF.
Have tried PDFs of varying sizes.
Not sure if it's related to not being able to install this AutoGPTQ dependency as it is erroring out:
pip install -r https://github.com/PanQiWei/AutoGPTQ/releases/download/v0.2.2/auto_gptq-0.2.2+cu118-cp310-cp310-win_amd64.whl
ERROR: Invalid requirement: 'PK\x03\x04\x14\x00\x00\x00\x08\x00�.�V���/��\t\x00\x00`(\x00!\x00\x00\x00autogptq_cuda.cp310-win_amd64.pydĽ\t|TE\x120�f&\x03�\x11ޠ' (from line 1 of https://github.com/PanQiWei/AutoGPTQ/releases/download/v0.2.2/auto_gptq-0.2.2+cu118-cp310-cp310-win_amd64.whl)
Full error:
load INSTRUCTOR_Transformer
max_seq_length  512
Traceback (most recent call last):
File "C:\Users\2023\h2ogpt\src\gradio_runner.py", line 2339, in update_user_db
return _update_user_db(file, db1=db1, chunk=chunk, chunk_size=chunk_size,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\2023\h2ogpt\src\gradio_runner.py", line 2470, in _update_user_db
db = get_db(sources, use_openai_embedding=use_openai_embedding,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\2023\h2ogpt\src\gpt_langchain.py", line 97, in get_db
db = Chroma.from_documents(documents=sources,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\2023\miniconda3\Lib\site-packages\langchain\vectorstores\chroma.py", line 564, in from_documents
return cls.from_texts(
^^^^^^^^^^^^^^^
File "C:\Users\2023\miniconda3\Lib\site-packages\langchain\vectorstores\chroma.py", line 519, in from_texts
chroma_collection = cls(
^^^^
TypeError: Chroma.init() got an unexpected keyword argument 'anonymized_telemetry'

https://github.com/h2oai/h2ogpt/issues/426#issuecomment-1633241606

Which steps exactly was confusing or missing?

@Hybredm I think you must have different chroma etc.  Can you do:
pip freeze &> freeze.log

and share?
Here's mine (after a few more items.  You can also run the test:
pytest -s -v tests/test_requirements.py::test_requirements

which will test everything except peft version.
(h2ogpt) jon@pseudotensor:~/h2ogpt$ pip freeze
absl-py==1.4.0
accelerate==0.20.3
aiofiles==23.1.0
aiohttp==3.8.4
aiosignal==1.3.1
alt-profanity-check==1.2.2
altair==5.0.1
antlr4-python3-runtime==4.9.3
anyio==3.7.1
appdirs==1.4.4
APScheduler==3.10.1
argilla==1.12.0
arrow==1.2.3
arxiv==1.4.7
asttokens==2.2.1
async-timeout==4.0.2
attrs==23.1.0
Authlib==1.2.1
auto-gptq==0.2.2
backcall==0.2.0
backoff==2.2.1
beautifulsoup4==4.12.2
better-profanity==0.7.0
bioc==2.0
bitsandbytes==0.39.0
boto3==1.26.101
botocore==1.29.101
bravado==11.0.3
bravado-core==5.17.1
cachetools==5.3.1
certifi==2023.5.7
cffi==1.15.1
chardet==5.1.0
charset-normalizer==3.2.0
chromadb==0.3.25
click==8.1.4
clickhouse-connect==0.6.6
cmake==3.26.4
colorama==0.4.6
coloredlogs==15.0.1
comm==0.1.3
commonmark==0.9.1
contourpy==1.1.0
cryptography==41.0.1
cycler==0.11.0
dataclasses-json==0.5.9
datasets==2.13.0
debugpy==1.6.7
decorator==5.1.1
Deprecated==1.2.14
dill==0.3.6
diskcache==5.6.1
docker-pycreds==0.4.0
docutils==0.20.1
duckdb==0.8.1
effdet==0.4.1
einops==0.6.1
et-xmlfile==1.1.0
evaluate==0.4.0
exceptiongroup==1.1.2
execnet==2.0.2
executing==1.2.0
fastapi==0.100.0
feedparser==6.0.10
ffmpy==0.3.0
filelock==3.12.2
filetype==1.2.0
fire==0.5.0
flatbuffers==23.5.26
fonttools==4.40.0
fqdn==1.5.1
frozenlist==1.3.3
-e git+https://github.com/lm-sys/FastChat.git@807f15809c3290ab53cf8e94da2abf7b19b528d0#egg=fschat
fsspec==2023.6.0
future==0.18.3
gitdb==4.0.10
GitPython==3.1.32
google-auth==2.21.0
google-auth-oauthlib==1.0.0
gpt4all==0.3.3
gradio==3.35.2
gradio_client==0.2.7
greenlet==2.0.1
grpcio==1.56.0
h11==0.14.0
h2ogpt==0.1.0
hnswlib==0.7.0
httpcore==0.16.3
httptools==0.6.0
httpx==0.23.3
huggingface-hub==0.15.1
humanfriendly==10.0
idna==3.4
importlib-metadata==6.8.0
iniconfig==2.0.0
InstructorEmbedding==1.0.1
intervaltree==3.1.0
iopath==0.1.10
ipykernel==6.24.0
ipython==8.14.0
isoduration==20.11.0
jedi==0.18.2
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.3.1
jsonlines==3.1.0
jsonpointer==2.4
jsonref==1.1.0
jsonschema==4.18.0
jsonschema-specifications==2023.6.1
jupyter-console==6.6.3
jupyter_client==8.3.0
jupyter_core==5.3.1
kiwisolver==1.4.4
langchain==0.0.202
langchainplus-sdk==0.0.20
layoutparser==0.3.4
linkify-it-py==2.0.2
lit==16.0.6
llama-cpp-python==0.1.68
lm-dataformat==0.0.20
loralib==0.1.1
lxml==4.9.3
lz4==4.3.2
Markdown==3.4.3
markdown-it-py==2.2.0
markdown2==2.4.9
MarkupSafe==2.1.3
marshmallow==3.19.0
marshmallow-enum==1.5.1
matplotlib==3.7.1
matplotlib-inline==0.1.6
mdit-py-plugins==0.3.3
mdurl==0.1.2
monotonic==1.6
mpmath==1.3.0
msg-parser==1.2.0
msgpack==1.0.5
multidict==6.0.4
multiprocess==0.70.14
mypy-extensions==1.0.0
neptune==1.2.0
nest-asyncio==1.5.6
networkx==3.1
nh3==0.2.14
nltk==3.8.1
numexpr==2.8.4
numpy==1.23.5
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-cupti-cu11==11.7.101
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
nvidia-cufft-cu11==10.9.0.58
nvidia-curand-cu11==10.2.10.91
nvidia-cusolver-cu11==11.4.0.1
nvidia-cusparse-cu11==11.7.4.91
nvidia-ml-py==11.525.131
nvidia-nccl-cu11==2.14.3
nvidia-nvtx-cu11==11.7.91
nvitop==1.1.2
oauthlib==3.2.2
olefile==0.46
omegaconf==2.3.0
onnxruntime==1.15.1
openai==0.27.8
openapi-schema-pydantic==1.2.4
opencv-python==4.8.0.74
openpyxl==3.1.2
orjson==3.9.2
outcome==1.2.0
overrides==7.3.1
packaging==23.1
pandas==1.5.3
parso==0.8.3
pathtools==0.1.2
pdf2image==1.16.3
pdfminer.six==20221105
pdfplumber==0.9.0
peft @ git+https://github.com/huggingface/peft.git@0b62b4378b4ce9367932c73540349da9a41bdea8
pexpect==4.8.0
pickleshare==0.7.5
Pillow==10.0.0
pip-licenses==4.3.0
platformdirs==3.8.1
playwright==1.33.0
pluggy==1.2.0
portalocker==2.7.0
posthog==3.0.1
prettytable==3.8.0
prompt-toolkit==3.0.39
protobuf==3.20.0
psutil==5.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==12.0.1
pyasn1==0.5.0
pyasn1-modules==0.3.0
pycocotools==2.0.6
pycparser==2.21
pydantic==1.10.11
pydub==0.25.1
pyee==9.0.4
Pygments==2.15.1
PyJWT==2.7.0
PyMuPDF==1.22.3
pynvml==11.5.0
pypandoc==1.11
pypandoc-binary==1.11
pyparsing==3.1.0
pypdf==3.9.1
pyphen==0.14.0
PySocks==1.7.1
pytesseract==0.3.10
pytest==7.2.2
pytest-xdist==3.2.1
python-dateutil==2.8.2
python-docx==0.8.11
python-dotenv==1.0.0
python-magic==0.4.27
python-multipart==0.0.6
python-pptx==0.6.21
pytz==2023.3
PyYAML==6.0
pyzmq==25.1.0
referencing==0.29.1
regex==2023.6.3
requests==2.31.0
requests-file==1.5.1
requests-oauthlib==1.3.1
responses==0.18.0
rfc3339-validator==0.1.4
rfc3986==1.5.0
rfc3987==1.3.8
rich==13.0.1
rouge==1.0.1
rouge-score==0.1.2
rpds-py==0.8.10
rsa==4.9
s3transfer==0.6.1
sacrebleu==2.3.1
safetensors==0.3.1
scikit-learn==1.2.2
scipy==1.11.1
selenium==4.10.0
semantic-version==2.10.0
sentence-transformers==2.2.2
sentencepiece==0.1.99
sentry-sdk==1.28.0
setproctitle==1.3.2
sgmllib3k==1.0.0
shortuuid==1.0.11
simplejson==3.19.1
six==1.16.0
smmap==5.0.0
sniffio==1.3.0
sortedcontainers==2.4.0
soupsieve==2.4.1
SQLAlchemy==2.0.18
stack-data==0.6.2
starlette==0.27.0
svgwrite==1.4.3
swagger-spec-validator==3.0.3
sympy==1.12
tabulate==0.9.0
tenacity==8.2.2
tensorboard==2.13.0
tensorboard-data-server==0.7.1
termcolor==2.3.0
text-generation==0.6.0
textstat==0.7.3
threadpoolctl==3.1.0
tiktoken==0.4.0
timm==0.9.2
tokenizers==0.13.3
tomli==2.0.1
toolz==0.12.0
torch==2.0.1
torchvision==0.15.2
tornado==6.3.2
tqdm==4.65.0
traitlets==5.9.0
transformers==4.30.2
trio==0.22.1
trio-websocket==0.10.3
triton==2.0.0
typer==0.7.0
typing-inspect==0.9.0
typing_extensions==4.7.1
tzdata==2023.3
tzlocal==5.0.1
uc-micro-py==1.0.2
ujson==5.8.0
unstructured==0.7.4
unstructured-inference==0.5.1
uri-template==1.3.0
urllib3==1.26.16
uvicorn==0.22.0
uvloop==0.17.0
validators==0.20.0
Wand==0.6.11
wandb==0.15.5
watchfiles==0.19.0
wavedrom==2.0.3.post3
wcwidth==0.2.6
weaviate-client==3.20.0
webcolors==1.13
websocket-client==1.6.1
websockets==11.0.3
Werkzeug==2.3.6
wrapt==1.14.1
wsproto==1.2.0
xlrd==2.0.1
XlsxWriter==3.1.2
xxhash==3.2.0
yarl==1.9.2
zipp==3.16.0
zstandard==0.21.0

Closing, since clear has to be wrong langchain version.

source: gharchive/issue

authors: Hybredm pseudotensor

license_type: permissive

license_source: github-api


text:
Take aria attributes into consideration?

If you've used aria-label on an element that usually needs a heading, h5o now says "Untitled" whereas a screen reader would use the aria-label (or aria-labelledby) as the title. Maybe it should reflect how screen readers would parse the document?

I think this is ultimately the thing that led me to uninstalling it.
Things like <nav>, <header> and <footer> sections all saying "Untitled" whereas they actually have clear and well thought out aria-label or aria-labelledby attributes, ultimately led me to creating less than satisfactory markup (simply to please the plugin).

authors: Nettsentrisk tomkiss

license: mit

license_source: bigquery


text:
Append to csv/file on update

Is there a way to instead of running the script and it downloading and saving output to a file each time, we can just append the latest tweets into the file?  Some way of comparing what we have versus what we're missing?
Thanks

Hi and thanks for the hint, now as now you can use the since and until args to filter out a specific date range and so use the until arg of the day before of the latest tweet. The day before because you may not have been able to download all the tweets of the "latest day".
Until we don't provide a solution for your situation I suggest you to proceed in that way, hope this helps.

repo: haccer/twint

license: MIT


text:
Added button to change application stage to complete

Description

Please include a summary of the change and, if applicable, which issue is fixed. Please also include relevant motivation, context, and screenshots.

Testing

Please briefly note how you tested and verified your changes.

Reviewers

Tag one or two teammates to review your code. Ideally, you should tag people who have worked on the same files or similar features to notify them of your changes. If you pair programmed with someone, please indicate that here as well.

LGTM! Feel free to merge @ninatchang

source: gharchive/pull-request

authors: katiejiang ninatchang


text:
Add Hue Havoc

Author name
Author: Gus Ruben
About your game
What is your game about?
Competing with another player to cover the most of the canvas with your color of paint
How do you play your game?
Press any key to start. Once the game has been started, both players have 20 seconds to move around using WASD and IJKL respectively to cover as much of the canvas as possible in their paint color.
Code
Check off the items that are true.

[x] The game was made using the Sprig editor.
[x] The game is placed in the in the /games directory.
[x] The code is significantly different from all other games in the Sprig gallery (except for games labeled "demo").
[x] The game runs without errors.
[x] The name of the file/game contains only alphanumeric characters, -s, or _s.
[x] The game name is not the same as the others from gallery

Image (If an image is used)

[x] The image is in the /games/img directory.
[x] The name of the image matches the name of your file.

Thanks for your PR!

I LOVE this game- the title screen, especially is amazing. Great job- I'm merging this, and you'll see it in the gallery soon.
If you're a teenager, you can request a free Sprig console at this link and join our online Slack community to chat with other people making Sprig games.

authors: gusruben recursiveforte

repo: hackclub/sprig


text:
Add additional production settings

sync-ed  branch with other branches in hackoregon-aws-infrastructure and added the additional changes to bring all templates in github up to the production stack level.  final changes were in master.yaml and civic-lab-service/service.yaml

I don't know why, but for some reason even after merging in the other PR's, the "Files changed" tab still shows the other commits from me as changes from master, rather than exactly the same as what we now have in master (since I committed).
This is another case where it's better to branch off master and only change the things that aren't already addressed in previous PR's.
Now I'm wondering if this PR will merge correctly, or if something unusual will happen (since GitHub still reports that there are a lot more changes in this PR than is actually true)?  Ideally this would get closed and a cleaner PR committed, if that's something you're up for @iant01 (otherwise I can do it).

Closed in favour of a clean PR #10 that doesn't have the colliding/duplicate changes that are already merged to master.


text:
How to convert dseq results to dataframe?

Hello :)
After dealing with a lot of errors, I've finally got the expected table out of deseq script -
`Log2 fold change & Wald test p-value: strain S288C vs RM11
baseMean  log2FoldChange     lfcSE      stat    pvalue  padj
0     0.000000             NaN       NaN       NaN       NaN   NaN
1     0.000000             NaN       NaN       NaN       NaN   NaN
2     0.000000             NaN       NaN       NaN       NaN   NaN
3     0.160284        0.879404  2.528475  0.347800  0.727990   NaN
4     2.424527        4.707459  2.135109  2.204786  0.027469   NaN
...        ...             ...       ...       ...       ...   ...
6692  0.000000             NaN       NaN       NaN       NaN   NaN
6693  1.950290       -0.904463  1.170501 -0.772715  0.439691   NaN
6694  0.641902        1.432367  1.793678  0.798564  0.424543   NaN
6695  2.444357        0.447478  1.025930  0.436168  0.662715   NaN
6696  1.110214        0.328981  1.382793  0.237910  0.811951   NaN
[6697 rows x 6 columns]`
I now trying to move forward and filter the table by log2foldchange>1 and pvalue<0.05, but it seems that the results dataframe i'm trying to get is empty. This is my code -
`import pandas as pd
from pydeseq2.default_inference import DefaultInference
from pydeseq2.dds import DeseqDataSet
from pydeseq2.ds import DeseqStats
Read sample info into DataFrame with 'sample' column as index
sample_info_df = pd.read_csv('sample_info.tsv', sep='\t', index_col='sample')
Read count data into DataFrame (assuming counts.tsv is in the same directory)
count_data_df = pd.read_csv('counts.tsv', sep='\t', skiprows=[0])
Drop irrelevant columns
count_data_df = count_data_df.drop(columns=['Geneid', 'Chr', 'Start', 'End', 'Strand', 'Length'])
Transpose the count data
transposed_df = count_data_df.transpose()
Create an instance of DefaultInference
inference = DefaultInference(n_cpus=8)
Create a DESeqDataSet object
dds = DeseqDataSet(
counts=transposed_df,
metadata=sample_info_df,
design_factors=['batch', 'strain'],
refit_cooks=True,
inference=inference
)
Run DESeq2 analysis
dds.deseq2()
results = DeseqStats(dds, inference=inference)
print(results.summary())
summary_df = pd.DataFrame(results.summary())
filtered_results = summary_df[summary_df['pvalue'] < 0.05]`
and this is the errors i'm getting (also with log2foldchange filtering) (ignore the padj I tried to filter by, my mistake)
AttributeError: 'NoneType' object has no attribute 'summary'
File "/home/nofar/new.py", line 40, in <module>     filtered_results = results.summary()[(results.summary()['log2FoldChange'] > 1) & (results.summary()['padj'] < 0.05)]                                           ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^ TypeError: 'NoneType' object is not subscriptable
I would be happy to get some help,
Thanks in advanve, Nofar :-]

I didn't mean to upload this with these big headlines, but it's cool ><

After producing the deseq statistics (DeseqStats) one of the attributes of the returned object is the results pandas dataframe you are after
results = DeseqStats(dds, inference=inference)
print(results.results_df) # Will print the dataframe you are after
print(type(results.results_df)) # To validate that this is indeed a pd.DataFrame object

https://pydeseq2.readthedocs.io/en/latest/api/docstrings/pydeseq2.ds.DeseqStats.html#pydeseq2.ds.DeseqStats.results_df

I still get an error, and this time -
Traceback (most recent call last):   File "/home/nofar/new.py", line 37, in <module>     print(results.results_df) # Will print the dataframe you are after           ^^^^^^^^^^^^^^^^^^ AttributeError: 'DeseqStats' object has no attribute 'results_df'
I tried to change the results name, or put antother identification to it, such as: results_df = results.results, but I  still can not access to the Deseq dataframe..

Did you run the results.summary()?
results = DeseqStats(dds, inference=inference)
results.summary()
print(results.results_df) # Will print the dataframe you are after
print(type(results.results_df)) # To validate that this is indeed a pd.DataFrame object

assuming this issue is resolved

repo: hadasvolk/CompLabNGS


text:
Error: Note was not preceded by a highlight

Hello, the plugin does not work with MyClippings.txt.
Attached image of the error in the console

This is a very similar issue as https://github.com/hadynz/obsidian-kindle-plugin/issues/15. Please see my comment on if you would be open to sharing your My Clippings.txt file.
What's slightly different about your use case, is that the error you've encountered was no about being unable to parse highlight data. But not being able to associated a personal "note" that you've written against the right highlight.
I can make a fix to the plugin so it is not as "strict" in enforcing such an assumption, so the rest of your file is parsed. I'll let you know as soon as I make it.

Whilst it is not clear to me why you ran into this error (looking at your my clippings file will help me in doing so), I made an. update to the plugin to ignore the error that you just encountered in 0.1.1. This means that the rest of the highlights in your file will be parsed.
Give it a go and let me know how it goes.

I just sent the My clippings file to your email for your analysis. I will try with the update you mention, Thank you very much.

I have tested the new update, however, the plugin does not pick up all the underlines in my notes leaving some incomplete. Here is a screenshot of the console.

I would like to know, if there is a possibility for the plugin to pick up the date information when an underline or annotation was made. I understand that this can be achieved using Nunjuck but I am not proficient in Java Script so it is a challenge for me.

Thanks for emailing me your My Clippings.txt file via email.

the plugin does not pick up all the underlines in my notes leaving some incomplete.
Can you give me examples of highlights (underlines) in your file that are missing?

For example: In the file My Clippings, there is a highlight named "BBC "which is a highlight on a newspaper made whit the calibre program and sended to my kindle. It seems that maybe is an error whit that kind of books. the same goes for "Revista Crisis" or "Rebellion"

I just tested your file, and I can see all three books loaded fine. Did you mean that there are highlights within those books that are missing?

yes, the files are created but the plugin only collects a little underlining, not all the ones performed. that is, the texts are incomplete

It is possible that the highlights you are expecting are missing from your My Clippings file in the first place.
Do you mind if you give me examples of highlights that are inside My Clippings but not extracted by the plug-in?

Of corse.
This image represents the Highlights retrived by the pugin.

An this is all the highlites that I made on my kindle. This was imported manually.

You are not comparing apples to apples.
The plugin uses your My Clippings.txt file, and the data in that file only. I believe the screenshots you are looking at are from another application (Calibre?).
If you open the My Clippings.txt file that you sent me by email, you will see that "BBC Mundo (calibre)" only has 5 highlights, which is what the plugin has produced.
From what I can see so far, the plugin is parsing the data in your file as expected. Is it possible that you sent me an outdated My Clippings.txt file that doesn't have all your latest highlights?

authors: EhuddR hadynz

repo: hadynz/obsidian-kindle-plugin


text:
Distance Aware Data Aug

"During training, similar to GT-AUG (Shi, Wang, and Li 2019), we add the sampled points and bounding box into the training sample for
data augmentation." How did you do this, can you please provide an example for one frame.

I have found it at "pcdet/datasets/augmentor/database_sampler.py" called DADataBaseSampler, I think this works for the augmentation method they proposed.

authors: Soodep yanglanq

repo: hailanyi/TED


text:
Feat: 订阅剧集能分类横向展示吗？

有时候订阅的剧集很多，但是一个页面可能只显示某个剧集。

如果是某个频道横向滚动布局是不是可以省很多位置，能加个按钮改成类似这种布局吗

感谢您的建议，这部分受b站接口返回的数据限制，做成这种效果有点麻烦，不过您提出的意见后续如果可能的话将尝试实现的

authors: djkloop hakadao

repo: hakadao/BewlyBewly


text:
Fixed member opening after create/copy operations

Changes
The member URI used to open the created/copied was wrong and prevented it from being opened right after its creation.
The commands now use vscode.open and getMemberUri to open the new member.
Member copy/creation process has been enhanced:

Target name/path gets validated by vscode showInput API
User can choose to retry in case of failure
The process runs with a progress notification

Checklist

[x] have tested my change

@sebjulliand I just spotted this PR and looked at the changes - and something caught my eye: Member without extension is no longer supported after this PR.
I noticed it because I did quite some coding to ensure the extension supported exactly that. Since you can add source members without extension (actually the default on the ADDPFM command), I figured our extension should do the same, and I added some code for supporting copying to a member without extension. Are we removing this support?
Before this PR the user could add, copy to or rename to a member without extension by specifying the member name followed by a . and nothing more. The . was a requirement to make the user be deliberate in having no extension for the member and not just have forgotten to enter an extension.
It is probably rarely used, but since it's allowed on the server, not supporting this could cause trouble for some users - if they have a policy for some members having no extension (like a README or similar). They will no longer be able to use such a naming standard after this PR.

Thanks for the clarification @chrjorgensen . I didn't understand at first why the method checking a member path would look for the . but not if an actual extension was provided after the . .
I'll remove the extra control looking for the actual extension name and leave a comment there to make sure we know it's not a bug but a feature 😁

I didn't understand at first why the method checking a member path would look for the . but not if an actual extension was provided after the . .

I thought it was obvious...! 😆 No, my bad for not documenting this in the first place, and making you waste some time.

I thought it was obvious...! 😆 No, my bad for not documenting this in the first place, and making you waste some time.

Hey no worries at all, I didn't waste any time at all! It took me 2 seconds to remove that extra control and I learned a new thing 😉

authors: chrjorgensen sebjulliand

repo: halcyon-tech/vscode-ibmi


text:
Allow to configure timeouts for dns

Adding timeout configuration for dns resolver.

@alaibe @hallelujah Could you please add documentation to show examples of how to configure this?

authors: alaibe monfresh


text:
continuous integration / continuous delivery med Docker og Heroku

Skriv Dockerfile og Heroku.yml for å gjøre klart for CI/CD.
Dokumentasjon:
https://devcenter.heroku.com/articles/build-docker-images-heroku-yml
https://devcenter.heroku.com/articles/pipelines

Implementert!

Implementert!

repo: hallvors/hjemmekorps


text:
⚠ ClamAV has degraded performance

In b7acd87, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 243 ms

Resolved: ClamAV performance has improved in 6c4bedd after 11 minutes.

repo: hamboneZA/caffeine


text:
⚠ ClamAV has degraded performance

In 634793f, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 427 ms

Resolved: ClamAV performance has improved in ec6f1a8.


text:
⚠ ClamAV has degraded performance

In 2e5a5f0, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 174 ms

Resolved: ClamAV performance has improved in 96d72da.


text:
⚠ ClamAV has degraded performance

In 907743a, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 88 ms

Resolved: ClamAV performance has improved in 31b4ecc.


text:
⚠ ClamAV has degraded performance

In dee250a, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 443 ms

Resolved: ClamAV performance has improved in 1ce4a92.


text:
⚠ ClamAV has degraded performance

In eb76609, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 206 ms

Resolved: ClamAV performance has improved in af9ec7a.


text:
Bump source and target compatibility to Java 1.8

Hamcrest still supports Java 1.7 source and target compatibility.
Java 1.7 is out of support for years already, I suggest to bump to Java 1.8.
Once that is done, we can finally leverage some useful Java 1.8 features such as lambda expressions in the Hamcrest sources.
Due to quirks and differences with generic type arguments, the change is not as straightforward as it looks, but I'll take a stab at it nonetheless.
I'll prep a PR, but I'll need someone to remove the Java 1.7 compilation from the matrix job in the Travis build.

duplicates #207
see also #206

authors: edysli peterdemaeyer

repo: hamcrest/JavaHamcrest


text:
Connection Pooling

Poolboy
Poolboy works by providing exclusive access to a given resource in a "transaction".
Pros:

Dynamic connection allocation is taken care of for you.

Cons:

Connection affinity is required for cursors, which means a transaction may be required for the full length of the cursor. For long running change feeds, this will require one connection per changefeed.
Pipelining is not utilized as each connection is checked out until a response is provided.

While my assumption was to use Poolboy, it seems that there are better approaches available. There are too many tradeoffs to be made for not a whole lot of benefit.
Hand rolled
An ideal connection pool will do the following:

Connect to N clients in a cluster.
Load balance queries between connections.
Route query to master replica for the target table.
If a connection dies, remove that connection from the rotation.
If a query fails because of a server connectivity issue, retry query on another host.

Initial design:
A supervisor tree will be over N connections and a coordinator.
Client processes send queries directly to the connection, just like we currently do. In order to know which connection to contact, the client can request a connection from the coordinator. When the client is finished with the query it informs the coordinator (in order for the coordinator to properly balance connections).
This would require zero changes to Exrethinkdb.Connection. The connection would be dumb to pooling. We could provide a Exrethinkdb.ConnectionPool module with run/next/close that wrap the logic for interacting with the coordinator. The end API for the user would be the same. We will also provide a use Exrethinkdb.ConnectionPool macro. Ideally you would replace use Exrethinkdb.Connection with use Exrethinkdb.ConnectionPool and your application would still work perfectly.
Routing queries to proper replicas can be done fairly well in the coordinator by looking at table("foo_table") |> config |> run. The coordinator can do this on a timer (once every minute is probably sufficient). It can then use this information to route the connection properly. In the event of stale_reads, the coordinator can load balance between non-master connections. We'll have to add table to the Query struct, but that's pretty straightforward.
Exrethinkdb.ConnectionPool.run will report back to the coordinator the success or failure of a query.  The coordinator will also be monitoring the Connection processes. There will also be logic to retry on a different server if a failure occurs (in the event of stale_reads. otherwise, retrying won't be useful until a new master is selected).
I think this custom approach will be fairly simple. It will be designed so that failure leads to a worst case scenario of routing requests to the less optimal replica.

Their is an issue open for Ecto that solves this by "reserving" a worker
https://github.com/elixir-lang/ecto/issues/569 I'd be pretty strongly
against hand rolling because this kind of thing is full of pitfalls. We
could also ping @fishcakez and see if maybe his sbroker might be a stronger
use case here.
On Wed, May 6, 2015 at 11:15 PM, Peter Hamilton notifications@github.com
wrote:

Poolboy
Poolboy works by providing exclusive access to a given resource in a
"transaction".
Pros:

Dynamic connection allocation is taken care of for you.

Cons:

Connection affinity is required for cursors, which means a
transaction may be required for the full length of the cursor. For long
running change feeds, this will require one connection per changefeed.
Pipelining is not utilized as each connection is checked out until a
response is provided.

While my assumption was to use Poolboy, it seems that there are better
approaches available. There are too many tradeoffs to be made for not a
whole lot of benefit.
Hand rolled
An ideal connection pool will do the following:

Connect to N clients in a cluster.
Load balance queries between connections.
Route query to master replica for the target table.
If a connection dies, remove that connection from the rotation.
If a query fails because of a server connectivity issue, retry
query on another host.

Initial design:
A supervisor tree will be over N connections and a coordinator.
Client processes send queries directly to the connection, just like we
currently do. In order to know which connection to contact, the client can
request a connection from the coordinator. When the client is finished with
the query it informs the coordinator (in order for the coordinator to
properly balance connections).
This would require zero changes to Exrethinkdb.Connection. The connection
would be dumb to pooling. We could provide a Exrethinkdb.ConnectionPool
module with run/next/close that wrap the logic for interacting with the
coordinator. The end API for the user would be the same. We will also
provide a use Exrethinkdb.ConnectionPool macro. Ideally you would replace use
Exrethinkdb.Connection with use Exrethinkdb.ConnectionPool and your
application would still work perfectly.
Routing queries to proper replicas can be done fairly well in the
coordinator by looking at table("foo_table") |> config |> run. The
coordinator can do this on a timer (once every minute is probably
sufficient). It can then use this information to route the connection
properly. In the event of stale_reads, the coordinator can load balance
between non-master connections. We'll have to add table to the Query
struct, but that's pretty straightforward.
Exrethinkdb.ConnectionPool.run will report back to the coordinator the
success or failure of a query. The coordinator will also be monitoring the
Connection processes. There will also be logic to retry on a different
server if a failure occurs (in the event of stale_reads. otherwise,
retrying won't be useful until a new master is selected).
I think this custom approach will be fairly simple. It will be designed so
that failure leads to a worst case scenario of routing requests to the less
optimal replica.
--
Reply to this email directly or view it on GitHub
https://github.com/hamiltop/exrethinkdb/issues/32.

I think table affinity is not going to be provided by existing solutions.
Table affinity will cut down on network bottlenecks on the server side.
The ecto solution solves a very different problem of nested transactions.
The solution they use (correct me if I'm wrong) is to permanently checkout
a connection for a long lived process to use. The alternatives proposed are
to provide the worker in the gen_call.
We are dealing with a very different set of constraints. The unit of
exclusivity for a postgres connection is a full query/response or even a
full transaction. For rethinkdb it's a query or a response. Since we can
interleave queries and responses we only need exclusive access in very
small chunks and the current implementation does it very well.
The performance bottleneck is purely server side. We will not have nor want
multiple connections to the same host.
The hand rolled proposal is built on top of independently useful building
blocks. The current connections are already done and work well. The
coordinator is going to be necessary for table affinity in any solution we
come across.
The ideal drop in solution would be something that:

keep track of established connections.
reconnect broken connections.

Number 2 is exactly what a supervisor is for. This is already an out of the
box solution. The only problem we need to solve is how a client find out
which connections currently are alive. I'm fairly certain which_children
will go a long way in solving that.
I'm going to take a very lightweight pass at this and see how difficult it
actually is.
On Thu, May 7, 2015, 8:26 AM Jason S. notifications@github.com wrote:

Their is an issue open for Ecto that solves this by "reserving" a worker
https://github.com/elixir-lang/ecto/issues/569 I'd be pretty strongly
against hand rolling because this kind of thing is full of pitfalls. We
could also ping @fishcakez and see if maybe his sbroker might be a stronger
use case here.
On Wed, May 6, 2015 at 11:15 PM, Peter Hamilton notifications@github.com
wrote:

Poolboy
Poolboy works by providing exclusive access to a given resource in a
"transaction".
Pros:

Dynamic connection allocation is taken care of for you.

Cons:

Connection affinity is required for cursors, which means a
transaction may be required for the full length of the cursor. For long
running change feeds, this will require one connection per changefeed.
Pipelining is not utilized as each connection is checked out until a
response is provided.

While my assumption was to use Poolboy, it seems that there are better
approaches available. There are too many tradeoffs to be made for not a
whole lot of benefit.
Hand rolled
An ideal connection pool will do the following:

Connect to N clients in a cluster.
Load balance queries between connections.
Route query to master replica for the target table.
If a connection dies, remove that connection from the rotation.
If a query fails because of a server connectivity issue, retry
query on another host.

Initial design:
A supervisor tree will be over N connections and a coordinator.
Client processes send queries directly to the connection, just like we
currently do. In order to know which connection to contact, the client
can
request a connection from the coordinator. When the client is finished
with
the query it informs the coordinator (in order for the coordinator to
properly balance connections).
This would require zero changes to Exrethinkdb.Connection. The connection
would be dumb to pooling. We could provide a Exrethinkdb.ConnectionPool
module with run/next/close that wrap the logic for interacting with the
coordinator. The end API for the user would be the same. We will also
provide a use Exrethinkdb.ConnectionPool macro. Ideally you would
replace use
Exrethinkdb.Connection with use Exrethinkdb.ConnectionPool and your
application would still work perfectly.
Routing queries to proper replicas can be done fairly well in the
coordinator by looking at table("foo_table") |> config |> run. The
coordinator can do this on a timer (once every minute is probably
sufficient). It can then use this information to route the connection
properly. In the event of stale_reads, the coordinator can load balance
between non-master connections. We'll have to add table to the Query
struct, but that's pretty straightforward.
Exrethinkdb.ConnectionPool.run will report back to the coordinator the
success or failure of a query. The coordinator will also be monitoring
the
Connection processes. There will also be logic to retry on a different
server if a failure occurs (in the event of stale_reads. otherwise,
retrying won't be useful until a new master is selected).
I think this custom approach will be fairly simple. It will be designed
so
that failure leads to a worst case scenario of routing requests to the
less
optimal replica.
--
Reply to this email directly or view it on GitHub
https://github.com/hamiltop/exrethinkdb/issues/32.

—
Reply to this email directly or view it on GitHub
https://github.com/hamiltop/exrethinkdb/issues/32#issuecomment-99909472.

@hamiltop after talking with @ericmj and @fishcakez they've convinced me that maybe we don't need connection pooling and that I am probably wrong.

Table affinity is what's got me convinced I want to do some level of
connection management. I hacked together a lightweight naive pool and the
code is dead simple.
I skimmed the conversation in irc and I think your previous comment here
about this being a smarter client then something like postgres is accurate
and changes the discussion a bit.
I'll hopefully have my code up soon.
On Thu, May 7, 2015, 9:07 AM Jason S. notifications@github.com wrote:

@hamiltop https://github.com/hamiltop after talking with @ericmj
https://github.com/ericmj and @fishcakez https://github.com/fishcakez
they've convinced me that maybe we don't need connection pooling and that I
am probably wrong.
—
Reply to this email directly or view it on GitHub
https://github.com/hamiltop/exrethinkdb/issues/32#issuecomment-99923747.

https://github.com/hamiltop/exrethinkdb/blob/connection_pool/lib/exrethinkdb/connection/pool.ex
used like:
hosts = [[host: "localhost", port: 28015], [host: "localhost", port: 28015]]
{:ok, conn_pool} = Exrethinkdb.Connection.Pool.start_link([hosts: hosts])
conn = Exrethinkdb.Connection.Pool.get_connection(conn_pool)
Exrethinkdb.Query.table("people") |> Exrethinkdb.Connection.run conn
# alternatively, run from the conn_pool and it will pick a connection for you
Exrethinkdb.Query.table("people") |> Exrethinkdb.Connection.Pool.run conn_pool

It's naive, but accomplishes the basic goals. The optimizations would be in how get_connection works in order to provide table affinity.
As an already-implemented, cursors store the pid of their connection internally and will use that connection when next is called.

@hamiltop,  I really like the ideas that you've laid out for the hand rolled solution because it takes into account the unique characteristics of RethinkDB.  Poolboy, and other technologies are going to make assumptions that are part of a different paradigm.  I also like the fact that you were able to build a working version based on the code that's already been written.  As @jeregrine mentioned, there are bound to be Gremlins lurking in the dark, but that seem to be the case for just about everything in this line of work.  At this point, I can't think of a good reason not to run with your solution.

Some interesting results:
Three configurations:

Single local connection (single)
A Pool of 4 connections to localhost (pool)
A pool of 4 connections to localhost plus 1 connection to a remote host (cluster). The remote host has significant network latency (this is going from my laptop to my iMac over WiFi).

All of these were run with use_outdated: true. Without allowing outdated reads, the cluster mode has no real benefit if you aren't sharding your data. If you shard your data, the results are similar to use_outdated: true in that the clustering has benefit.
The first test was a non-indexed read in a table of 10k records (looking for one record), repeated 40 times across 40 Tasks for a total of 1600 queries and 1.6M reads (non-indexed reads count one read per record). This was very much CPU bound in rethinkdb.
Real results now
{:pool, {24016272, :ok}}
{:single, {23215845, :ok}}
{:cluster, {17805260, :ok}}
Again, reversed
{:cluster, {18998542, :ok}}
{:single, {23591027, :ok}}
{:pool, {23255329, :ok}}

The next test was for an indexed get in the same table, but repeated 4000 times across 40 Tasks for a total of 160000 queries and 160000 reads (one read per get). This was CPU bound on the elixir side.
Real results now
{:pool, {19722294, :ok}}
{:single, {20719733, :ok}}
{:cluster, {35041397, :ok}}
Again, reversed
{:cluster, {30579648, :ok}}
{:single, {20782334, :ok}}
{:pool, {20139856, :ok}}

Conclusions
For heavy non-indexed reads (or anything CPU bound in the DB) clustering is a benefit. For heavy indexed reads (or anything light on CPU in the DB) clustering slows things down (or at best doesn't really help much).
Additionally, fine tuning the pool is important. When I ran the cluster test with 1 local and 1 remote connection it was much slower than when I ran it with 4 local and 1 remote. This was in a very lopsided network, but I've encountered lopsided clusters aplenty in production environments.
I think the takeaway is that a developer will need to weight hosts in the pool according to their requirements. The configuration should be something like:
[
  [host: "db1", port: 28015, weight: 2],
  [host: "db2", port: 28015, weight: 3],
  [host: "db3", port: 28015, weight: 1]
]

Additionally, table affinity is definitely required. Hitting the connection that isn't the master for the table was costly.

The results are exactly what I would expect them to be given the parameters you described.  In my mind this seems to indicate that initial design is working properly.  This will add the flexibility needed to get the most out of RethinkDB and the application.  A couple of things that I was thinking about were pools that had equally weighted hosts.  For example, if you have a RethinkDB cluster in data-center A, and another RethinkDB Cluster in data-center B, it seems logical to have the following configuration:
[
[host: "dca_db1", port: 28015, weight: 0],
[host: "dca_db2", port: 28015, weight: 0],
[host: "dca_db3", port: 28015, weight: 0]
[host: "dcb_db1", port: 28015, weight: 1],
[host: "dcb_db2", port: 28015, weight: 1],
[host: "dcb_db3", port: 28015, weight: 1]
]
This would allow the application servers in data-center A to utilize the local network for best performance, but fall back to the cluster in data-center B.  Given that they are equally weighted in each group, I would expect that the load would be balanced among them.
I have been kicking around the idea of adding tags that could be used to specify things like racks, data-centers, etc, but the weight parameter may cover most cases.  At this point I'm not sure if they would help or hurt, but I thought the idea was at least worth mentioning.
--Nick

weight: 0 would need to be a special case, as otherwise weights are
traditionally applied as P(use this host) = (weight of host)/sum(weights)
which in the case of weight 0 would be 0% probability. We could probably
special case it so that is sum(weights) is 0 the we just round robin.
I did find a bug in some code not pushed yet. :random.uniform has a state
local to each process. The current code (my new code has not been pushed
yet) resulted in all processes using the same round robin ordering of
connections. The tests were very useful in exposing that.
On Fri, May 8, 2015 at 1:29 PM Nick notifications@github.com wrote:

The results are exactly what I would expect them to be given the
parameters you described. In my mind this seems to indicate that initial
design is working properly. This will add the flexibility needed to get the
most out of RethinkDB and the application. A couple of things that I was
thinking about were pools that had equally weighted hosts. For example, if
you have a RethinkDB cluster in data-center A, and another RethinkDB
Cluster in data-center B, it seems logical to have the following
configuration:
[
[host: "dca_db1", port: 28015, weight: 0],
[host: "dca_db2", port: 28015, weight: 0],
[host: "dca_db3", port: 28015, weight: 0]
[host: "dcb_db1", port: 28015, weight: 1],
[host: "dcb_db2", port: 28015, weight: 1],
[host: "dcb_db3", port: 28015, weight: 1]
]
This would allow the application servers in data-center A to utilize the
local network for best performance, but fall back to the cluster in
data-center B. Given that they are equally weighted in each group, I would
expect that the load would be balanced among them.
I have been kicking around the idea of adding tags that could be used to
specify things like racks, data-centers, etc, but the weight parameter may
cover most cases. At this point I'm not sure if they would help or hurt,
but I thought the idea was at least worth mentioning.
--Nick
—
Reply to this email directly or view it on GitHub
https://github.com/hamiltop/exrethinkdb/issues/32#issuecomment-100353032
.

You have a couple of issues regarding failure handling with the proposed pooling method:
The case where all connections are unavailable isn't handled. This could happen if all connections are :restarting, in which case a client will crash when generating the random integer. Perhaps that is desired.
A supervisor blocks while starting a child. In the current implementation connect and handshake occur in the init/1 callback. This means that the supervisor/the pool is blocked and all client processes requesting a worker are blocked too while a single connection process starts.
If one of the rethinkdb servers is down or there is a issue with connecting/sending data, a connection process will fail to connect. This will either result in the supervisor blocking for a significant time while a connection process tries to start (as above) or more likely the supervisor will shutdown because the max_restarts limit was reached because a connection process keeps failing to start in rapid succession.
Finally the poolling strategy does not provide any backpressure or load shedding to client processes beyond the connection processes' message queues. Designing around a minimal number of connection processes that pipeline requests makes this is a potential bottleneck.

Thanks for chiming in @fishcakez
The case for "unable to acquire a connection" has not been addressed at any level yet, but either a "let it crash" philosophy or responding with {:error, msg} should suffice for both the individual connection case and the pool case.
To check my understanding, the init function of a GenServer is run as part of GenServer.start_link and therefore it causes the supervisor to block. Is it a common pattern to send oneself a :connect message to be processed immediately after init? How is this otherwise handled?
How is the max_restarts issue (where one buggy child kills all the other healthy children as it brings down the supervisor) solved elsewhere? By not using supervision?
Backpressure is applied to individual clients through GenServer.call. The client process will block until it receives a response. This does nothing to address an unbounded number of client processes. It also does not deal with load shedding, as stated. Would a simple solution be for an individual connection to keep track of active clients and respond :too_busy? The client should then try another connection or in the case of not using a connection pool just fail. Change feeds would need to be accounted for differently because they don't have the same resource footprint as active connections.
While I'm exploring the idea of a custom pool with table affinity, I'm committed to making Connection compatible with existing connection pooling solutions like poolboy. Not using pipelining isn't the end of the world, and this is how it is done in other languages. This custom pooling is an optimization. Perhaps there's a hybrid solution that can be found.

" We could probably special case it so that is sum(weights) is 0 the we just round robin."
-- That is the behavior that I had in mind with my sample configuration.  Would it be possible to round robin any weights that are the same?  In my example config above, the idea was to round robin the primary cluster, then fail over to another cluster and round robing those connections.  If that's not possible, I suppose the config could look like this, as long as 0 was selected before the higher number weights:
[
[host: "dca_db1", port: 28015, weight: 0],
[host: "dca_db2", port: 28015, weight: 0],
[host: "dca_db3", port: 28015, weight: 0]
[host: "dcb_db1", port: 28015, weight: 1],
[host: "dcb_db2", port: 28015, weight: 2],
[host: "dcb_db3", port: 28015, weight: 3]
]
"This custom pooling is an optimization. Perhaps there's a hybrid solution that can be found."
-- Darn those gremlins :)

The only two extension points I'd like are:

Oversubscribing checkouts and connections so we can take advantage of
pipelining.
Custom pooling strategies (so table affinity can be done).

It may be worth seeing if an existing solution can be adapted. It may also
be useful to dive down the rabbit hole and see how riak does client side
routing.
On Fri, May 8, 2015 at 4:10 PM Nick notifications@github.com wrote:

" We could probably special case it so that is sum(weights) is 0 the we
just round robin."
-- That is the behavior that I had in mind with my sample configuration.
Would it be possible to round robin any weights that are the same? In my
example config above, the idea was to round robin the primary cluster, then
fail over to another cluster and round robing those connections. If that's
not possible, I suppose the config could look like this, as long as 0 was
selected before the higher number weights:
[
[host: "dca_db1", port: 28015, weight: 0],
[host: "dca_db2", port: 28015, weight: 0],
[host: "dca_db3", port: 28015, weight: 0]
[host: "dcb_db1", port: 28015, weight: 1],
[host: "dcb_db2", port: 28015, weight: 2],
[host: "dcb_db3", port: 28015, weight: 3]
]
"This custom pooling is an optimization. Perhaps there's a hybrid solution
that can be found."
-- Darn those gremlins :)
—
Reply to this email directly or view it on GitHub
https://github.com/hamiltop/exrethinkdb/issues/32#issuecomment-100391625
.

https://github.com/inaka/worker_pool/compare/master...marianoguerra:issue_16_custom_strategies seems to be interesting. I'd rather use Poolboy just for experience and expertise within the Elixir ecosystem.
#1 could probably be accomplished via some hack layer on top of a custom strategy. Something like "when creating a worker, wrap an existing connection if X and Y else start a new connection".

Not using pipelining isn't the end of the world, and this is how it is done in other languages.

You can still pipeline without using pools, either by making the pipelining explicit or by having a Task like API where you can wait for multiple pipelined queries to complete.

Tasks are something I'd like to add anyway.
I don't think I'll be intellectually satisfied without oversubscription,
but we can get to practical and useful without it.
As far as table affinity, anything wrong with N pools, one per server? Then
we can bolt a layer on top pointing the client to the correct pool for a
given query (with fallback to any pool if needed).
On Sat, May 9, 2015, 4:30 AM Eric Meadows-Jönsson notifications@github.com
wrote:

Not using pipelining isn't the end of the world, and this is how it is
done in other languages.
You can still pipeline without using pools, either by making the
pipelining explicit or by having a Task like API where you can wait for
multiple pipelined queries to complete.
—
Reply to this email directly or view it on GitHub
https://github.com/hamiltop/exrethinkdb/issues/32#issuecomment-100467326
.

Re: Tasks
https://github.com/hamiltop/exrethinkdb/issues/33

To check my understanding, the init function of a GenServer is run as part of GenServer.start_link and therefore it causes the supervisor to block. Is it a common pattern to send oneself a :connect message to be processed immediately after init? How is this otherwise handled?

Exactly how you describe, casting a reconnect message to self in init/1 is very common.

How is the max_restarts issue (where one buggy child kills all the other healthy children as it brings down the supervisor) solved elsewhere? By not using supervision?

The fault tolerance in OTP comes about by isolating errors at suitable levels. A supervision tree does this in layers, errors slowly bubble up the supervision tree as processes keep crashing. Hopefully at some point as close to the bottom of the tree as possible the state is fixed and normal service resumes.
Not being able to connect to the database (network partition) is a realistic error and it is reasonable to handle it. Ideally this error is contained so that connections to other databases are not effected. If the other connections are shutdown because the max restart limit is reached the fault has spread further than is necessary.
A circuit breaker or fuse type system (such as https://github.com/jlouis/fuse) can be used to raise an alarm (http://www.erlang.org/doc/man/alarm_handler.html) and throttle retries, possibly backing off between attempts (such as https://github.com/ferd/backoff).

Would a simple solution be for an individual connection to keep track of active clients and respond :too_busy?

Yes.

I'm committed to making Connection compatible with existing connection pooling solutions like poolboy.

You could try a slightly different take on the usual semantics if the max_overflow is set to 0. The poolboy transaction need only be to setup the asynchronous request (or task as in #33). It would not have to remain for the lifetime of the request (i.e. checkin the connection and then await the result). I am unsure what performance would be like but it might be interesting to try as it would not require changing your Connection process.

The original goal of connection pooling pre 1.0.0 was that Exrethinkdb.Connection should work without any modifications with a connection pooling system. I think this discussion has been very useful. I think with a few modifications to Exrethinkdb.Connection, getting poolboy to work is just a matter of following some guidelines around configuration.
Modes of use

Traditional Pool. Check out a connection and check it back in when you are done. For changefeeds and other queries that are tied to specific connections, you need to keep the connection checked out until you close the feed. Otherwise, that connection might be terminated or restarted. You can, however, use that checked out connection in as many processes as you like. This will be especially useful for tasks.
Static Pool. max_overflow is set to 0. When you check out a connection you can check it back in immediately. After checking it in, you are still fully enabled to make requests on that connection. Changefeeds and other queries just work, no need to worry about hogging the connections in the pool.

I think the original goal of solving identified problems and then providing an example of how to use it with poolboy is still the correct approach. The other issue discussed here is table affinity and I'm not going to include smart routing in 1.0.0 for now.
Identified problems

Blocking in init. New issue opened at https://github.com/hamiltop/exrethinkdb/issues/34
Connection failure. New issue opened at https://github.com/hamiltop/exrethinkdb/issues/35
Throttling/Back Pressure/Load Shedding. New issue opened at https://github.com/hamiltop/exrethinkdb/issues/36

Any discussions on those topics should be moved to the issues opened. Any new issues can and should be discussed in this thread.

I'm going to reduce this problem down to a fairly simple one. Table affinity is not necessary with a rethinkdb-proxy. Throttling/Back Pressure/Load Shedding is still a problem, as is handling timeouts properly. It should work with poolboy out of the box, but that has yet to be tested.

RethinkDB.Connection works fine with poolboy right now:
pool_options = [name: {:local, :hello_pool}, worker_module: RethinkDB.Connection, size: 10, max_overflow: 10]
:poolboy.start_link(pool_options, [])
a = :poolboy.checkout(:hello_pool)
RethinkDB.Query.table_list |> RethinkDB.Connection.run(a)

Some advisable defaults:
size == max_overflow
Cursors, changefeeds, and some other queries are tied to a connection process. If that connection is terminated, then the query will fail. By setting them equal, you can be a little loose about mutual exclusion and check a connection back in even though you are still using it. The driver itself won't care, it properly multiplexes requests.
Changefeeds
Since changefeeds are persistent, mixing them into the pool makes for some difficult balancing. It might be worthwhile to create a separate pool for changefeeds in your application.
I may create a RethinkDB.Pool library with theses defaults, but I don't want to make the driver dependent on poolboy.

Removing 1.0.0 milestone. The goal was to have a connection that worked easily with poolboy, which is what we have.

Recently discovered that RethinkDB has core <-> connection affinity. So while the Elixir side scales well with multiple clients, a single connection is going to perform slightly worse on the server side. Connection pooling should be a priority.

Agreed that Connection pooling should be a priority, it is kinda frustrating without this feature

How is it frustrating? Can you provide some examples?
On Mon, Apr 18, 2016, 2:56 AM Leng Sheng Hong notifications@github.com
wrote:

Agreed that Connection pooling should be a priority, it is kinda
frustrating without this feature
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/hamiltop/rethinkdb-elixir/issues/32#issuecomment-211305076

I updated the README to show how to use poolboy out of the box.

I have been working on a adapter for Ecto for a while (:rethinkdb_ecto).
Currently it supports most of the Ecto DSL (including basic support for joins), storage and migrations.
As Ecto 2.0 requires adapters to support DBConnection, I rewrote RethinkDB.Connection to implement the DBConnection behavior. It is almost ready now (SSL support missing, need to rewrite some tests).
Basically, DBConnection uses Connection as the default pool implementation. It also has built-in support for :poolboy and :sbroker. Which means that moving from  Connection to DBConnection should not break things.
Right now, the only thing you have to do to get started is to replace
worker(RethinkDB.Connection, [[name: :foo]])

with
supervisor(RethinkDB.Connection, [[name: :foo]])

To use a different pool implementation (:poolboy for example), we could implement something like this:
defmodule Repo do
  use RethinkDB.Connection, pool: DBConnection.Poolboy
end

If you are interested in using DBConnection, I will get my fork ready (fix a few things) and create a pull request.
Note that i also implemented the new V1_0 handshake (RethinkDB >= 2.3.0) with support for user authentication (SCRAM). I skipped both V0_3 and V0_4 which may be required to connect to older version of RethinkDB.

Awesome! I appreciate the help.
My concerns are:

No required dependency on Poolboy or sbroker. We should make those
optional (shouldn't be hard).
Should work with 2.0.0+
Persistent checkouts from the pool and oversubscription are a priority.
One changefeeds per connection is not going to scale well. We need a
mechanism to oversubscribe. Poolboy doesn't really handle that but sbroker
might. A possible convention might be to create two pools one for
changefeeds and one for normal queries.

Otherwise, please submit the PR! Love the help.
On Tue, Apr 19, 2016, 5:31 AM Mario Flach notifications@github.com wrote:

I have been working on a adapter for Ecto for a while (:rethinkdb_ecto
https://github.com/almightycouch/rethinkdb_ecto).
Currently it supports most of the Ecto DSL (including basic support for
joins), storage and migrations.
As Ecto 2.0 requires adapters to support DBConnection, I rewrote
RethinkDB.Connection to implement the DBConnection behavior
https://github.com/almightycouch/rethinkdb-elixir/commit/34e53e505026c2764c549d0c31916bd28437494f.
It is almost ready now (SSL support missing, need to rewrite some tests).
Basically, DBConnection uses Connection as the default pool
implementation. It also has built-in support for :poolboy and :sbroker.
Which means that moving from Connection to DBConnection should not break
things.
Right now, the only thing you have to do to get started is to replace
worker(RethinkDB.Connection, [[name: :foo]])
with
supervisor(RethinkDB.Connection, [[name: :foo]])
To use a different pool implementation (:poolboy for example), we could
implement something like this:
defmodule Repo do
use RethinkDB.Connection, pool: DBConnection.Poolboyend
If you are interested in using DBConnection, I will get my fork ready
(fix a few things) and create a pull request.
Note that i also implemented the new V1_0 handshake (RethinkDB >= 2.3.0)
with support for user authentication (SCRAM). I skipped both V0_3 and V0_4
which may be required to connect to older version of RethinkDB.
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/hamiltop/rethinkdb-elixir/issues/32#issuecomment-211902104

DBConnection does provide pooling (and poolboy and sbroker are optional dependencies) and the behaviour with pooling should be the same as without pooling, it just a case of adding the pool: module option to choose the pool. This means a connection is always limited to one client process.
Ecto 2.0 does not require all adapters to use DBConnection, only the SQL adapters (as DBConnection is required for the (concurrent) sandbox).
I have no idea about changefeeds but I assume they are approximately pubsub. This won't play well with DBConnection unfortunately because the socket is passed to the client process on each call. Postgrex uses a different process to support postgres notifications.
I am not sure what the best strategy for pooling changefeeds is. The simplest strategy might be to use poolboy and only carry out the subscription to a changefeed in the poolboy transaction (checkin/checkout). Then remain subscribed to the changefeed after checking in the process to poolboy. Likely it would be a good idea to use a :fifo queue otherwise one changefeed may end up with all the subscriptions. This would allow pooling and not pooling changefeeds. I am not sure if this will be good enough but hopefully it is. Whereas using sbroker would mean always pooling.

Yeah, I noticed that DBConnection passing the socket to the client process means we can't multiplex queries.
In an ideal world I think a tunable "oversubscription" setting would be what we want. Then maintain two pools, one for changefeeds and one for normal queries. Even in the query case, oversubscription on the connection is efficient.

Shackle is a pooling library that supports multiplexing: https://github.com/lpgauth/shackle. It works quite differently to DBConnection but provides similar abstractions.

Is your fork ready for a discussion of merging into the mainline?

Hi @hamiltop, The fork is ready to be merged without conflics.
All the tests are passing successfully with the exception of 5 tests in test/connection_test.exs which must be rewritten to work with the DBConnection behaviour. Some new test should also be written to test the connection pools.
There are a few implementation details that should be discussed thought:

The connection logic is implemented in the Multiplexer module.

Each connection has two processes, for sending and receiving (see recv_loop/2).
Calling send_recv/3 function blocks until the response is received (or connection timeout) without blocking the connection. This is done using GenServer.call/3 and GenServer.reply/3.
Only the latest authentication method (server >= 2.3) has been implemented. Older authentication methods must be implemented if required.

To use a specific pool backend, pass the module to the :pool option:

Within a supervision tree: supervisor(MyApp.Repo, [[pool: DBConnection.Poolboy]])
For available options, check: DBConnection.start_link/2.
Default is DBConnection.Connection, :poolboy and :sbroker are optional dependencies.

Opened #106 to discuss

authors: Linicks almightycouch darkredz ericmj fishcakez hamiltop jeregrine

repo: hamiltop/exrethinkdb


text:
最大熵模型的依存句法分析没有训练接口吗？代码中只看到直接加载训练好的模型

注意事项
请确认下列注意事项：

我已仔细阅读下列文档，都没有找到答案：

首页文档
wiki
常见问题

我已经通过Google和issue区检索功能搜索了我的问题，也没有找到答案。
我明白开源社区是出于兴趣爱好聚集起来的自由社区，不承担任何责任或义务。我会礼貌发言，向每一个帮助我的人表示感谢。
[x] 我在此括号内输入x打钩，代表上述事项确认完毕。

版本号

当前最新版本号是：
我使用的版本是：1.6.8

我的问题

复现问题

步骤

首先……
然后……
接着……

触发代码
    public void testIssue1234() throws Exception
    {
        CustomDictionary.add("用户词语");
        System.out.println(StandardTokenizer.segment("触发问题的句子"));
    }

期望输出

期望输出

实际输出

实际输出

其他信息

你好，请参考：https://github.com/hankcs/HanLP/blob/136da834f0a1a92987d733e67d0560c4a80a144e/src/main/java/com/hankcs/hanlp/corpus/dependency/model/MaxEntDependencyModelMaker.java

你好，请参考：https://github.com/hankcs/HanLP/blob/136da834f0a1a92987d733e67d0560c4a80a144e/src/main/java/com/hankcs/hanlp/corpus/dependency/model/MaxEntDependencyModelMaker.java

收到，谢谢

@hankcs 我看到此dependency/model路径下没有NeuralNetworkDependencyParser的训练接口，是没有实现吗

@hankcs 我看到此dependency/model路径下没有NeuralNetworkDependencyParser的训练接口，是没有实现吗

没有实现。深度学习时代手写的矩阵运算没有实用价值。

感谢您对HanLP1.x的支持，我一直为没有时间回复所有issue感到抱歉，希望您提的问题已经解决。或者，您可以从《自然语言处理入门》中找到答案。
时光飞逝，HanLP1.x感谢您的一路相伴。我于东部标准时间2019年12月31日发布了HanLP1.x在上一个十年最后一个版本，代号为最后的武士。此后1.x分支将提供稳定性维护，但不是未来开发的焦点。
值此2020新年之际，我很高兴地宣布，HanLP2.0发布了。HanLP2.0的愿景是下一个十年的前沿NLP技术。为此，HanLP2.0采用TensorFlow2.0实现了最前沿的深度学习模型，通过精心设计的框架支撑下游NLP任务，在海量语料库上取得了最前沿的准确率。作为第一个alpha版本，HanLP 2.0.0a0支持分词、词性标注、命名实体识别、依存句法分析、语义依存分析以及文本分类。而且，这些功能并不仅限中文，而是面向全人类语种设计。HanLP2.0提供许多预训练模型，而终端用户仅需两行代码即可部署，深度学习落地不再困难。更多详情，欢迎观看HanLP2.0的介绍视频，或参与论坛讨论。
展望未来，HanLP2.0将集成1.x时代继承下来的高效率务实风范，同时冲刺前沿研究，做工业界和学术界的两栖战舰，请诸君继续多多指教，谢谢。

authors: colin0000007 hankcs mfxss

repo: hankcs/HanLP


text:
Add stopping criteria for gradio model_worker

Added stopping criteria for model_worker to eliminate self-repeating responses in gradio demo.

Hi, thanks for the contribution.
I wonder if you can provide which model, an example image and instruction for us to test?
Also, is it related to https://github.com/haotian-liu/LLaVA/issues/1086#issuecomment-1928936542?
Thank you again!

I was testing on llava-v1.6-34b but this would fix every model worker that's spawned using serve/model_worker.py, including #1086.
The first example image and prompt (ironing on taxi) on llava-v1.6-34b should be good for testing purposes, the model would start talking about random stuff and adding random symbols after answering the actual question.

Hi, can you share the command you used to spawn the workers/demo servers? I could not reproduce that on my side. Thanks.

Sorry I did't mentioned the quantizations.
CUDA_VISIBLE_DEVICES=0,1 python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path /data/0001.llava-v1.6-34b/ --load-4bit

Yes, the demo your link doesn't seems to have the repetition issue.

Some extra information of my environment with reproducible repetition on latest code: Nvidia A10*4 with Driver Version: 545.23.08    CUDA Version: 12.3, clip_vit manually downloaded from huggingface and changed vision_tower in config file to relink. I haven't dig deeper yet but seems to me the problem is coming from this commit not working for some cases and I'm just reverting it.
env.txt

one thing to debug. can you share your gradio_web_server logs?

`2024-02-07 01:06:54 | INFO | gradio_web_server | load_demo. ip: 127.0.0.1
2024-02-07 01:06:54 | INFO | gradio_web_server | Models: []
2024-02-07 01:06:54 | ERROR | stderr | /root/miniconda3/envs/llava/lib/python3.10/site-packages/gradio/components/dropdown.py:163: UserWarning: The value passed into gr.Dropdown() is not in the list of choices. Please update the list of choices to include:  or set allow_custom_value=True.
2024-02-07 01:06:54 | ERROR | stderr |   warnings.warn(
2024-02-07 01:09:12 | INFO | gradio_web_server | load_demo. ip: 127.0.0.1
2024-02-07 01:09:12 | INFO | gradio_web_server | Models: ['0001.llava-v1.6-34b']
2024-02-07 01:09:15 | INFO | gradio_web_server | add_text. ip: 127.0.0.1. len: 33
2024-02-07 01:09:16 | INFO | gradio_web_server | http_bot. ip: 127.0.0.1
2024-02-07 01:09:16 | INFO | gradio_web_server | model_name: 0001.llava-v1.6-34b, worker_addr: http://localhost:40000
2024-02-07 01:09:16 | INFO | gradio_web_server | ==== request ====
{'model': '0001.llava-v1.6-34b', 'prompt': '<|im_start|>system\nAnswer the questions.<|im_end|><|im_start|>user\n\nWhat is unusual about this image?<|im_end|><|im_start|>assistant\n', 'temperature': 0.2, 'top_p': 0.7, 'max_new_tokens': 512, 'stop': '<|im_end|>', 'images': "List of 1 images: ['b939abf2c4553ce07e642170aee3a3d7']"}
2024-02-07 01:11:04 | INFO | gradio_web_server | add_text. ip: 127.0.0.1. len: 33
2024-02-07 01:11:06 | INFO | gradio_web_server | http_bot. ip: 127.0.0.1
2024-02-07 01:11:06 | INFO | gradio_web_server | model_name: 0001.llava-v1.6-34b, worker_addr: http://localhost:40000
2024-02-07 01:11:06 | INFO | gradio_web_server | ==== request ====
{'model': '0001.llava-v1.6-34b', 'prompt': '<|im_start|>system\nAnswer the questions.<|im_end|><|im_start|>user\n\nWhat is unusual about this image?<|im_end|><|im_start|>assistant\n', 'temperature': 0.2, 'top_p': 0.7, 'max_new_tokens': 512, 'stop': '<|im_end|>', 'images': "List of 1 images: ['b939abf2c4553ce07e642170aee3a3d7']"}
2024-02-07 01:12:06 | INFO | gradio_web_server | The image depicts a man ironing a shirt while standing on a street, with a yellow vehicle in the background. The unusual aspect of this scene is that ironing clothes is typically done indoors, in a more controlled environment, such as a home or a professional laundry facility. The setting of the street is not a common place to iron clothes, and it's not a safe or practical location for this activity due to the potential for accidents and distractions from traffic. Additionally, the man is using a portable ironing board, which is not a typical piece of equipment one would expect to see on a city street.
The juxtaposition of the man ironing in the middle of the street with the urban backdrop creates a humorous and unexpected visual, which is likely the intention behind the scene. It's a staged photo meant to be amusing or thought-provoking, rather than depicting a real-life scenario.
Please note that the image is staged and not a depiction of a real event.
The image is a staged photo meant to be humorous or thought-provoking, rather than depicting a real-life scenario.
Please note that the image is staged and not a depiction of a real event.
Please note that the image is staged and not a depiction of a real-life scenario.
Please note that the image is staged and not a depiction of a real-life scenario.
Please note that the image is staged and not a depiction of a real-life scenario.
Please note that the image is staged and not a depiction of a real-life scenario.
Please note that the image is staged and not a depiction of a real-life scenario.
Please note that the image is staged and not a depiction of a real-life scenario.
Please note that the image is staged and not a depiction of a real-life scenario.
Please note that the image is staged and not a depiction of a real-life scenario.
Please note that the image is staged and not a depiction of a real-life scenario.
Please note that the image is staged and not a depiction of a real-life scenario.
Please note that the image is staged and not a depiction of a real-life scenario.
Please note that the image is staged and not a depiction of a real-life scenario.`

Thanks for providing the additional information. I think I may know the cause -- if your model is local.
Can you check if this is the case? and what happens if you fix this mod: https://huggingface.co/liuhaotian/llava-v1.6-34b/commit/e2a1f782a20d26b855072029738bfd0107d85e96

Repetition is fixed after updating generation_config.json, cheers.

Great! I'll close this PR for now, thanks.

authors: HTplex haotian-liu

repo: haotian-liu/LLaVA


text:
Update dependencies

Every version of fast-safe-stringify prior to v1.1.0 has been deprecated, see: https://github.com/davidmarkclements/fast-safe-stringify/issues/4

@arb why not 1.1.x like in all other hapi projects ?

Because it's not a dependency we control.

authors: Marsup arb minustime

repo: hapijs/good-http


text:
Deprecate?

Anyone using this module?

Yep, there's activity:

Via: https://www.npmjs.com/package/visionary
I should definitely update our dependencies though.

Activity doesn't mean anyone moving to v17 is going to want it...

Sure thing. I'll look into v17 in more detail. It also looks like glue is going to get updated for v17 too, and using a manifest file with glue is the primary reason for visionary. But maybe things are so new that won't be a good pattern to continue using.
If we don't deprecate this module, I'm happy to bring it update to date.

v17's realm.parent will affect how we think about this module.  It's possible the functionality could even be provided by vision.  Imagine if vision had multiple instead of once and could set the manager for the realm in which it is registered when options are provided.

I use visionary to simplify my glue manifest configuration, like so.
    {
      plugin: 'vision',
    },
    {
      plugin: {
        register: 'visionary',
        options: {
          engines: { hbs: 'handlebars' },
          path: relativeToRoot('public'),
        },
      },
    },

how would it look if it was builtin to vision?

Would have to rope-in @wswoodruff, but I imagine that realm.parent would allow vision to make any plugin that does this,
await server.register(Vision);
server.views({/* view manager options */});

to be able to do,
await server.register({
    plugin: Vision,
    options: {/* view manager options */}
});

What I am not sure of is if vision is actually interested in this functionality, if it's considered beyond the scope of the package, or not worth the complexity.  But it might be an interesting option!  @jedireza from your perspective how does dropping server.root in hapi v17 affect visionary?

from your perspective how does dropping server.root in hapi v17 affect visionary?

@devinivy I'm unsure at this point. I need to understand more about the differences between server.realm and server.root.
I intend to wait and see where vision lands. Deprecating unnecessary code is always good, let's hope that's possible with the updates in that module.

@jedireza Glad to have your input! After speaking with @devinivy, here's my thinking -- open to feedback: I can see vision taking advantage of realm.parent by setting the manager on it during registration, and if options are passed in, will call .views() on realm.parent for the user. If options are not passed in during registration, it would be up to the user to call .views(). That would allow the functionality Devin described in his comment

Currently pluginOptions is unused in vision =P

One use case for multiple registrations of vision would be (1) a module that manages template rendering for emails, (pass in the email templates path to that plugin) -- and (2) the website's template views -- they'd have different managers and the email module would register vision

Vision is already designed to be registered multiple times– it should be registered by any plugin that needs it, which is the simplest way to manage a dependency on vision: register it yourself rather than relying on someone else to do it for you!
From my perspective, this change would add complexity to vision and perhaps complicate its API, since registration+options and server.views(options) would behave identically.  On the flip side, it would encompass the functionality of visionary, which I think otherwise would have to leverage what are currently vision internals in order to maintain its current functionality.  Tricky stuff!

I have a PR for multiple: true in vision here: https://github.com/hapijs/vision/pull/135
Effectively, the only thing this adds is the ability to use
await server.register({
    plugin: Vision,
    options: {/* view manager options */}
});

which will have the same effect as
await server.register(Vision);
server.views({/* view manager options */});

i'm another glue user and am finally starting to try to get my hapi servers updated to v17. to clarify, is there currently a way to use vision with glue without visionary, or do some of the details above still need to be worked out before a server using glue and vision can be upgraded?

@travi I'm still working on the PR mentioned above ☝ to allow for vision to receive view manager options during registration (so it can work with glue)
So not right now but soon

ok. thats what i thought i was understanding, but thank you for confirming. saves me from spinning my wheels before it is actually possible. if there is anything that help is need for to get this through, i'm happy to help out when i have time.

@travi Vision 5.3.0 has passing manager options to register options

What's the verdict on this? Upgrade or deprecate?

✋ deprecate

authors: Danlock devinivy hueniverse jedireza travi wswoodruff

repo: hapijs/visionary


text:
Iterator for Character

Character::begin, Character::end etc.

Character will not have iterator. Instead, it contains code point sequence which have iterators.


text:
New Devices Unresponsive

Evening
I've been using the service for a year or 2 and it's working great. I've recently added a few new devices to my profile, and they are not performing as expected.
I add a new switch to my profile, I see the new switch in my google home account, I assign the switch to a home and a room, same as I've done with all the other devices. I then go into NR, drop in new send / receive google assistant blocks, link them to my account, select the new device from the dropdown, same as I've done for my other switches.
Except for some reason, messages are not coming through from google when I toggle the switch there or get through when I toggle the switch from NR.
Ideas please?

Has this resolved it's self?
What do you mean by "or get through when I toggle the switch from NR."? Sending a status update using the response node should not cause any messages to be sent, it just updates the logical status in the Google Home Graph (the Google Home App does NOT update in real time)

Good afternoon. Apologies, was also away.
To answer the first question, no this doesn't seem to have resolved itself.
Maybe to clarify from my side. Attached a few screen grabs as well.
The pool light is an existing switch I loaded quite some time back. When I toggle the switch in Google Home, I get basically real time feedback on the command, as you can see on the time stamps.
With any new switches that I add, they no longer respond at all to me toggling them in google home., regardless of how long I give it They are visible, there just seems to be a breakdown somewhere that stops the toggle message getting through.

I've just added a new switch to my account and then added the new node to Node-RED and it's working just fine, so I can't reproduce this.
Do you get events if you toggle the new device via voice?
Also there was a report from another user of things not working if the device name contained "pool" but that was only via voice and that is all on Google's side of things and I have no control over that.
If you can tell me your account name and the device name I can monitor the system to see if I can see requests to toggle it coming through

And true to form, this seems to be behaving flawlessly now.... mutter mutter mutter.... haha
Appreciate the help.
To comment on your pool thing, I have exactly the same issue. Tell google to turn on the pool light, and it's never heard of it. But tell google to turn the pool on, and it switches the light on.... But then to switch off, it works by telling google to turn the pool light off.... so who knows what they're playing at...

authors: baxtmi hardillb


text:
QuestionTypeSeeder error

Hi,
I found that the migration for the QuestionTypeSeeder throws an error.

From my knowledge the create() supports only one record. The insert() supports many records but does not automatically update the timestamps. One option would be:
$questionTypes = [
            [
                'name' => 'multiple_choice_single_answer',
            ],
            [
                'name' => 'multiple_choice_multiple_answer',
            ],
            [
                'name' => 'fill_the_blank',
            ]
        ];

        foreach ($questionTypes as $questionType) {
            QuestionType::create($questionType);
        };

All the best
Alex

Hey @alex1897. Thank you for reporting this. Can you provide what version of Laravel you are using?

Hi @harishdurga. Yes sure I have the newest version Laravel 9.21.5.

Please feel free to reopen the issue if you are still having the issue.

I tested this and the migration throws an error now:

I guess the reason for that is that you've changed in #17:
- namespace Database\Seeders;
+ namespace Harishdurga\LaravelQuiz\Database\Seeders;

Can you try running the following command?
php artisan db:seed --class=\\Harishdurga\\LaravelQuiz\\Database\\Seeders\\QuestionTypeSeeder

I will update the seeder command in the README accordingly then.

repo: harishdurga/laravel-quiz


text:
[NFT] cross-chain universal identity

Description
To build a simple service to map NFT state from Ethereum to Harmony
Context
NFT as universal identity.
Use NFT as PFP is popular. Users may choose to use the same PFP across multiple chains.
Users don't need to bridge their NFT to Harmony while users can still use the NFT as identity.
Noted this is for identity only, but for collateral/trading usage as users may trade the original NFT in Ethereum w/o limitation.
This bounty is to build a simple service to monitor the state of the NFT smart contract on Ethereum and map the corresponding state to Harmony.  Running an on-demand or cronjob type service to continuously import the state.
Acceptance Criteria

[ ] an open-source repo with the design and implementation of the NFT mapper
[ ] smart contract deployed to harmony testnet/mainnet
[ ] a whitelist of the NFT mapped from Ethereum to Harmoy
[ ] the mapping assets displayed on https://bridge.harmony.one
[ ] demo of the NFT mapper
[ ] deploy document

Reward
$5,000 in ONE Token
Reference Design Doc
https://docs.google.com/document/d/1Q1XROsuvFQvMfEXTjiZjB934HpfTZo_O26OZ34k1JK8/edit#

@LeoHChen Hi Leo, I was working on this bounty but tonight I find there is already another repository for this at harmony:
https://github.com/jhd2best/identity-token-relayer
I'm an hour or two from finishing it ( all core functions are added just need some glue codes and docs ) based on the referenced doc but since I find that repo I'm not sure whether should I continue to work on it or not and whether this bounty is still available or not.
Here is my repo: https://github.com/Ajand/harmony-identity
repo is a bit messy because I was not sure whether it was already done or not and I didn't clean it up.

Hi, @Ajand, sorry the bounty was taken and finished.  Please look into other bounties. Thanks.

authors: Ajand LeoHChen

repo: harmony-one/bounties