text:
More Pandas dtypes and more flexible variable naming

Pandas DataFrame supports more dtypes than 'int64', 'float64' and 'bool', therefor added a bunch of extra dtypes for the data variable.
From now on the label variable can be a Pandas DataFrame with the same dtypes as the data variable or it can be a numpy.ndarray
If label is a Pandas DataFrame will be converted to float.
If no feature_types is set, the data dtypes will be converted to 'int' or 'float'.
If data is Pandas DataFrame, feature_names passed through will be ignored and overwritten by the column names of the Pandas DataFrame.
The feature_names may contain every character except [, ] or <
List of possible dtypes:
'int8', 'int16', 'int32', 'int64'
'uint8', 'uint16', 'uint32', 'uint64'
'float16', 'float32', 'float64'
'bool'

An example to test various changes:
import xgboost as xgb
import pandas as pd

col_1 = [76.1, 44, 20.4, 42, 74, 73, 6, 57, 59.224, 26, 64, 28, 99, 90.8, 40, 77, 14, 2, 97, 33]
col_2 = [62, 31, 51, 19, 1.4, 80, 74, 15.22, 42, 26, 29.7, 95, 2, 99, 12.87, 58, 30, 68, 92, 45]
col_3 = [1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0]
target = [1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0]

# X and y both DataFrames.
# X uses column names with an underscore
X = pd.DataFrame({'COL_1':col_1,
                  'COL_2':col_2,
                  'COL_3':col_3})
y = pd.DataFrame({'TARGET':target})

# Testing different types
# All three can be 'int8', 'float16', 'uint32' etc.
X['COL_1'] = X['COL_1'].astype('uint8')
X['COL_2'] = X['COL_2'].astype('float32')
X['COL_3'] = X['COL_3'].astype('bool')

# Testing a vary strange column name
strange_column_name = 'v3Ry_$t@ng3 N@m3'
X.rename(columns={'COL_1':strange_column_name}, inplace=True)

X_train = X[0:15]
X_test = X[15:20]

# Can be a numpy.ndarray when using y_train = y[0:15].values
y_train = y[0:15]

param_dist = {'objective':'binary:logistic', 'n_estimators':2}

clf = xgb.XGBModel(**param_dist)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)

print(y_pred)

@tqchen I have a few questions about the two errors I get when using test_basic.py on the new code:

The first error is because you could now use all characters, except [ ] and <. Are you alright with that and should I change the test_basic.py file?
The second error is that the feature_types that are coming back, are not 'int', 'q', 'i'. To try to be consistent, I changed all 'q' and all 'i' to 'int'. Is there a reason to also want to use 'q' and 'i' in Python?

======================================================================
FAIL: test_dmatrix_init (__main__.TestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\python\xgboost-master\tests\python\test_basic.py", line 47, in test_dmatrix_init
    feature_names=['a', 'b', 'c', 'd', 'e=1'])
AssertionError: ValueError not raised

And
======================================================================
FAIL: test_pandas (__main__.TestBasic)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\python\xgboost-master\tests\python\test_basic.py", line 104, in test_pandas
    assert dm.feature_types == ['int', 'q', 'i']
AssertionError

Thanks!

I think the major reason of using q, i is to be consistent with c++ side's feature name specification. I think we can use the more intuitive names, as long as things are consistent, and translation is done properly

This code: mapper = {'int64': 'int', 'float64': 'q', 'bool': 'i'} converts a 'float64' to a 'q', which is a 'long long' in C and an 'integer' in Python (and apparently a '__int64' on Windows). I think a float should stay a float, don't you?

And the 'bool' converts to an 'i', which is an 'int' in C and Python. Why not use 'int' like the 'int64'? I think it will keep the variable types consistent in Python if we only use 'int' and 'float'.

Yes, I think we can go with int and float and bool
Note that the c++ side accepts only i, q and int. for now.  c.f. https://github.com/dmlc/xgboost/tree/master/demo/binary_classification#dump-model
So some of these conversions may have to be with c++ side. Either do the conversion on python side, or change the defs of c++ side later

It is up to you. If you would commit the changes all types of int and bool become int, all types of float become float. As far as I can tell it runs fine with all the changes.
The dump-model seems to work fine and dumps a nice model with the Pandas column names:
import xgboost as xgb
import pandas as pd

col_1 = [76.1, 44, 20.4, 42, 74, 73, 6, 57, 59.224, 26, 64, 28, 99, 90.8, 40, 77, 14, 2, 97, 33]
col_2 = [62, 31, 51, 19, 1.4, 80, 74, 15.22, 42, 26, 29.7, 95, 2, 99, 12.87, 58, 30, 68, 92, 45]
target = [1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0]

X = pd.DataFrame({'COL_1':col_1,
                  'COL_2':col_2})
y = pd.DataFrame({'TARGET':target})

X['COL_1'] = X['COL_1'].astype('float32')
X['COL_2'] = X['COL_2'].astype('int16')

X_train = X[0:15]
X_test = X[15:20]

y_train = y[0:15]

dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test)

param = {'max_depth':2, 'eta':1, 'silent':1, 'objective':'binary:logistic' }
num_round = 2

bst = xgb.train(param, dtrain, num_round)

bst.dump_model('dump.nice.txt')

Content 'dump.nice.txt':
booster[0]:
0:[COL_1<4.5] yes=1,no=2,missing=1
	1:leaf=0
	2:[COL_1<4.5] yes=3,no=4,missing=3
		3:leaf=0
		4:leaf=0.315789
booster[1]:
0:[COL_1<4.5] yes=1,no=2,missing=1
	1:leaf=0
	2:[COL_1<4.5] yes=3,no=4,missing=3
		3:leaf=0
		4:leaf=0.0698864

The change of c++ side might affect some other packages such as R, so we want to do it more carefully.  I think we can switch the names in python side first, and remaps them when calling into the dump API. Can you do that?

Sorry, I don't know what you are asking... Without changing anything on the c++ side, it seems to work fine with the changes I made.

I have not changed anything to the dump_model and the get_dump functions, they work perfectly as they are, also (even better) when using a pandas DataFrame.
The changes I made to the _maybe_from_pandas function changes all the 'q' and 'i' references to 'int' and 'float'.
The only line that still has a reference to 'q' is: ftype = from_pystr_to_cstr(['q'] * flen) on line 940 of the core.py file. I think this needs to stay this way to convert a list of Python str to C pointer.

On my side one question remains, should the test_basic.py file be edited? If so by me or you?

@JohanManders  "i" is not for "int", it's "indicator variable" likely to be bool. From my understanding, feature type only for specifying output dump string format, irrelevant from actual data types. You can see my gist how each feature type affect to the output (node label).
https://gist.github.com/sinhrks/cc9a88f74074fc296e12#file-add_featuretypes-ipynb
This closes #534.

@sinhrks Thanks for testing that!
Based on your tests, it seems that i is used for bool, int can be used for int and both q and float can be used for a float. Too bad the c++ side doesn't recognize bool, but int and float work fine.
I have changed my code.

I would like to check update status on this. Can you make the changes such that the current testcase pass? Or update the testcases if it is proposed new behavior?

@tqchen I have updated the testcases so that a float can stay a float and = is a legal character for a feature_name, but < is not.
I double checked everything and I think these changes are an improvement.

@JohanManders Can you rebase to resolve some conflicts and try again so that the errors are more obvious?

@terrytangyuan I thought it was easier to make the adjustments to my master branch (https://github.com/JohanManders/xgboost), because this branch keeps having errors. Could you close this pull request and make a new pull request from my master branch? Let me know what you think.

Yeah feel free to work on your own forked repo and open a new PR whenever it's ready.

Opened a new PR with the changes: https://github.com/dmlc/xgboost/pull/597
Thanks @terrytangyuan!

authors: JohanManders sinhrks terrytangyuan tqchen


text:
[Feature Request] Infisical Plugin

Description
Support Infisical Machine Identities for picking secrets from Infisical projects.
Motivation
This is where our secrets live and want need to integrate DMNO and Inifisical to build out proof of concept
Proposed Solution
No response
Alternatives
No response
Additional Information
No response

Thanks for this, if there are any additional requirements, definitely let us know!

@rawkode - definitely will build this at some point, but priority wise, is this still something you're wanting soon or are you happy with using the 1pass plugin?

Closing - the Infisical plugin is live and working well.

source: gharchive/issue

authors: philmillman rawkode theoephraim

repo: dmno-dev/dmno

license: MIT

license_type: permissive

license_source: github-api


text:
name prop

Is your feature request related to a problem? Please describe.
event.target.name is missing
Describe the solution you'd like
Add name prop
Describe alternatives you've considered
either will use native element or doing separate handlers
Additional context
Add any other context or screenshots about the feature request here.

The event onChange is not provide an event argument cause there are quite a lot places from where you can submit an event. And that would not be an input.
So you need to mock event, by creating the object with event.target.value by yourself

authors: dmtrKovalenko maddhruv

repo: dmtrKovalenko/material-ui-pickers

license: mit

license_source: bigquery


text:
Stageout executor handles exit codes wrong

Hi,
I was tracking down some false positives at Vanderbilt (where stageouts were thought to have succeeded but they actually failed) and found a painful WMCore bug. See the following snippet, which is a simplified version of the actual code (https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Storage/Execute.py#L77)
Python 2.7.13 (default, Dec 17 2016, 23:03:43)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import subprocess
>>> child = subprocess.Popen("exit 127", shell=True)
>>> err = child.wait()
>>> print err
127
>>> os.WIFEXITED(err)
False
>>> os.WIFSIGNALED(err)
False
>>> os.WIFSTOPPED(err)
True
>>> os.WSTOPSIG(err)
0

The code actually returns 0 to its caller, instead instead of 127! This isn't good! In particular, that's the exit code that bash uses if it can't find a script. I'm going to add a separate PR to fix the vandy specific stuff to test for this, but otherwise that code probably needs to be fixed up.

Thanks Andrew for reporting a problem.

@ticoann @amaltaro is there a patch lining up for this issue ?
@PerilousApricot is there any issue with the rolling back to srm https://ggus.eu/index.php?mode=ticket_info&ticket_id=128809#update#17
I am assuming that other sites could be affected and this might have serious consequence on storage consistency.
Jean-Roch (CRC)

@vlimant, There is no patch yet but I will investigate. I am not quite understand how this code works. Did you observe the same problem for any other sites? It seems that part of code was there since the beginning. I would suspect we would have a lot more cases like this before.

@vlimant Ugh, my laptop is in the shop, so I'm temporarily using another machine that doesn't have my grid cert, so I can't see the GGUS ticket. If it's the "lost files" ticket, then I think moving to gfal fixes it.
To prevent from having to update WMCore each time we want to change our local stageout scripts, we store them in GPFS. What I think happened is that either a) GPFS wasn't properly mounted on a WN or b) some jobs landed in singularity, which doesn't have GPFS bind-mounted. When that happened, WMCore tried to run an unexistent script, and bash uses 127 for that exit code:
[root@foreman ~]# /bin/sh -c "nonexistent" ; echo $?
/bin/sh: nonexistent: command not found
127

But, WMCore does a weird thing and tries to return the signal number instead of the exit code, and 127 gets parsed/converted into 0
    elif os.WIFSTOPPED(err):
        return os.WSTOPSIG(err)

To your point, @vlimant -- this could happen anytime a stageout plugin returns 127. I don't see any need for any of the if statements between the err = wait() and the return err. Spot-checking a couple plugins, it doesn't seem like anyone depends on that behavior.

The problem commit is this one: https://github.com/dmwm/WMCore/commit/175338d0b0a87111d8fdd2f3aad00d5699797e52
It assumes that the integer returned by popen2.Popen3.wait has the same semantics as the integer returned by subprocess.Popen.wait.
popen2 returns the status code from, basically, os.wait, which is compatible with the os.WIF* functions and does the right thing.
subprocess returns a bitshifted status that doesn't seem to have any particular semantics aligned with POSIX and is incompatible with the os.WIF* functions.  It causes this code to translate exit status 127 (which is impossible to return from POSIX's wait) into 0.  Garbage in, garbage out.

Thank you very much, Brian, I didn't know (I wouldn't have known that). I will create the patch for this.

authors: PerilousApricot bbockelm ticoann vlimant

repo: dmwm/WMCore


text:
Don't set EventsPerJob bigger than RequestNumEvents

Of course, also considering the FilterEfficiency and the usual multiple of EventsPerLumi.
This JIRA ticket reports an workflow that goes to failed:
https://its.cern.ch/jira/browse/CMSCOMPPR-5394
and the reason is that the TimePerEvent is so so small that the job splitting gets automatically set to 42 million events per job (420k lumis per job), while it's requesting only 100k events.

@khurtado I'll be off today and tomorrow, perhaps you can take this one and I test/resume this fix on Monday?
This is how the final EventsPerJob and EventsPerLumi are calculated:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMSpec/StdSpecs/StdBase.py#L83
but the change will have to go to the StepChain and the TaskChain specs, e.g.:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WMSpec/StdSpecs/TaskChain.py#L564-L580


text:
Last changes not published in maven?

Hi there,
I'm trying to include realm browser in my project but version 0.0.3 is not compatible with realm 0.84.1.
It seems that last code is not uploaded to maven.
Do you plan to do it?
Many thanks!

Hi, unfortunately no.
This library is not more supported.
If you want to add new feature or fix a bug, grab source code and do it. If you think your
fix or feature would be useful to other developers, I can add link of your repository to
this README file. Use following e-mail to contact me: dmytrodanylyk@gmail.com
Thank you for using my libraries.

authors: dmytrodanylyk drrodio

repo: dmytrodanylyk/realm-browser


text:
Standardise language around recipe / module versions

Last year we passed an RFC to decouple patch-level releases in the core modules. This allowed, for example, silverstripe/framework to receive a patch version outside of the standard release cycle, and allowed modules like silverstripe/cms to not receive a patch version during a hotfix (which would have had identical contents to the version before).
This is a win in many regards, but it also provides massive opportunity for confusion when it comes to describing a release. For example, the '4.5.2' hotfix release includes silverstripe/framework 4.5.3 and silverstripe/cms 4.5.1. Some tooling looks at one of these modules to determine the version of 'Silverstripe' in use, and there are other places that use the term 'CMS & Framework', which is even more confusing. What we want to refer to in all of these cases is the version of the Silverstripe CMS Recipe (tbc).
This issue attempts to compile a full list of places where we are using incorrect terminology and/or referring to the version information on the wrong package, so that we can adjust them all.

[ ] The UI in the CMS references the cms module version version rather than the CMS Recipe version – GitHub issue
[ ] The UI in Silverstripe Cloud references the framework version rather than the CMS Recipe version - Jira issue
[ ] The older changelogs in CWP (2) need to be tweaked retroactively to reference CMS Recipe
[ ] The changelogs in the main documentation don't reference CMS Recipes

It may be best to provide a breakdown of the module versions included in each release?
We should add a note to the changelog index page explaining that they track CMS Recipe versions

[ ] silverstripe.org/download references "CMS & Framework version" rather than CMS Recipe version
[ ] Internal documentation on how to reference CMS Recipe versions going forward
[ ] Topics in community Slack should reference CMS Recipe (currently 'Release')

Added an existing issue in the description for:

The UI in Silverstripe Cloud references the framework version rather than the CMS Recipe version (https://silverstripe.atlassian.net/browse/SC-1842)

Is elemental the right place to do this?

@michalkleiner Nope, this definitely landed in the wrong bucket 😅

authors: Cheddam brynwhyman michalkleiner

repo: dnadesign/silverstripe-elemental


text:
Feat/#41 유효성 검사 로직을 추가합니다.

What is this PR? :mag:

feat: 유효성 검사를 위한 zod + RHF를 설치한다.
feat: 유효성 검사 로직을 추가합니다.
refactor: 기능 로직을 커스텀훅으로 분리합니다.

Changes :memo:
유효성 검사를 위한 zod  RHF를 설치하였습니다.
Zod를 사용하기 위한 resolver와 RHF를 사용하였습니다.
유효성 검사 로직을 추가하였습니다.
유효성 검사 로직의 경우 <Controller/> 태그의 rules 사용하여 구현을 하였으며 /Project/Create 페이지를 본다면 사용법을 참고하실 수 있습니다.
반복되는 컴포넌트, 훅을  분리하였습니다.
기존에 많이 사용하던 컴포넌트들을 하나의 컴포넌트로 분리하여 관리를 하였습니다. 또한 useSingleImage와 같이 여러군데에서 사용이 가능한 훅도 커스텀 훅으로 분류하였습니다.
Screenshot :camera:

질문
Image 파일의 관리방법에 대해
프로젝트 등록 시 FormData 형태로 전송해달라고 하셨는데, 수정 페이지에서 이미지를 보여주기 위해서는 이미지 주소만 있으면 될 것 같아 이미지 파일을 어떻게 관리할지에 대해 고민하고 있습니다.
이로 인해 수정 시 번거로운 문제가 발생할 수 있을 것 같아, 이미지 업로드를 위한 별도의 API가 추가되면 좋을 것 같아 보입니다.
1. 파일(File) 형태로 관리하는 경우:
등록:
이미지를 file 형태로 서버에 전송하여 관리.
이미지를 보여줄 때는 file에서 string 형태의 URI로 변환해서 클라이언트에 제공.
수정:
서버에서 기존에 받아온 URI는 string 형태이므로, 수정 시 이미지를 다시 file 형태로 변환해 서버로 전송.
파일 업로드 및 변환 과정이 추가로 필요함.
장점:
서버에서 이미지 파일을 직접 관리할 수 있어 확장성 있음.
이미지 파일이 항상 최신 상태로 유지될 가능성이 높음.
단점:
수정 시 기존 URI를 다시 file 형태로 변환하는 과정이 번거로움.
매번 파일 업로드가 필요하기 때문에 성능에 영향을 미칠 수 있음.
2. API를 새로 만들어 URI(String)로 관리하는 경우:
등록:
이미지를 서버에 업로드한 후, 이미지를 URI(string) 형태로 관리.
프로젝트 생성 시 file 형태로 보내지 않고, 이미 서버에 등록된 이미지의 URI를 그대로 전송.
수정:
이미지를 수정할 필요가 없을 때는 URI만 유지하고, 수정이 필요할 경우 새로운 이미지를 업로드하여 URI만 교체.
파일 형태로 변환하는 과정이 생략되어, 수정 시 부담이 적음.
장점:
이미지 업로드와 전송 과정이 간소화됨.
기존 URI를 그대로 사용하여 수정 시 성능 이점이 있음.
이미지가 이미 서버에 저장되어 있어, 불필요한 파일 변환 없이 URI로 간단히 관리 가능.
단점:
서버에 저장된 이미지가 언제나 최신 상태인지 확인하는 추가적인 관리가 필요할 수 있음.
결론:
1번 방법은 이미지 파일을 서버에서 직접 관리하는 방식으로 확장성은 있지만, 수정 시 파일 변환과 업로드가 반복되어 번거로울 수 있습니다.
2번 방법은 URI로 이미지를 관리하는 방식으로 성능과 효율성이 더 높아 수정 시 유연하게 대응할 수 있습니다.
만약 자주 이미지 수정을 해야 하거나, 빠르고 가벼운 관리가 필요하다면 2번 방법(API를 새로 만들어 URI로 관리)이 더 적합할 것 같습니다.

참고자료

파일 업로드 API가 추가되기로 결정.

source: gharchive/pull-request


text:
fix opacity on default fills

This got broken in #183 - adds a test

Going to land this TBR


text:
DNN Tracking DLL Dependencies fails when multiple Versions are in Use

Description of bug
Two modules using the same DLL in different versions can easily do so by placing them in a subfolder in the bin and setting the Web.Config rules for version redirects. This works at runtime, but fails on module installs and updates. Here's a real example where 2sxc and DnnSharp run into a problem (see also the issue on 2sxc:

2sxc uses EF 1.1 and the MS Dependency Injection 1.1 and places all this in bin/2sxc
PlantAnApp (DnnSharp) uses EF 2.1 and places all this in bin/PlantAnApp

Because the web.config redirects manage all this, it's not a problem. But on Module-Install and Upgrade it fails. The reason is that DNN tries to help by tracking installed DLLs, but doesn't keep track of the folder. So even though the files don't conflict, DNN checks the DB and assumes a newer version has already been installed (Table: Assemblies):

Note: for historic reasons the EF in 2sxc has the wrong version (it believes it's EF v11 or 12 because that's the 2sxc version). This may lead to more confusion, because actually installing 2sxc after PlantAnApp works, but updating PlantAnApp will fail then. But the core problem is that the version ignores the folder it's in.
Current behavior
This leads to the following problems:

If PlantAnApp is installed first, 2sxc with EF v1.1 wouldn't install, because DNN believes it already has a newer version.
If 2sxc with EF 1.1 is installed first, the entry would say v1.1 and Would allow PlantAnApp to install it as well. That would bump the version to 2.1. After that updating 2sxc with a newer EF (say 1.2) would fail, because DNN believes it already has a newer version.

Expected behavior
DLL Versions in Assemblies should include the folder they are in, as the same DLL can be installed in different versions.
Affected version

[x] All DNN versions as of now up and till v09.09.01

If you're maintaining your own collection of DLLs in its own folder, and managing your own binding redirects, etc., then I'm not sure that you're getting any value out of using the Assembly component, rather than just a simple File component. Are there features of the Assembly component you're looking to keep?

@bdukes I'm not sure what you mean. We're just listing our DLLs in our Dnn Manifest since we assume that well-behaved modules should do this, and I think it's necessary that they get installed within the bin (sub) folder.
What features would Assembly components have? I'm not really aware of any features in this scope.

Using an Assembly component (as opposed to a File component) provides the following features:

Registers the assembly in the Assemblies table (records what version of an assembly each package installed)
Creates a binding redirect for strong-named assemblies
Checks the version of installed assemblies to avoid overwriting newer assemblies with older assemblies

It doesn't seem to me that any of those features are of use to these assemblies that are installed to an alternate folder. Certainly a well-behaved module should declare their shared assemblies that are going into the bin folder using the Assembly component, but assemblies that you're managing and that are not shared probably don't benefit by using the Assembly component.

I just looked at the .dnn file in your latest install package and wanted to respond to the comments 🙂
The File component will report file extensions that are not on the allowed list. However, the ResourceFile component will not.
https://github.com/2sic/2sxc/blob/9ba57100fa532aef41ec96a9bb0f347ac89aeb73/Src/Dnn/ToSic.Sxc.Dnn.Main/DnnPackageBuilder/2sic_2SexyContent.dnn#L266-L275
It looks like this is what you're doing now for pdb files, you could do the same for your bin/2sxc/*.dll files.
https://github.com/2sic/2sxc/blob/9ba57100fa532aef41ec96a9bb0f347ac89aeb73/Src/Dnn/ToSic.Sxc.Dnn.Main/DnnPackageBuilder/2sic_2SexyContent.dnn#L286-L293
Regarding PDB files, it looks like you implemented support for pdb files in the Assembly component in DNN 9.1.1.
https://github.com/2sic/2sxc/blob/9ba57100fa532aef41ec96a9bb0f347ac89aeb73/Src/Dnn/ToSic.Sxc.Dnn.Main/DnnPackageBuilder/2sic_2SexyContent.dnn#L145-L148

I have to say that I agree with @bdukes on this one.
The nested/sub-folder approach is not necessarily an approach that is truly publicized, and requires additional work (config section additions, etc).
I think that the usage of the file component is a better option for these.

@bdukes and @mitchelsellers thanks for your insights.
Note that even though we contributed the .pdb fix we cannot use it ourselves, because 2sxc can still be installed on Dnn 7.4.2.
@bdukes From what I understand this is what would resolve the issue:

We could change our DLLs which are in a subfolder should become ResourceFile dependencies and not Assembly dependencies
DnnSharp should do the same - this would be important for @rezagarrow

I assume this would look like this:
<component type="ResourceFile">
  <resourceFiles>
	<basePath>bin\2sxc</basePath>
	<resourceFile>
	  <name>Microsoft.Extensions.DependencyInjection.Abstractions.dll</name>
          <name>Microsoft.Extensions.DependencyInjection.dll</name>
          <name>Microsoft.Extensions.Logging.Abstractions.dll</name>
          <name>...</name>
	</resourceFile>
  </resourceFiles>
</component>

I think this would be simple to do for both of us, but there is a problem making the change. I assume we should somehow tell the installer that the old Assembly entries should be cleared, even though we're still including the file. So it's a kind of CleanUp but then again the files should stay there.
Do you have a suggestion how we could add this to the manifest and not break new installs or upgrades? I simply have no clue on this. Many thanks!

The Resource File component takes a zip file and unzips it into a given folder:
        <component type="ResourceFile">
          <resourceFiles>
            <basePath>bin/2sxc</basePath>
            <resourceFile>
              <name>2SxcBinResources.zip</name>
            </resourceFile>
          </resourceFiles>
        </component>

(or you can remove the basePath element and include that path in the zip file itself, which is what you're doing for the PDB files).
For clearing out the Assemblies table, I don't think there's a good way other than running your own SQL script.  Something like this:
DELETE a
FROM {databaseOwner}[{objectQualifier}Assemblies] a
JOIN Packages p ON a.PackageID = p.PackageID
WHERE p.Name = '2SexyContent'

@bdukes ok, I was a bit afraid that it would need SQL 😢. Thx for the update.

ok, we can close this for now.

authors: bdukes iJungleboy mitchelsellers


text:
LOOCV for Transfer Learning using Skorch

First and foremost, thanks a lot for the amazing "transfer learning" tutorial in Skorch.
Can you please guide how to apply leave-one-out cross-validation to this tutorial?
https://github.com/dnouri/skorch/blob/master/notebooks/Transfer_Learning.ipynb

In most cases skorch should already work with LOO.
Here is a minimal code example:
import numpy as np
from skorch import NeuralNetClassifier
from skorch.toy import make_classifier
from sklearn.model_selection import LeaveOneOut
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_classification

X, y = make_classification()
X = X.astype(np.float32)

# when using CV, make sure to set train_split=None to prevent internal train/valid split
# also, set verbose=0 to prevent excessively printing logs
net = NeuralNetClassifier(make_classifier(), verbose=0, train_split=None)
scores = cross_val_score(net, X, y, cv=LeaveOneOut(), scoring='accuracy')
print(np.mean(scores))

That being said, in the mentioned notebook, a Dataset is used for X. This doesn't combine well with sklearn, which will try to split the dataset and make a mess. It is thus easier to make this work if your input data is e.g. a numpy array or torch tensor (or dictionary with the help of SliceDict).
That also being said, I would not recommend using LOO for the given problem since this will result in 1 split -> 1 training for each sample, which takes far too long. How about making a manual train/validation split, train the model and optimize hyperparameters based on the internal split, and when finished testing on the validation data?

@monajalal Was your issue solved?

Closing this since the question got answered.

authors: benjamin-work monajalal ottonemo

repo: dnouri/skorch


text:
ARM Chromebook - Crouton fails to complete installation if a single package fails

I've just run into the following issue trying to install crouton on my Samsung CB+ :
chronos@localhost / $ sudo sh ~/Downloads/crouton -t xfce,core,cli-extra,chrome,audio,extension,touch -r trusty -p /media/removable/SD\ Card/
All goes well up to the point where I notice the message:
Google Chrome does not yet have an ARM build. Installing Chromium instead.
which later on results to the following error:
Package chromium-browser is not available, but is referred to by another package. This may mean that the package is missing, has been obsoleted, or  is only available from another source However the following packages replace it:   chromium-browser-l10n E: Package 'chromium-browser' has no installation candidate Failed to complete chroot setup. Unmounting /media/removable/SD Card/chroots/trusty...
Seems that the whole installation fails if a single package fails to install. Any ideas how to overcome this?
Thanks

@namp,
Thanx for flagging this.
As a workaround, I'd remove the chrome target from your installation command string and then attempt to add the chromium browser in the chroot after the installation.
This will get you going and give the devs some time to figure out what's changed and how to fix it.
Hope this helps,
-DennisL

Yes, it seems like the most sensible thing to do.
However issuing the command again without the chrome target and with -u does not work. The chroot has to be deleted and the installation to be started from scratch, otherwise the same error occurs.
Many thanks for the answer

@namp,
That's because a crouton update ('-u'), uses the targets previously specified.
If you want to update it instead of installing a new one then you'd have to edit & remove the chrome target listed in your chroot in both /.crouton-targets & /etc/crouton/targets.
Hope this helps,
-DennisL

Oh - I did not know that. Thanks!
Still, the problem remains as another package failed to install (lsb-release apparently) and the whole installation failed as well.

authors: DennisLfromGA namp

repo: dnschneid/crouton


text:
croutonxinitrc-wrapper: Kick xrandr harder

Workaround is racy, let's try multiple times until xrandr keeps
quiet.

+👟

authors: dnschneid drinkcat


text:
Get listed on the FMI tools page

Great work!
Please list your toolbox on https://fmi-standard.org/tools/ , see https://github.com/modelica/fmi-standard.org/blob/main/CONTRIBUTING.md#updating-the-tools-list how to add your tool.

Thanks to adding your tool on the FMI tools page.
As https://github.com/dnv-opensource/mlfmu/tree/main/examples is linked from Examples & Compatibility there:
please

provide information with which validation tools from https://fmi-standard.org/validation/ and other importing tools you have tested the FMUs
provide the built FMUs with binaries

Remaining tasks:

[ ] Add FMU binaries
[ ] Add README for https://github.com/dnv-opensource/mlfmu/tree/main/examples to refer to docs, FMU file locations, and validation tools used.

@StephanieKemna : Thanks for adding the Readme. Pleas note that FMUcheck does only a static validation of the FMU, it would be great to use also the other tools listed on https://fmi-standard.org/validation/ (fmpy or fmusim from the Reference FMUs) to simulate the FMUs, and add this information to the Readme.

Thanks @chrbertsch - we can have a look at this (might be after the winter holidays). I will check about the STC url now.

authors: StephanieKemna chrbertsch

repo: dnv-opensource/mlfmu


text:
Mongos should be allowed to run initdb.d scripts too

Basically, I'm trying to configure shards when running mongos:
sh.addShard("rs1/db")

But the scripts in docker-entrypoint-initdb.d only run for mongod:
https://github.com/docker-library/mongo/blob/59fba668ae47e3ec7f7917a56db4f94d6d774765/3.6/docker-entrypoint.sh#L192-L318
Does it make sense to run for mongos too?

Is this a duplicate of https://github.com/docker-library/mongo/issues/339?
https://github.com/docker-library/mongo/issues/339#issuecomment-485578503
The docker-entrypoint-initdb.d scripts run during an initialization period (and only if the database is empty), during which the container is only listening on localhost so trying to initiate a cluster during that period isn't possible as it won't resolve it's own container hostname.

Actually I did a pretty big jerry rig and was able to configure it, but I did from other container running mongod, so it would work with mongos
The hack is nohup the script with an sleep, so the command runs and the servers are started, then the config is applied.
something like this:
nohup bash -c "sleep 10 && echo 'rs.initiate({_id: \"mongors1\", members: [{_id: 0, host: \"db\"}]})' | mongo --host db" &

It's really really ugly, but it works.
So I'm not sure how exactly related it is with the other issue, but having the ability to run scripts with mongos too would simplify my workflow.

Not sure, if this is directly related, but i had a similiar issue.
I wanted to initialize a replicaset in my local dev container.
I ended up with the following workaround.
#!/bin/bash

if [ "$1" != "CHILD" ]
    then
    # we are the main process so we need to start our child
    echo "STARTING CHILD"
    /bin/bash $BASH_SOURCE CHILD &
else
    # we are the CHILD so we run the actual init code:
    echo "Starting BG init script. Wating for Mongo to listen on $MONGO_HOST"
    ## waiting for mongo to be available
    until nc -z $MONGO_HOST 27017
    do
        sleep 1
    done
    echo "going on.. Initialize replset"
    # initializing replicaset
    mongo --eval  "rs.initiate({ _id: \"$MONGO_REPLSET\", version: 1,  members: [  { _id: 0, host : \"$MONGO_HOST\" },  ] });"

    mongo --eval 'rs.status()'
fi

This required me to set MONGO_HOST to the name of the service (in my compose file) and MOGO_REPLSET to the replSet Name.

I actually didn't figure out how the workarounds mentioned here can be used to run a mongos command in initialization step, but I have the same issue. If anybody can explain what I can do to add a shard automatically on startup, I would be so grateful.
I need to setup a sharded cluster that has the config server and shards up and running. The only requirement here is to run a mongos container and perform sh.addShard() on it.

authors: Shogoki alitoufighi ravanscafi wglambert

repo: docker-library/mongo


text:
Add env var to skip initdb

I'm trying to build a data container to ship developer databases (up to date migrations, fixture data, etc which are managed elsewhere). We could use a docker-entrypoint-initdb.d script to move a bundled data folder to PGDATA (similar to what's suggested in #105), but then the entire initdb process was a (one time) waste.
I'm proposing a new USE_SEED_DB env var that would skip initdb and instead move everything in a "seed" directory to PGDATA. This would set the expectation that scripts in docker-entrypoint-initdb.d/ would act on the seed db and not a fresh one.
I'm willing to submit a PR for this.

We could use /docker-entrypoint-seed-db.d for example.

I've opted to use a data container based on scratch (or rather tianon/true since docker-compose requires a command).
Pro: image can be used without running (used to need to run docker-entrypoint.sh to set things up)
Con: mounted volume directory may not have 700 permissions (must being careful when creating data image).
I've submitted #110 to set 700 explicitly.

repo: docker-library/postgres


text:
(Question) logger not working

I've been using python:3.6 as the base image for my Dockerfile for a while now. I hadn't built it from scratch for a long time, but when I did it today for testing purposes, I discovered errors that never occured before.
In my container, I use crontab, and I log it using the built-in logger command. However, since I rebuilt my container, I get the error "logger: socket /dev/log: No such file or directory". I cannot fathom where else the error could come from.
Judging from this post maybe it never should have worked, but it did until today.
Can someone explain this to me ? Many thanks.

Definitely a case of "never should have worked" unless you were explicitly mounting /dev/log before and not doing so any more.
It's really hacky, but you could try using something like echo thing to log > /proc/1/fd/1 to put things into the Docker logs.
For further help, I'd suggest a dedicated support forum, such as the Docker Community Forums, the Docker Community Slack, or Stack Overflow.

Thanks for the very fast answer, I'll find a hack.

authors: Louis-d-C tianon

repo: docker-library/python


text:
python:3.6 image compatibility issue with uWSGI

The recently released python:3.6 can no longer install an older uWSGI.
docker pull python:3.6
docker run --rm -it python:3.6 pip install uwsgi==2.0.16

Results in:
    plugins/router_basicauth/router_basicauth.c: In function ‘htpasswd_check’:
    plugins/router_basicauth/router_basicauth.c:73:5: error: ‘struct crypt_data’ has no member named ‘current_salt’
       73 |   cd.current_salt[0] = ~cpwd[0];
          |     ^
    ----------------------------------------

Used to work before the latest updates :)
Let me know if you need any extra info or have questions! :)

Note: i removed the part about alpine because i am dumb 😉

FYI, uWSGI==2.0.19.1 seems to install just fine on python:3.6. Also, you can e.g. use python:3.6-alpine and install the libraries by hand and get a working version with uWSGI==2.0.16.
So this can be closed from our point of view :blush:


text:
Pulling python 2.7 image from v2 docker registry returns nil

ERROR: Service 'build_image' failed to build: Get https://registry-1.docker.io/v2/library/python/manifests/2.7: <nil>

I'm running docker 1.10.0-rc1
 docker version
Client:
 Version:      1.10.0-rc1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   677c593
 Built:        Fri Jan 15 18:17:17 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.10.0-rc1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   677c593
 Built:        Fri Jan 15 18:17:17 2016
 OS/Arch:      linux/amd64

docker-machine version 0.6.0-rc1, build 174e390

Thanks @stevvooe

@mrlamroger I think @aaronlehmann deserves your praise.

haha woops. Thanks @aaronlehmann!

:hamster:

authors: mrlamroger stevvooe

repo: docker/distribution


text:
Correct ErrAuthenticationFailure message

This was "authentication failured". Change it to "authentication
failure".

Current coverage is 53.01%

Merging #1410 into master will decrease coverage by -4.10% as of e387b63

@@            master   #1410   diff @@
======================================
  Files          123     123
  Stmts        11357   11338    -19
  Branches       801     729    -72
  Methods          0       0
======================================
- Hit           6487    6011   -476
+ Partial        801     729    -72
- Missed        4069    4598   +529

Review entire Coverage Diff as of e387b63

Uncovered Suggestions

+0.28% via ...ge/driver/gcs/gcs.go#182...213
+0.24% via ...ge/driver/gcs/gcs.go#85...111
+0.21% via ...ge/driver/gcs/gcs.go#471...494
See 7 more...

Powered by Codecov. Updated on successful CI builds.

Looks good to med.

authors: aaronlehmann codecov-io dmp42


text:
Update helper_lib.sh

Update function comment for current code..

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "patch-1" git@github.com:kevinll/docker-bench-security.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Ammending updates the existing PR. You DO NOT need to open a new one.

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "patch-1" git@github.com:kevinll/docker-bench-security.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Ammending updates the existing PR. You DO NOT need to open a new one.

Hi @kevinll and thanks for correcting this.

authors: GordonTheTurtle kevinll konstruktoid

repo: docker/docker-bench-security


text:
table of content links does not render correctly in the repository overview

For example the table of contents in https://registry.hub.docker.com/u/bgruening/galaxy-blast/ does not render nicely. You get something like the following:
 *   [Installed tools](#installed-tools)
 *   [Usage](#usage)
 *   [Users & Passwords](#user–passowrds)
 *   [Reproducibility of your search results](#reproducibility-of-your-search-results)
 *   [Using large external BLAST databases](#using-large-external-blast-databases)
 *   [Requirements](#requirements)
 *   [Restarting Galaxy](#restarting-galaxy)
 *   [History](#history)
 *   [Support & Bug Reports](#support–bug-reports)
 *   [Licence (MIT)](#license-mit)

To quote from https://docs.docker.com/docker-hub/official_repos/:
A long description
Include a comprehensive description of your image (in Markdown format, GitHub flavor preferred).

Github is rendering it correct: https://github.com/bgruening/docker-galaxy-blast
Link to original bug report: https://github.com/bgruening/docker-galaxy-blast/issues/2

Thanks for reporting this issue!  I too am experiencing it.

Hi, this is a hub issue.
This here is the open-source registry bug tracker, that does not provide this feature.
Can you report this directly to docker support instead?
https://support.docker.com
Thanks.

authors: WhisperingChaos bgruening dmp42

repo: docker/docker-registry


text:
Link for Docker Engine API v1.37 goes to v1.36 API

File: engine/api/version-history.md
The text Docker Engine API v1.37 links to the v1.36 API

@mbentley Hi Matt - This appears to be fixed. Please let us know if you still see a problem. Thank you!

authors: mbentley paigehargrave


text:
I found a broken link

Page: https://success.docker.com/article/how-do-i-set-the-docker-daemon-options
Clicked link on word 'docs' (marked with docs) in text:
"
All the Docker daemon options are documented at in the Docker docs.
..."
Went to dead link

Thanks for sharing this issue. It appears that this issue is no longer applicable to the current docs because the parent page no longer exists. If you feel this issue still exists, please let us know.

authors: WilliamDugdale craig-osterhout


text:
fixed typo from adressses to addresses

Proposed changes

Unreleased project version (optional)

Related issues (optional)

Thanks, @adamoverton :)

authors: adamoverton johndmulhausen


text:
Add 17.06.1 release notes

Add release for 17.06.1 and refresh the CLI for stable and edge directories
cc/ @andrewhsu

@thaJeztah I'll look at what needs to change to vendor the dockerd reference instead of using the fake YAML. cc/ @johndmulhausen

@thaJeztah the swagger files are fetched at build time. It's still fetched from the 17.06 branch, so no change at this time. Unless there is yet another new API version for this?

Just pushed a commit that pulls the dockerd.md files from upstream.

@thaJeztah let me know when the fixes have been done upstream and I will amend the CLI commit to have the improved output.

@mstanleyjones feel free to merge before that, I'd have to find a spot to make the change (and change is only a minor typo)

authors: mstanleyjones thaJeztah


text:
Fix "virtual" explanation for "docker ps -s"

Proposed changes
Copy from Slack #docs conversation:
Xinfeng Liu [16:35]
Hi,
from https://github.com/moby/moby/blob/master/daemon/getsize_unix.go#L39
and https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/#container-size-on-disk virtual size: the amount of data used for the read-only image data used by the container.   The source codes and docs are not consistent,  I want to confirm which is correct?
Also please see new issue for this: https://github.com/docker/docker.github.io/issues/5270 (edited)
Sebastiaan van Stijn [17:19]
@xinfeng was just reading it this morning (didn't reply on that issue yet); yes, I think an error slipped into the docs: size: is the size used by the writable layer of the container, and virtual size is "writable layer + read-only layers (AKA "the image")". To get the size of "the image" on disk, you'd have to do "virtual size - size". To calculate the total size on disk for N containers running image:foo; the size would be: SUM(size of containers) + 1 x "virtual size - size"
Unreleased project version (optional)

Related issues (optional)
see: https://github.com/docker/docker.github.io/issues/5270

Thanks!
Also added some examples in https://github.com/docker/docker.github.io/issues/5270#issuecomment-343428138 (in case we want a full example to show - but may not be needed)

authors: thaJeztah xinfengliu


text:
Update emergency-repair.md

fixed backtick -> apostrophe that was breaking formatting

Deploy preview for docsdocker ready!
Built with commit 66e27e2d4c7e459eaca32b5e6353137bc7e95ad0
https://deploy-preview-7820--docsdocker.netlify.com

authors: GordonTheTurtle adamancini


text:
Allow additional configuration to be loaded from /etc/sysconfig/docker.d files

If the /etc/sysconfig/docker file is managed by a configuration management system then it does not allow individual configuration of options. Ideally we could set some default options that a user can append their own options to.
OPTIONS=--selinux-enabled

and then another user can extend those options with configuration in the docker.d folder
OPTIONS=-H fd://127.0.0.1

The options can then be combined at runtime without needing additional config management changes to the system. The daemon in the above example would be started with docker -d --selinux-enabled -H fd://127.0.0.1

If your CMS is fully controlling /etc/sysconfig/docker, and doesn't support controlling individual lines in the file, then you can always have your CMS use /etc/sysconfig/docker.d.
Meaning have your CMS create /etc/sysconfig/docker as:
for file in /etc/sysconfig/docker.d/*; do
 [ -f "$file" ] && . "$file"
done

In my own personal opinion, if your CMS is breaking your ability to configure docker, I don't think it's docker's responsibility to fix the situation.

Why not just have that functionality built into the default config? Many other services use .d folders not because something needs "fixing" but because it makes the service more flexible and provides more options for configuration.
That also doesn't solve the problem of combining options if two separate files declare OPTIONS. It can be fixed by declaring each file with OPTIONS=${OPTIONS} --some-option which might be fine but I think better variable loading could be included.

@rothgar It has been detected that this issue has not received any activity in over 1 year. Can you please let us know if it is still relevant:

For a bug: do you still experience the issue with the latest version?
For a feature request: was your request appropriately answered in a later version?
Thank you!

authors: GordonTheTurtle phemmer rothgar

repo: docker/docker


text:
docker-compose and passwords from environment

I was experimenting with docker-compose and I saw something a little bit disturbing in my output of phpinfo() function:  LAMPDB_1_ENV_MYSQL_ROOT_PASSWORD	lamppass
my docker-compose.yml:
lampapp:
build: .
ports:
- "8000:80"
links:
- lampdb
- lampmemcache
volumes:
- /var/
lampdb:
image: "mysql:5"
volumes:
- ./mysql:/etc/mysql/conf.d
environment:
- MYSQL_ROOT_PASSWORD=lamppass
lampmemcache:
image: "memcached"
I know that it is set by 'environment:' but my question: Is it O.K. (I think it is not) any application from my 'composed' docker infrastructure to see set passwords from the installation process ?

Opz. This is for docker/compose not for docker/docker. Closing here, reporting at docker/compose

@ctmnz actually, this is a behavior of docker (not docker-compose). You can find a warning about this in the user-guide; https://docs.docker.com/userguide/dockerlinks/#environment-variables

**Warning: ** It is important to understand that all environment variables originating from Docker within a container are made available to any container that links to it. This could have serious security implications if sensitive data is stored in them.

Thanks for the elaboration @thaJeztah

authors: ctmnz thaJeztah


text:
docker is blocked when starts to attach a container

How to produce:

docker run -it centos /bin/bash
Ctrl-D to exit
docker start -a <cid>
Now docker is blocked.

lldb prints the thread info
(lldb) thread list
Process 52896 stopped

thread #1: tid = 0x1cb6a6, 0x000000000004316b dockerruntime.mach_semaphore_wait + 11, stop reason = signal SIGSTOP thread #2: tid = 0x1cb6a9, 0x0000000000043183 dockerruntime.mach_semaphore_timedwait + 19
thread #3: tid = 0x1cb6aa, 0x000000000004316b dockerruntime.mach_semaphore_wait + 11 thread #4: tid = 0x1cb6ac, 0x0000000000043243 dockerruntime.kevent + 35

docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2
OS/Arch (server): linux/amd64
docker info
Containers: 2
Images: 23
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 27
Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 4.0.3-boot2docker
Operating System: Boot2Docker 1.6.2 (TCL 5.4); master : 4534e65 - Wed May 13 21:24:28 UTC 2015
CPUs: 4
Total Memory: 1.957 GiB
Name: boot2docker
ID: PBDP:CLVG:6BCP:MMCF:457N:7MUQ:57QU:SAA4:2YK6:2GLX:XSUQ:A6VU
Debug mode (server): true
Debug mode (client): false
Fds: 23
Goroutines: 45
System Time: Sat Jun 20 15:18:59 UTC 2015
EventsListeners: 0
Init SHA1: 7f9c6798b022e64f04d2aff8c75cbf38a2779493
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Username: shafreeck
Registry: [https://index.docker.io/v1/]
I use boot2docker on Mac OS X
uname -a
Darwin matrix 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64

Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

This affects 1.7.0 as well :/ @shafreeck could you also paste here uname -a and docker info?

It turns out that's my fault. Use docker start -a -i cid instead of docker start -a , -i indicates to use the interactive mode. I will close this issue.

@shafreeck thanks!

authors: GordonTheTurtle runcom shafreeck


text:
Add image and container labels in logs

It would be nice to push container/image labels in the logs of our containers. We use this to give more context to the logs sent to our syslog/logstash. The log-opt parameter doesn't seem to enable this yet.

Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

Hi @jfroche,
Thanks for reporting this, but I think this has already been propose there : #15816. If you think the same, I think we could close this one as a duplicate from #15816 :wink:.

thanks ! this is a duplicate

authors: GordonTheTurtle jfroche vdemeester


text:
1.9: docker ps truncates image name

If you run docker ps on 1.9 you only get the first 12 characters of the image name
$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                  PORTS               NAMES
fa8aedb577d8        ibuildtheclo        "/sleep.sh"         1 seconds ago       Up Less than a second   8080/tcp            backstabbing_northcutt

Yeah.... that's no good.  Running --no-trunc fixes this, but your going to have some unhappy users with this one.
Client:
 Version:      1.9.0-dev
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   17062ae
 Built:        Tue Sep 29 16:05:42 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0-dev
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   17062ae
 Built:        Tue Sep 29 16:05:42 UTC 2015
 OS/Arch:      linux/amd64

Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

Indeed, I don't like this change either. It was introduced in master a few weeks ago. We should reevaluated for 1.9.

#15472  is to fix the problem

Please don't truncate image:tag. Otherwise we're losing information.
OTOH, trunc ID is ok. Because trunc'd ID is still valid information and we don't lose anything.

authors: GordonTheTurtle calavera coolljt0725 ibuildthecloud wcwxyz


text:
Flaky Test : TestExternalVolumeDriverBindExternalVolume

See https://jenkins.dockerproject.org/job/Docker-PRs/40742/console
09:50:37 ----------------------------------------------------------------------
09:50:37 FAIL: docker_cli_external_volume_driver_unix_test.go:446: DockerExternalVolumeSuite.TestExternalVolumeDriverBindExternalVolume
09:50:37
09:50:37 docker_cli_external_volume_driver_unix_test.go:448:
09:50:37     dockerCmd(c, "run", "-d", "--name", "testing", "-v", "foo:/bar", "busybox", "top")
09:50:37 docker_utils_test.go:108:
09:50:37     c.Assert(result, icmd.Matches, icmd.Success)
09:50:37 ... result *cmd.Result = &cmd.Result{Cmd:(*exec.Cmd)(0xc423579340), ExitCode:125, Error:(*exec.ExitError)(0xc421991d60), Timeout:false, outBuffer:(*cmd.lockedBuffer)(0xc420498870), errBuffer:(*cmd.lockedBuffer)(0xc420498900)} ("\nCommand:  /go/src/github.com/docker/docker/bundles/17.05.0-dev/binary-client/docker run -d --name testing -v foo:/bar busybox top\nExitCode: 125\nError:    exit status 125\nStdout:   7423fa50ee8f1f1a79bdaac59ac0157f66e4d6aa3a82551265518103eb13271d\n\nStderr:   /go/src/github.com/docker/docker/bundles/17.05.0-dev/binary-client/docker: Error response from daemon: endpoint with name testing already exists in network bridge.\n\n")
09:50:37 ... expected cmd.Expected = cmd.Expected{ExitCode:0, Timeout:false, Error:"", Out:"", Err:""}
09:50:37 ...
09:50:37 Command:  /go/src/github.com/docker/docker/bundles/17.05.0-dev/binary-client/docker run -d --name testing -v foo:/bar busybox top
09:50:37 ExitCode: 125
09:50:37 Error:    exit status 125
09:50:37 Stdout:   7423fa50ee8f1f1a79bdaac59ac0157f66e4d6aa3a82551265518103eb13271d
09:50:37
09:50:37 Stderr:   /go/src/github.com/docker/docker/bundles/17.05.0-dev/binary-client/docker: Error response from daemon: endpoint with name testing already exists in network bridge.
09:50:37
09:50:37
09:50:37 Failures:
09:50:37 ExitCode was 125 expected 0
09:50:37 Expected no error
09:50:37
09:50:37
09:50:38
09:50:38 ----------------------------------------------------------------------

This looks like a legit networking issue.
ping @sanimej

Happened again here https://jenkins.dockerproject.org/job/Docker-PRs/42434/console
/cc @mavenugo @aboch


text:
Allow path from normal volume existing to overwrite in start Binds

Fixes #9981
Allows a volume which was created by docker (ie, in
/var/lib/docker/vfs/dir) to be used as a Bind argument via the container
start API and overwrite an existing volume.
For example:
docker create -v /foo --name one
docker create -v /foo --name two

This allows the volume from one to be passed into the container start
API as a bind to two, and it will overwrite it.
This was possible before 7107898d5cf0f86dc1c6dab29e9dbdad3edc9411

To clear up some confusion:

10:04 cpuguy83: icecrime: Yeah, I can see the confusion... Probably easiest to checkout the test in there, but basically this...
10:05 cpuguy83: icecrime: Some people are wanting to take the host path of a normal volume (ie, /var/lib/docker/vfs/dir/) and use that as the host path of a bind-mount in another container.
10:07 cpuguy83: icecrime: Which would be ok too, however in these cases there is often already a volume specified for the dir in the container
10:07 cpuguy83: icecrime: for example VOLUME /foo, and then they want to docker run -v /var/lib/docker/vfs/dir/<id>:/foo
10:09 cpuguy83: icecrime: since Docker goes through the whole "createVolumes" thing on both start and create, it also has to check each volume if it needs to be re-initialized or not (ie, was a new one passed in via volumes-from or Binds)
10:10 cpuguy83: icecrime: The volumes subsystem is storing, by host path, if a volume is a Bind volume or a normal one (handled by the vfs graph driver)
10:11 cpuguy83: icecrime: So currently docker is looking up that path people are wanting in their bind /var/lib/docker/vfs/dir/<id> and it says, "oh nope, that's not a bind" and skips it, since it thinks it's just a normal volume, and as such should never need to be re-initalized
10:12 cpuguy83: icecrime: So, one of the things I've done separately is re-factor this whole mess so that's a bit more obvious, but the fix in gh10365 is just a minimal fix to get it working for this situation.

Thanks @cpuguy83, this helps a lot! So if I understand correctly: bind-mounts and normal volumes are essentially the same behavior, what matters to Docker is keeping track of directories it created itself because it's responsible for their disposal.

One extra question: do we absolutely need this in 1.5.0 (i.e., is it a regression from 1.4.1)? If we don't, should we wait for the ongoing refactor?

what matters to Docker is keeping track of directories it created itself because it's responsible for their disposal.

(this caught my eye); @cpuguy83 would that mean that with this change, docker will see volumes "copied" during the Fig/Compose re-creation step to be a "bind mount", thus docker rm -v container would not cleanup the volume?
TBH I don't know how Fig/Docker handles this currently, just wondering, because that would make it harder to cleanup volumes.
(I think this PR was because of a regression when using Fig/Compose with Docker 1.5)

@cpuguy83 I really don't understand :( Can you explain this magic about isBind: true?

@icecrime This is a regression that occurred in 1.4 when volumes started being created at the same time the container is created (as opposed to just when the container was started).
@LK4D4 "volumes" are really two things, mount points and data. The stuff in daemon/volumes is really dealing with mount points. isBind is saying that the mount is a bind (even though the volume is not). On the /volumes side, IsBindMount is unfortunately named at this point, but is really saying that it's using the host fs as opposed to the vfs graph driver fs.

I got idea, but some day we need to refactor volumes code :)
LGTM

@LK4D4 @cpuguy83
I need to prioritize for the 1.6 timeframe to refactor this code because having this many issues and churn in the same files is a huge sign that there are serious problems.

@crosbymichael Agreed, much of this is already done here: https://github.com/cpuguy83/docker/tree/cleanup_daemon_volumes

@cpuguy83 rebase please! :)

rebased

ping

That cleanup is here: #10422
I think we can close this one in favor of that in the absence of a 1.5.1 release.

Considering that 1.6.0 will be ~March 22nd, it's highly unlikely we do a 1.5.1. I'm closing this but keeping the milestone in case it happens for some reason. Thanks!

@dmp42 You really have to unauth waffle.io :)

LGTM

authors: LK4D4 cpuguy83 crosbymichael icecrime jfrazelle thaJeztah tiborvass


text:
Tell users about how VOLUME initializes the new mount point/volume

Signed-off-by: Doug Davis dug@us.ibm.com

/cc @cpuguy83

LGTM from a technical point of view.

But we may want to add a note in that this does not happen for bind-mounts or with volumes-from.

@cpuguy83 to make sure I grok it... you mean it doesn't apply when:

you specify an existing host path on the -v
attaching to another container's volume
right?

@duglin Correct

@cpuguy83 ok will adjust accordingly - thanks

Updated

looks like a great addition to the docs on a potentially confusing area for the uninitiated.  @moxiegirl, want to do your review magic? :)

oh no!  now I'll have to change the entire section!  :-)

Hey, hey, hey... @estesp  @duglin --- I read your comments after I commented.  Hmm, I don't think they would have stopped me...bwahahahahahahah!  She's mad with the commenting power.

@moxiegirl oh wait, I missed some - hold on....

@moxiegirl ok picked 'em up. Again, with minor tweaks but mainly copy-n-paste.  see what you think.

Thanks Doug for the PR and the extra work. Looks Good To Me oh, right, LGTM

authors: cpuguy83 duglin estesp moxiegirl


text:
Cleanup verifyDaemonSettings function

Signed-off-by: Ma Shimiao mashimiao.fnst@cn.fujitsu.com
parameter networking is deprecated.
Using NetworkMode to replace NetworkDisabled for determining where print warning or not when IPv4Forwarding is disabled.
verifyHostConfig can totally replace verifyDaemonSettings

Hi, check at start time is necessary, we can think about call verifyHostConfig instead but not remove the verify, you can take a look at this one: https://github.com/docker/docker/pull/12619

Hi @hqhq,
I have a different opinion with you.
As I said in #12619
ContainerStart() already calls verifyHostConfig do it before calling container.Start().
So, there is not necessary to do double check.  We just need to enforce verifyHostConfig()

ping @icecrime , @LK4D4 , @duglin , @jfrazelle

This seems reasonable but as I was looking at the code around this I started to wonder about the split between the daemon.ContainerStart() and container.Start().  While not all of the checks are the same, there are some duplicated (like checking to see if its already running).  Should we be looking to move more of these checks into one layer and not duplicate or split them? By having some in daemon and some in container we run the risk of missing some (or getting inconsistent results) if people call the wrong layer.   Would it make sense to have the daemon code do the minimum (look up the container, set hostConfig) and then defer the rest of the logic to the container layer?

And, of course, if that does make sense we could do that for all of the various daemon/container methods that overlap.  Another round of daemon clean-up/reorg!

@duglin , Moving checks to one layer is great!
This must can avoid duplicate or split.
But in my opinion, the daemon code should do these checks. We should apply these configure before making sure they are all OK.
After setting up correct conditions, container layer can just do basic logic to realize the function.
How do you think about it?

@duglin , Moving checks to one layer is great!
This must can avoid duplicate or split.
But in my opinion, the daemon code should do these checks. We should apply these configure before making sure they are all OK. And we should find out error as soon as possible.
After setting up correct conditions, container layer can just do basic logic to realize its function.
How do you think about it?

@Mashimiao I think it depends on which type of check we're talking about.
In my quick scan of the code I saw three types of checks:
1 - e.g. in Start(), is the container already running?  I think this should be done within the container because doing it in the daemon doesn't remove the need to do it in container.Start().
2 - e.g. is some new config value correct? I think doing this in the daemon is correct as long as the container itself doesn't do the same check.
3 - e.g. in Start(), is the container's current config ok? I think this should be in the container because it will need to be checked prior to running starting the container.
I guess the rule I'm following is.... if the function container.foo() does some check then equivalent daemon.foo() should not perform the same check.  Of course, it does beg the question of why we have foo() in daemon and container  :-)
In the case of daemon.ContainerStart() vs container.Start(), it appears that the daemon code is just there to convert name->id and verify/set hostConfig.  Its not clear to me why daemon.setHostConfig() doesn't call verifyHostConfig() itself, assuming setHostConfig() is modified to throw an error when trying to update the config on a paused/running container.  IMO the LogEvent("die") should be moved into container.Start() too, then the err from Start() could just be returned as-is.Then the func just becomes:
func (daemon *Daemon) ContainerStart(name string, hostConfig *runconfig.HostConfig) error {
    container, err := daemon.Get(name)
    if err != nil {
        return err
    }

    // This is kept for backward compatibility - hostconfig should be passed when
    // creating a container, not during start.
    if hostConfig != nil {
        if err := daemon.setHostConfig(container, hostConfig); err != nil {
            return err
        }
    }

    if err := container.Start(); err != nil {
        return err
    }

    return nil
}

Which reduces it down to what it really is, just a thin layer to convert user-provided data.
Ideally, I think it should be container.SetHostConfig(hostConfig) but I might be missing something.

@duglin  Your logic seems very reasonable. I agree with you.
If nobody objected to this, I would update this PR tomorrow according to your opinion.
Thanks a lot.

Please wait for others, like @LK4D4 , to speak-up first because I could be missing something and I don't want you to waste your time doing a re-org if its not the correct approach.

@duglin @Mashimiao I think it is pretty good suggestion.

@duglin  I think I need some help.
I'm not sure what cause testcase failed.

@duglin , I think I need your help.
I'm not sure why testcase always marked as failed.
I have worked on this for several day, but can't find out why.

@Mashimiao can you please rebase?

Hi @duglin  rebased. But Build timed out (after 60 minutes). Marking the build as failed. still exists.

@Mashimiao I think another rebase is needed.  But, first, with all of the changes that have been made in these areas, are these changes still needed?  If so, that's fine- I just wasn't sure.

@Mashimiao Need rebase :/

@Mashimiao please let us know what you want to do with this otherwise we may close it.

Hi @duglin, @LK4D4
I think it's a little complicated to rebase it.
I want to close this one and submit a new PR to fix the similar problem
WDYT?

@Mashimiao Of course, if you think it would be simpler. Sorry for this, we had so big merges recently :(

OK, I will close this one and

authors: LK4D4 Mashimiao duglin hqhq


text:
Add Splunk logging driver #16207

Allow to send Splunk logs using Http Event Collector
Signed-off-by: Denis Gladkikh denis@gladkikh.email
Fixes #16207

This PR is for #16207

Updated OP's description to include the issue it fixes (so that it automatically closes it if this PR is merged)

Thanks very much for also implementing bash completion. That's really cool stuff!

@albers thank you for review! Just updated PR

Bash completion LGTM. Thanks for updating.

Didn't read the code, but I'm +1 for a splunk driver since we already have many logging drivers.
I do have to state again the fact that it's unfortunate we have to compile all of these drivers in, and would prefer to have these as run-time Go plugins whenever they are ready. I understand having out-of-process logging plugins can be a significant performance hit.

ping @sdurrheimer zsh completion: new log-driver

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "splunk_log_driver" git@github.com:splunk/docker.git somewhere
$ cd somewhere
$ git rebase -i HEAD~3
editor opens
change each 'pick' to 'edit'
save the file and quit
$ git commit --amend -s --no-edit
$ git rebase --continue # and repeat the amend for each commit
$ git push -f

Ammending updates the existing PR. You DO NOT need to open a new one.

@outcoldman Is there any easy way to test driver?

@LK4D4 it depends on the definition of "easy". I did all my tests manually. To test it we need to

Download Splunk
Extract it somewhere, in case if it will be inside docker container - we will need to have access to any mapped volume with supported file system.
The test scripts can preconfigure HTTP Event Collector and start Splunk.
After we can just send some logs to Splunk and verify that these events were indexed.

Do you think that we can perform first 2 steps? Do you have an example?

@outcoldman I mean test manually, not unit-tests. Yes, I thought maybe you have docker image with splunk.

Ah, I see, so you are asking for instructions how to test it manually. This make sense.

You can use my own Docker Image with Splunk (https://hub.docker.com/r/outcoldman/splunk/) - this is not officially supported by Splunk image, but will be good enough for testing. Or you can download Splunk directly in Docker Dev image in mapped volume, let's say at /opt/splunk, you can download it from http://www.splunk.com/en_us/download.html
You need to expose port 8000 for Web Interface and 8088 for HTTP Event Collector.
After you will start Splunk open 8000 to get access to the Web Interface, you will need to configure HTTP Event Collector, see http://docs.splunk.com/Documentation/Splunk/6.3.0/Data/UsetheHTTPEventCollector (use port 8088 for HTTP Event Collector, it is a default port, you can use HTTP or HTTPS, if you want to use HTTPS - you will need to configure custom certificates or use ca certificate for Docker daemon to configure Splunk driver or just ignore verification).
I used these parameters for HTTP (let me know if you also want to test HTTPS) to start nginx container (replace path to your Splunk instance and token given to you by Splunk on previous step)

docker run --log-driver=splunk --log-opt splunk-token=176FCEBF-4CF5-4EDF-91BC-703796522D20 --log-opt splunk-url=http://splunk:8088  -d -p 80:80 nginx

After to test the driver I have send few curl requests to nginx container using

curl http://localhost:80/?justatest=hello

In Splunk Web Interface in search string just run "*" and you should see your events.

@outcoldman thanks, I'll try. Moving to code-review in the meantime.

btw need rebase, sorry :)

@LK4D4 np, rebased!

I would recommend accepting both username & password, as well as http event collector tokens.
As it is, I am unable to test this PR because I am using splunk cloud, which does not yet support the http event collector.

Hi @phemmer
Thanks for the input. Event Collector is available in Splunk Cloud, but it needs to be enabled via a support request. There's no UI yet for self-service. But it is definitely available.
As to the username / password. Going forward we will encourage people not to use the REST API to send events as it requires a username and password that can potentially leak access to Splunk. Event Collector tokens are not attached to user credentials, they are specifically designed just for logging events. It is a special endpoint that only allows sending data with a valid token. This makes it more secure from a Splunk perspective as using the token will not allow accessing any part of Splunk, you can't even authenticate to the management port (8089) with it.
You are correct that it is only in Splunk 6.3, but you can stand up dedicated Event Collector instances within a 6.2 environment and they will work fine. EC can run anywhere, and indexer, a forwarder, etc.

Thanks for the input. Event Collector is available in Splunk Cloud, but it needs to be enabled via a support request. There's no UI yet for self-service. But it is definitely available. Please contact Splunk Cloud support and they will get you up and running.

I have:

@phemmer btw you can always use syslog with splunk forwarder if you cannot upgrade to 6.3, see https://outcoldman.com/en/archive/2015/08/22/collecting-docker-logs-and-stats-with-splunk/

@phemmer btw you can always use syslog with splunk forwarder if you cannot upgrade to 6.3, see https://outcoldman.com/en/archive/2015/08/22/collecting-docker-logs-and-stats-with-splunk/

But that defeats the point of this logging driver :-)

@phemmer please email me: gblock@splunk.com. According to cloud folks it is avail now.

@phemmer not really, for some people it can be expensive to install forwarder on every machine where they have docker daemon, especially when we are talking about small tiny VMs in the cloud with 512mb RAM. Also it is much simpler to configure automatic deployment with splunk logger driver, especially if they want to make it secure.

@phemmer I agree with you. The driver is there to mitigate the need for deploying a forwarder :-)

@phemmer Update on this. Sounds like new customers get it now. Existing customers will be upgraded in the next few weeks. I am going to see if we can push yours up to happen sooner.

@phemmer Let us know if it'll work for you with splunk cloud.

As to the username / password. Going forward we will encourage people not to use the REST API to send events as it requires a username and password that can potentially leak access to Splunk. Also EC has been designed in a more scalable / available fashion than the receiver endpoints. Event Collector tokens are not attached to user credentials, they are specifically designed just for logging events. It is a special endpoint that only allows sending data with a valid token. This makes it more secure from a Splunk perspective as using the token will not allow accessing any part of Splunk, you can't even authenticate to the management port (8089) with the token.

Was really busy earlier and didn't have time to respond to this in full, but my thoughts:
I understand the argument behind tokens, and why you shouldn't use user credentials. But we're talking about a feature which is so new, many people don't even have access to it yet (as is my own situation). Yes, for older versions of splunk, you could use syslog/forwarder, but there are many reasons this might not be feasible.
Additionally, I'm always of the opinion that you shouldn't limit people's options because you don't think some of the options should be used. Said people might have unforeseen reasons for using these other options.

You can deploy dedicated 6.3 event collector instances in a hybrid
environment. I agree it is new, but a big problem with the REST API is also
scale. If wasn't really designed for massive scale. The simple receiver is
really slow and the streaming endpoint is really an HTTP hack that won't
scale out easily behind a load balancer.
On Wed, Sep 30, 2015 at 5:13 PM Patrick Hemmer notifications@github.com
wrote:

As to the username / password. Going forward we will encourage people not
to use the REST API to send events as it requires a username and password
that can potentially leak access to Splunk. Also EC has been designed in a
more scalable / available fashion than the receiver endpoints. Event
Collector tokens are not attached to user credentials, they are
specifically designed just for logging events. It is a special endpoint
that only allows sending data with a valid token. This makes it more secure
from a Splunk perspective as using the token will not allow accessing any
part of Splunk, you can't even authenticate to the management port (8089)
with the token.
Was really busy earlier and didn't have time to respond to this in full,
but my thoughts:
I understand the argument behind tokens, and why you shouldn't use user
credentials. But we're talking about a feature which is so new, many people
don't even have access to it yet (as is my own situation). Yes, for older
versions of splunk, you could use syslog/forwarder, but there are many
reasons this might not be feasible.
Additionally, I'm always of the opinion that you shouldn't limit people's
options because you don't think some of the options should be used. Said
people might have unforeseen reasons for using these other options.
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/16488#issuecomment-144579229.

@phemmer would you be willing to test the driver for us if we can get you a Splunk Cloud instance with EC enabled? I've talked to our Cloud Folks, and if you create a trial, we can the open it up with a token for you to test.

@phemmer would you be willing to test the driver for us if we can get you a Splunk Cloud instance with EC enabled? I've talked to our Cloud Folks, and if you create a trial, we can the open it up with a token for you to test.

My splunk cloud account is scheduled to be upgraded tomorrow. So I'll just wait for that.

Ok perfect
On Tue, Oct 6, 2015 at 6:58 PM Patrick Hemmer notifications@github.com
wrote:

@phemmer https://github.com/phemmer would you be willing to test the
driver for us if we can get you a Splunk Cloud instance with EC enabled?
I've talked to our Cloud Folks, and if you create a trial, we can the open
it up with a token for you to test.
My splunk cloud account is scheduled to be upgraded tomorrow. So I'll just
wait for that.
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/16488#issuecomment-146056624.

@cpuguy83 anything else pending approval on this aside from @phemmer's validation?

Are there not going to be any tests for this? Shouldn't be hard to mock out a fake  event collector server.

In regards to my testing, I'm waiting on support to enable the collector

Good call.
On Wed, Oct 7, 2015 at 8:00 PM Patrick Hemmer notifications@github.com
wrote:

Are there not going to be any tests for this? Shouldn't be hard to mock
out a fake event collector server.
In regards to my testing, I'm waiting on support to enable the collector
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/16488#issuecomment-146404017.

@phemmer this is something we'll look into. It doesn't look like tests are a requirement for now though we do have to manually test it. Thanks for the suggestion.

I tried with splunk enterprise and it seems to work. However it require rebase and fix comment from @itay

I'm still waiting for support to enable the collector on my splunk cloud account.
Supporting splunk cloud out of the box instead of making people wait weeks to be able to use docker with it would be great. </DeadHorse>

Totally agree, we're working in it.
On Tue, Oct 20, 2015 at 9:16 AM Patrick Hemmer notifications@github.com
wrote:

I'm still waiting for support to enable the collector on my splunk cloud
account.
Supporting splunk cloud out of the box instead of making people wait weeks
to be able to use docker with it would be great.
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/16488#issuecomment-149619590.

@LK4D4 thanks for testing.

@LK4D4 awesome, thanks! fixed what Itay found and rebased to the latest.

We're working on getting the collector open on @phemmer's account so he can
test.
On Tue, Oct 20, 2015 at 1:47 PM Denis Gladkikh notifications@github.com
wrote:

@LK4D4 https://github.com/LK4D4 awesome, thanks! fixed what Itay found
and rebased to the latest.
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/16488#issuecomment-149696784.

@outcoldman can you squash your commit to one now, when review is almost finished?

@LK4D4 sure, done!

LGTM

LGTM
moving to docs review.

ping @thaJeztah

Thanks @outcoldman, I added my comments inline :+1:
Also, ping @sdurrheimer if he's interested to update the zsh completions, thanks!

@thaJeztah docs are updated!

Thanks @outcoldman I'm still looking, will run a make docs to test if all works correctly.
It looks like you missed the change to move the API changes to the v1.22 docs; https://github.com/docker/docker/pull/16488/files#r42693762
I'll ask @moxiegirl to have a look as well

Thanks for updating, @outcoldman LGTM!

@thaJeztah @moxiegirl thank you for feedback - updated docs.

Thanks all!
On Fri, Oct 23, 2015 at 10:23 AM Denis Gladkikh notifications@github.com
wrote:

@thaJeztah https://github.com/thaJeztah @moxiegirl
https://github.com/moxiegirl thank you for feedback - updated docs.
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/16488#issuecomment-150638174.

@moxiegirl gave me a "LGTM" in person :+1: waiting for Janky to finish

Sweet!
On Fri, Oct 23, 2015 at 10:30 AM Sebastiaan van Stijn <
notifications@github.com> wrote:

@moxiegirl https://github.com/moxiegirl gave me a "LGTM" in person [image:
:+1:] waiting for Janky to finish
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/16488#issuecomment-150640165.

I'd recommend clarifying that one paragraph.  Otherwise, LGTM

Oh, boy.. looks like this needs a rebase :'( sorry for that @mountkin

@thaJeztah np, rebased!

Al green! merging \o/

@outcoldman @glennblock congrats!

Thank you all for helping! And my personal huge thank to the @LK4D4.
Docker folks, you are great!

Thanks for contributing @outcoldman open source FTW!

Yay!!!! This rocks!
On Fri, Oct 23, 2015 at 9:02 PM Sebastiaan van Stijn <
notifications@github.com> wrote:

Thanks for contributing @outcoldman https://github.com/outcoldman open
source FTW!
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/16488#issuecomment-150751170.

Thank you all! We are seeing a lot of interest in Docker in our Splunk
customer base. This is an awesome day!
On Sat, Oct 24, 2015 at 9:11 AM Glenn Block glenn.block@gmail.com wrote:

Yay!!!! This rocks!
On Fri, Oct 23, 2015 at 9:02 PM Sebastiaan van Stijn <
notifications@github.com> wrote:

Thanks for contributing @outcoldman https://github.com/outcoldman open
source FTW!
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/16488#issuecomment-150751170.

Keewl! =)

@thaJeztah Looks like this was missed and not cherry picked in.  Any idea when 1.9.3 is going out?

@moxiegirl I dunno why you think it should be in a milestone. It's a new feature == shouldn't be in the minor release.

There was some confusion about the timing, but this PR was merged after the 1.9.0 code freeze (1.9.0-rc1 was released on October 14th).
This driver will be part of the 1.10 release. For people wanting to give it a spin before that, it is available in the "Experimental" or "Master" builds (not intended for production)

Our understanding was it was part of 1.10. Thanks for confirming.
On Thu, Dec 10, 2015 at 4:48 PM Sebastiaan van Stijn <
notifications@github.com> wrote:

There was some confusion about the timing, but this PR was merged after
the 1.9.0 code freeze (1.9.0-rc1 was released on October 14th
https://github.com/docker/docker/releases/tag/v1.9.0-rc1).
This driver will be part of the 1.10 release. For people wanting to give
it a spin before that, it is available in the "Experimental" or "Master"
builds (not intended for production)
—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/pull/16488#issuecomment-163797851.

Maybe the splunk driver could receive more than one host in "--log-opt splunk-url" for the cases splunk running in cluster ?

Thanks for the suggestion. With HEC the recommendation is always to have a load balancer in front of the HEC pool. Based on that I don't think we need clustering support. Also that will complicate the implementation.

authors: GordonTheTurtle LK4D4 albers calavera glennblock juniorjbn moxiegirl outcoldman pandrew phemmer thaJeztah tiborvass


text:
Add user namespace check to check-chonfig.sh

Signed-off-by: Lei Jitang leijitang@huawei.com
#12648 has add support user namespace, thanks @estesp 's amazing work.
We should update the check-config.sh to check USER_NS now.

Thanks @coolljt0725!

However it should be in optional.

update, add user namespace check in optional.

LGTM

LGTM 🐼


text:
Rename docker run "--cpus" to "--limit-cpu"

This fix renames docker run "--cpus" to "--limit-cpu", based on related discussion in #28095. Related docs and tests have been updated accordingly
(Feel free to close it if there are concerns, and feel free for any suggestions).
Signed-off-by: Yong Tang yong.tang.github@outlook.com

please update bash/zsh completion as well

Thanks @AkihiroSuda. The PR has been updated for bash completion.
cc @albers

Bash completion LGTM, thanks @yongtang

SGTM but ping @crosbymichael @cpuguy83 @tiborvass

Either way I'm fine with it.
--limit-cpu makes me think more of percentage rather than cores.
Help output in either case should be much better

@cpuguy83 The help output has been updated. Please take a look.
I don't have a preference either way, though if we decide --limit-cpu, I am thinking it might be good to have --limit-memory as well?
For many users, it is natural to pair the restriction of memory and cpu together.
If we want to stay with --cpus, I can update the PR to only include the help output changes.

@yongtang Thanks, help output is much more clear now.

Wait why are we doing this?

@yongtang sorry, I don't agree with the original issue for this change and it was rushed. Lets close this for now and I'll speak with the issue author about it.
Thanks for all the help and being so fast on fixing things though ;) You are doing awesome work.

authors: AkihiroSuda albers cpuguy83 crosbymichael vdemeester yongtang


text:
Specify required node version

CONTRIBUTING.md only says that Node is needed but it does not say which version should be used. Correct me if I'm wrong but this suggests to me that the current version should be good. Especially because there is neither an .nvmrc file in the project nor an engines key set in package.json.
Turns out the tests won't even run on node 6.
Looking in .travis.yml I see that the CI tests run on 4.2.2 what suggests that this is the version everyone should develop against.
Options of fixing that are:

Adding the version to CONTRIBUTING.md (Which has a high risk of going stale I think)
Adding a .nvmrc
Add an engines key to package.json

Interestingly enough I thought I had made a PR for this - It would be nice to have it in the CONTRIBUTING.md, but as mentioned could go stale pretty easily.
The cleanest is probably to add the proper engines entry to package.json. Do you think you could make a PR for this?

Sure I will do it tomorrow (CEST)

authors: Fahrradflucht FrenchBen

repo: docker/kitematic


text:
Ports project volume cleanup from docksal#1058

Changes

All project_root named volumes were leftover after proxyctl cleanup ran, even for projects that no longer exist. After discussing with @lmakarov, backporting the change in https://github.com/docksal/docksal/pull/1058/files#diff-41adca34c3eac75ca9453c7cb0634f6aR1285 makes sense to have here.

Alternatively, we could replace:
	log "Removing dangling volumes..."
	docker volume prune -f

with:
	log "Removing dangling volumes..."
	docker volume ls -qf dangling=true | xargs -r docker volume rm

We cannot use docker volume ls -qf dangling=true | xargs -r docker volume rm due to https://github.com/docksal/docksal/issues/1056. It can only be used as a one-time fix if you know what you are doing.

authors: lmakarov lpeabody

repo: docksal/service-vhost-proxy


text:
Snapshot / Export Dialog

Based on these mocks.
dockstore/dockstore#2486
I want to get some feedback. These things aren't done:

Integration tests should have more coverage
ORCID not tested (need to set up locally)
Pixels to rem?
No line connecting icons as in mocks (probably won't do)

Some screenshots

We know that mat-icon and fontawesome can take care of snapshot and ORCID. Is there no other solution than importing a whole extra css file?

Yeah, not wild about it, but that's the only place I could find that icon. Kim gave me a couple, but they were the text portion only. I could try and get another one from her, or we could just go with numbers or no icons at all.

What there a reason that mat-horizontal-stepper wasn't used? It would've automatically brought back the lines connecting the steps.

I tried that at first and struggled to get it working. Custom icons behaved flakily; we're not really going from page to page like a typical stepper, you might need to start on the second step, e.g., if it has already been snapshotted. The code is in earlier commits, so we can revisit.

The link account button looks grey in the screenshot (which i've never seen that style before), but it looks white in the mocks (which kind of resembles what I've seen before)

I think that happens because it has focus (it's the first control), and the hover state is causing that. I'll take another look.

The link account button looks grey in the screenshot (which i've never seen that style before), but it looks white in the mocks (which kind of resembles what I've seen before)

I think that happens because it has focus (it's the first control), and the hover state is causing that. I'll take another look.

Confirmed that it's because of focus. Tabbing off it shows the white background. I'm leaning towards leaving it since that is presumably the correct behavior for a focused control.

Now looking like this:


text:
bitbucket bug

#3568 just check path is not empty

Do the other SourceCodeRepo have a problem like this? If it's a simple change, perhaps move it to the abstract class

Looks like GitHub and GitLab do not result in an error/weird sourcefile if you set the test path to be blank

repo: dockstore/dockstore


text:
"Subclassed" Categories webservice prototype

Hi there y'all!  I've finished another proof-of-concept of a piggybacked Categories webservice implementation, in which Category is a subclass of Collection and the two share the collection db table.
Please take a quick look.  The code is intentionally rough, there's dangling cruft, spelling errors (!), etc., so, as with the last prototype, I'm looking for high-level feedback as to the approach and its desirability versus the previous prototype and "pure Categories".   If we decide to graft this into develop, I'll make a few passes to ensure that the code is up to snuff, add elasticsearch support, write some tests, convert the db setup script to our migration scheme, etc.  If you see any functional problems that you don't think I'll find during that process, it would be great to know about those too.
Run scripts/install-categories.sh <username> to set up your local db.  It will create a 'SPECIAL' organization named "dockstoreCategories", add the specified existing user as it's admin, and add a column to the collection db table.
After that, fire up the webservice+ui, log in as the dockstoreCategories admin user, navigate to "/organizations/dockstoreCategories", and add some categories!
Some new endpoints are exposed, slightly different than the last prototype:
/categories
/categories/names/{name}
/entries/{id}/categories
There is some philosophical debate about the above naming, it's easy to change if we decide to do so.
Each CollectionEntry now includes a categoryNames property.
In this implementation, Category/Collection identity is decided at creation time and stored in db column collection.dtype, using JPA single table inheritance. Both Categories and Collections are created/managed via the existing Collection endpoints.
The 'SPECIAL' application status has been added and does double duty: both as a simple mechanism to hide the category-managing organization, and as a trigger for Category creation.  The current logic is to create a Category when the creating organization is 'SPECIAL', and a regular Collection otherwise.
Multiple special organizations may exist, and may each own categories, allowing us to partition category management.  The architecture allows categories to be owned by any organization, although it's not possible to configure the system that way via the endpoints at present.  A category owned by a public organization would appear both in that organization's list of collections, and in the global list of categories.
Via the category-specific endpoints, all categories are always visible to everyone.  That's an ok policy, right?
I'll need to add a db constraint to enforce unique lower(names) amongst Categories.
There may be a better adjective to describe a category-managing organization than 'special'.
This code currently bombs the ITs, because I don't have the db migration coded the appropriate way.
Whee!

Spoke to Denis about it. I prefer the other PR. It's more intuitive to me. But if everyone likes this more then 🤷

authors: garyluu svonworl


text:
SEAB-6465 Add DB fields to aid periodic tasks - topics

Description
Add a field to record what versions have been used for topic generation (and clears it if it is updated). Add an endpoint to return a latest version for consideration for topic generation.
Will need minor dockstore-support following PR
Review Instructions
n/a probably will need dockstore-support follow-up
Issue
https://ucsc-cgl.atlassian.net/browse/SEAB-6465
Security and Privacy
None
Please make sure that you've checked the following before submitting your pull request. Thanks!

[x] Check that you pass the basic style checks and unit tests by running mvn clean install
[x] Ensure that the PR targets the correct branch. Check the milestone or fix version of the ticket.
[x] Follow the existing JPA patterns for queries, using named parameters, to avoid SQL injection
[x] If you are changing dependencies, check the Snyk status check or the dashboard to ensure you are not introducing new high/critical vulnerabilities
[x] Assume that inputs to the API can be malicious, and sanitize and/or check for Denial of Service type values, e.g., massive sizes
[x] Do not serve user-uploaded binary images through the Dockstore API
[x] Ensure that endpoints that only allow privileged access enforce that with the @RolesAllowed annotation
[x] Do not create cookies, although this may change in the future
[x] If this PR is for a user-facing feature, create and link a documentation ticket for this feature (usually in the same milestone as the linked issue). Style points if you create a documentation PR directly and link that instead.

After we generate an AI topic, we save the request that we submitted to the AI system, in the form of a hash of the request. Then, to test if we need to regenerate, build the request that corresponds to the current entry state, hash it, and if differs from the saved hash, submit the request to the AI system to regenerate the topic.

Not exactly an objection, but I would like the first pass to be relatively simple and then we can collect data on effectiveness, billing, etc. before deciding to optimise (or prematurely optimise?) more and in what way we optimise.
Going to create https://ucsc-cgl.atlassian.net/browse/SEAB-6484 for further discussion on this and the "what branch or tag" to consider question.

Also followed by https://ucsc-cgl.atlassian.net/browse/SEAB-6485 for the CLI portion

(hoverfly failure seems to be separate issue from develop, noted in #dockstore)


text:
Hyperlink bug #4

Clicking on the hyperlink to see the modal, unreliable in two ways:

Sometimes the modal doesn’t appear and I need to click elsewhere on the screen and then the hyperlink again to see it.

Sometimes the modal shows up in the wrong place: the row on top of the hyperlink, the row under it, right on top of the hyperlink, to it’s left, or right, or far from the hyperlink, usually at the very bottom corner of the page, sometimes so far down that you need to scroll to click it.

Steps to replicate:

Click on a hyperlink
Click anywhere else on the page
Click the same hyperlink again
Repeat a few times - each time it will show up at a different place.

@Marinaobz Please test it again and confirm if it is resolved.

Works perfect!

authors: HMarzban Marinaobz


text:
Filter: support fuzzy match ?

Description
If doctest support filter prefix match or fuzzing match.
examples:
TEST_CASE("foo::") {}
TEST_CASE("zoo::") {}

# use this command to filter
path/to/test -tc=foo

Now, I test it's not support fuzzy match or prefix match.

Doctest support wildcards in test case filtering. Use this flag when you run the tests -tc=foo::*

authors: cdeln wendajiang

repo: doctest/doctest