text:
PyPI package has a wrong repository link

The PyPI project site of this project as well as the one of the other organisation projects ds-store and mac-alias all have outdated links. If you click source, you get GitHub 404 errors. It would be good if these could be updated to the current organisation repository links.

Thanks for the report. The links are definitely old - but at least for me, they're still resolving (the dmgbuild links both resolve to the dmgbuild/dmgbuild project on Github. However, they should definitely be updated to reflect this directly.

Oh, it depends where you click
If you click on Homepage on the left bar:

it leads to http://alastairs-place.net/projects/ds_store and redirects to this repository (https://github.com/dmgbuild/ds_store) correctly.
However, if you click on "Source" (which is what I did), it leads to https://github.com/al45stair/ds_store leading to a 404 error:

The licence and CI buttons on the top

directly lead to the correct links in this repository.

@freakboy3742 I'm sorry, but sadly this issue is still not resolved. :/
If you go to https://pypi.org/project/dmgbuild/, all the links are still as I described them. Maybe you need to release a new version or push these changes to pypi somehow for them to take effect?

Yes - I would need to publish a new release for these changes to take effect. However, given that this is a relatively cosmetic change, that in itself isn't a trigger for cutting a new release.

v1.6.2 has now been released, so the URLs are now updated.

Yay, that's really great to hear!

repo: dmgbuild/dmgbuild


text:
gluonnlp on Cuda 10 cannot import name 'replace_file'

I am trying to run gluonnlp on cuda 10. I have installed the following package:
mxnet-cu100              1.6.0b20191102
https://pypi.org/project/mxnet-cu100mkl/#history
As it seems like the only one that has mxnet 1.6.0 and supports cuda 10. I have no possibility to upgrade to cuda 10.1 which has the official release for mxnet 1.6.0
The error I get looks like I am still on mxnet 1.5.0 but:
import mxnet as mx
print(mx.__version__)

1.6.0

It looks like the pre-release is not compatible? Do you know of any way to use gluonnlp on cuda 10.0? I cannot neither downgrade nor upgrade the cuda version.
The original error is:
ImportError                               Traceback (most recent call last)
<ipython-input-2-12f1dea70852> in <module>
     13 from mxnet import gluon, autograd
     14 from mxnet.gluon.utils import download
---> 15 import gluonnlp as nlp
     16 nlp.utils.check_version('0.7.0')

~/.local/lib/python3.6/site-packages/gluonnlp/__init__.py in <module>
     23
     24 from . import loss
---> 25 from . import data
     26 from . import embedding
     27 from . import model

~/.local/lib/python3.6/site-packages/gluonnlp/data/__init__.py in <module>
     21 import os
     22
---> 23 from . import (batchify, candidate_sampler, conll, corpora, dataloader,
     24                dataset, question_answering, registry, sampler, sentiment,
     25                stream, super_glue, transforms, translation, utils,

~/.local/lib/python3.6/site-packages/gluonnlp/data/question_answering.py in <module>
     29
     30 from mxnet.gluon.data import ArrayDataset
---> 31 from mxnet.gluon.utils import download, check_sha1, _get_repo_file_url, replace_file
     32 from .registry import register
     33 from ..base import get_home_dir

ImportError: cannot import name 'replace_file'

can you try later versions of mxnet 1.6 from https://dist.mxnet.io/python
e.g. mxnet_cu100-1.6.0-py2.py3-none-manylinux1_x86_64.whl ?

Closing the issue, as you are using an unreleased nightly build of MXNet 1.6. Please upgrade to the released version (pip install --upgrade mxnet-cu100)

eric-haibin-lin thank you very very much! this solved my problem!
leezu, pip install --upgrade mxnet-cu100 this did not work as the mxnet-cu100 had 1.5.1
https://pypi.org/project/mxnet-cu100/
And we needed 1.6.1
Eric's solution worked!

@ktoetotam  I am not getting that 1.6.1 version file, can you help me in that.

source: gharchive/issue

authors: eric-haibin-lin ktoetotam leezu shibuML

repo: dmlc/gluon-nlp

license_type: permissive

license_source: github-api


text:
Wrote Amazon Linux CPU CI setup

Wrote Dockerfile for Amazon Linux CPU tests environment, dependency installation scripts, and fix for lint failure.

does this install jenkins?

No, this itself doesn't build Jenkins. This is for Jenkins to set up build slaves.
The Dockerfile is the build slave image, the test script will be called as a build step in Jenkins.

ok. CPU only? Are you going to make seperate gpu testing image?

Yes, CPU only for this commit. I'll write more for Ubuntu and GPU, plus any other varieties of environments we need.

source: gharchive/pull-request

repo: dmlc/mxnet


text:
[tutorial] UINT8 quantization inference example

I have converted tensorflow quantization model into onnx.
Referring Compile ONNX Models tutorial, I got runtime error:
NotImplementedError: Operator FakeQuantWithMinMaxVars not implemented
I have read this quantization inference paper, is that means I have to  implement FakeQuantWithMinMaxVars operator byself?
Would you please give a UINT8 quantization example?

please use https://discuss.tvm.ai/ for general discussions, the support for int8 inference is work in progress and we will put out more resources in next release cycle

@tqchen Is UINT8 quantization supported now? I also uses that BBS but hardly get any answers. Thanks!

Hi @tqchen
Because I need to import the pre-quantized model from Tensorflow but not from TF-lite, I need the FakeQuantWithMinMaxVars operator for the following steps.
Therefore, the issue-706 in "discuss.tvm.ai" may not be suitable for me
https://discuss.tvm.apache.org/t/tensorflow-operator-fakequantwithminmaxvars-not-implemented/706
Would you please put out more resources about the FakeQuantWithMinMaxVars?
or Could you tell me how to build/implement the non-existed op by myself?
Thanks

Hi @tqchen
Because I need to import the pre-quantized model from Tensorflow but not from TF-lite, I need the FakeQuantWithMinMaxVars operator for the following steps.
Therefore, the issue-706 in "discuss.tvm.ai" may not be suitable for me
https://discuss.tvm.apache.org/t/tensorflow-operator-fakequantwithminmaxvars-not-implemented/706
Would you please put out more resources about the FakeQuantWithMinMaxVars?
or Could you tell me how to build/implement the non-existed op by myself?
Thanks

authors: WeberChen ghostplant tpoisonooo tqchen

repo: dmlc/tvm


text:
Error when calling xgboost-spark 's XGBoost.trainWithDataFrame

For bugs or installation issues, please provide the following information.
The more information you provide, the more easily we will be able to offer
help and advice.
Environment info
Operating System:
Compiler:
Package used (python/R/jvm/C++):jvm
xgboost version used:0.7
Hi guys,
I use xgboost-spark in eclipse, however, I met problem:
Looks like I have error when calling "XGBoost.trainWithDataFrame", red slash is a sign.

Does any one have idea how to fix the problem?
Thanks & Regards!

What is the type of trainingSet?

@superbobry ,it is Dataset

@superbobry ,it is Dataset

@superbobry ,it is Dataset<>

@superbobry , it is Dataset"Row"

@superbobry , I use XGBoost.trainWithRDD， problem still exists:

@superbobry ， how can I fix the problem?

Can you post the compilation error?

@superbobry ， It is just "Description	Resource	Path	Location	Type
The method trainWithDataFrame(Dataset<?>, Map<String,Object>, int, int, ObjectiveTrait, EvalTrait, boolean, float, String, String) in the type XGBoost is not applicable for the arguments (Dataset, Map<String,Object>, int, int, null, null, boolean, float, String, String)	XgboostNew.java	/riskcontrol/src/main/java/com/xianjinkd/riskcontrol/access/algorithm	line 154	Java Problem". Compilation error.

@superbobry ， do you have any idea, how can I fix the problem?

repo: dmlc/xgboost


text:
xgb.load() error in R while the JSON model file path containing Chinese

I've got an error while using xgb.load() to load a JSON model file which was saved by xgb.save().
Like this:
Error in xgb.Booster.handle(modelfile = modelfile)
I found that if the model file path did not contain Chinese characters and it will work perfect!
The version of xgboost package is 1.7.5.1.
By the way, if I use xgb.save() with the extension 'model' and load the model file, it also works.
Hoping this issue will be fixed in the future release.

It seems to be a R-specific issue.

Nope, it works fine on my system (ubuntu 22.04) with R 4.1.2. Maybe it's the system locale configuration.

I've tried it on 2 different computers running Windows 10 and 11 with R 4.1.2,
but I've still got the same error.

The issue is specific to Windows. When opening files with Windows, we need to build std::filesystem::u8path object (to represent a UTF-8 string) and pass it to std::ifstream. See https://gitlab.com/inkscape/inkscape/-/merge_requests/3784.


text:
[coll] Reduce the amount of open files (socket).

Reduce the chance of hitting Too many open files.

cc @rongou .


text:
Added KDD Cup 2016 competition

Added ouselves to your showcase for xgboost winning major ML competitions.

Which reasons to delete other 3 links?

I don't think I deleted anything, or it wasn't my intention anyway.
Looking at the file changes I only see my addition, no deletions.

Sorry, my misstake.

authors: il-rle vladsandulescu


text:
[GPU-Plugin] Major refactor

Refactoring in preparation for integration with the rest of the codebase.
Reduced LOC by around 1000 in GPU algorithms and reduced number of files.

@teju85
I have refatored to remove the exact namespace, move code into a single file and reduced a few verbose sections of code.
I also removed the exact google tests for now. Feel free to re-add them at some point if you feel like updating them.

@RAMitchell
Sounds good to me. Thanks for letting me know.

Codecov Report

Merging #2644 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2644   +/-   ##
=======================================
  Coverage   37.63%   37.63%
=======================================
  Files          82       82
  Lines        7254     7254
  Branches      710      710
=======================================
  Hits         2730     2730
+ Misses       4386     4385    -1
- Partials      138      139    +1

Impacted Files
Coverage Δ

src/tree/tree_updater.cc
61.53% <ø> (ø)
:arrow_up:

src/data/sparse_page_dmatrix.cc
86.88% <0%> (-1.1%)
:arrow_down:

src/data/simple_dmatrix.cc
57.33% <0%> (+0.66%)
:arrow_up:

src/data/sparse_page_raw_format.cc
84.61% <0%> (+1.92%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba16475...db86cd7. Read the comment docs.

authors: RAMitchell codecov-io teju85


text:
Crash On Opening Picker If Date Exceeds Jan 1, 2100 While Using disablePast Prop

Environment

Tech
Version

material-ui-pickers
1.0.1

material-ui
3.2.2

React
16.5.2

Browser
N/A

Peer library
Luxon (1.4.5)

Steps to reproduce

Ensure the datepicker component has the disablePast property enabled
Select a date beyond Jan 1st (in a year before 2100)
Use the year selector to set the year to 2100
An invalid date message will appear
Clicking to open the DatePicker will crash

Expected behavior
When an invalid date is selected, opening the date picker will default to selecting the maximal date of Jan 1st, 2100.
Actual behavior
TypeError: Cannot read property 'set' of null
    at LuxonUtils.setHours (C:/Users/JSaunders/PhpstormProjects/cnx-react-lib/node_modules/material-ui-pickers/utils/luxon-utils/index.esm.js:91)
    at LuxonUtils.mergeDateAndTime (material-ui-pickers/utils/luxon-utils/index.esm.js:116)
    at Calendar._this.onDateSelect (material-ui-pickers/DatePicker/components/Calendar.js:49)
    at Calendar.componentDidMount (material-ui-pickers/DatePicker/components/Calendar.js:144)
    at commitLifeCycles (react-dom.development.js:14685)
    at commitAllLifeCycles (react-dom.development.js:15905)
    at HTMLUnknownElement.callCallback (react-dom.development.js:145)
    at Object.invokeGuardedCallbackDev (react-dom.development.js:195)
    at invokeGuardedCallback (react-dom.development.js:248)
    at commitRoot (react-dom.development.js:16075)
    at completeRoot (react-dom.development.js:17463)
    at performWorkOnRoot (react-dom.development.js:17391)
    at performWork (react-dom.development.js:17295)
    at performSyncWork (react-dom.development.js:17267)
    at interactiveUpdates$1 (react-dom.development.js:17558)
    at interactiveUpdates (react-dom.development.js:2208)
    at dispatchInteractiveEvent (react-dom.development.js:4913)

Minimal example
https://pastebin.com/1kuH5XTs - removing the disablePast property stops crashes on opening the datepicker.

Published @date-io/luxon v0.0.2 with the fix

authors: dmtrKovalenko jsau-

repo: dmtrKovalenko/material-ui-pickers

license: mit

license_source: bigquery


text:
return list of dictionary with common structure when common_dict=1

all reqmgr2 get call have common_dict param. When it is set to 1 it returns list of dictionary instead of dictionary with request name(RequestName) as a key.

https://reqmgr2-dev.cern.ch/reqmgr2/data/request?outputdataset=/Pythia8_Dijet15_pp502_TuneCUETP8M1/Integ_Test-MonteCarlo_Ext_reqmgr1_test-v1/GEN-SIM&common_dict=1
Previous key is under RequestName field.

@vkuznet, Valentin, please review.

@vkuznet,
Hi Valentin, I updated the patch so return structure is controlled by requestor. filter query will contol the return structure.
https://reqmgr2-dev.cern.ch/reqmgr2/data/request?outputdataset=/Pythia8_Dijet15_pp502_TuneCUETP8M1/Integ_Test-MonteCarlo_Ext_reqmgr1_test-v1/GEN-SIM&common_dict=1&filter=RequestName&filter=EventsPerLumi
Will return,
{"result": [
{
"RequestName": "sryu_MonteCarlo_Ext_Sryu_reqmgr1_160304_080525_8176",
"EventsPerLumi": 100
}]}

Seangchan,
even though this particular URL works, this creates another 2 problems:

client should to figure out somehow which fields are available to query.
I think it can be resolved if you'll provide API to look-up all filter fields, e.g.
https://reqmgr2-dev.cern.ch/reqmgr2/data/filters will return list of acceptable
filters.

with large list of filter parameters GET request can easily reach its limit,
whatever it is, and so far I can't query via POST, see below. We should be able
to pass a list of parameters in a request, rather specify each filter individually
as you provided in your example.

scurl -X POST -H "Content-type:application/json" -H "Accept:application/json" -d '{"outputdataset":"/Pythia8_Dijet15_pp502_TuneCUETP8M1/Integ_Test-MonteCarlo_Ext_reqmgr1_test-v1/GEN-SIM", "common_dict":1, "filter":["RequestName","EventsPerLumi"]}' "https://reqmgr2-dev.cern.ch/reqmgr2/data/request"
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
    <title>400 Bad Request</title>

400 Bad Request

Invalid spec parameter value: RequestString need to be specified

    Powered by CherryPy 3.2.2

On  0, ticoann notifications@github.com wrote:

@vkuznet,
Hi Valentin, I updated the patch so return structure is controlled by requestor. filter query will contol the return structure.
https://reqmgr2-dev.cern.ch/reqmgr2/data/request?outputdataset=/Pythia8_Dijet15_pp502_TuneCUETP8M1/Integ_Test-MonteCarlo_Ext_reqmgr1_test-v1/GEN-SIM&common_dict=1&filter=RequestName&filter=EventsPerLumi
Will return,
{"result": [
{
"RequestName": "sryu_MonteCarlo_Ext_Sryu_reqmgr1_160304_080525_8176",
"EventsPerLumi": 100
}]}

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6700#issuecomment-197484024

Hi Valentin,
For 1.  We can provide list of keys they can filter.  But I don't mean DAS will support flexible query from a large set of filter. I think users need to specify what they want ahead. If they want to know what information need to get providing inputdataset. They have to spell it out before this gets to add DAS.
Or if you want to get DAS more flexible. We can give the minimal set of filter with the api your suggested. But still the number of filters they can search need to be controlled, not give them all possible numbers and they can choose among them. If user really wants to do that they can directly query reqmgr2.
For 2. post method for the API is currently used by inserting the data. We could provide post call later if necessary. But for the filter, I would like to see what information they need to extract first. If we end up needing bunch of filters. We can send those information by default or one separate flag.

Seangchan,
the filters you introduced is posterior information. Users can't know a-priory
what is available unless you tell them though. DAS will not manipulate filters,
I'll put them into DAS maps and that what DAS will return all the time for that
or other APIs. Users will get records with this info.
How you gonna handle it is up to you and your best shot is to provide
minimum information and add gradually. Nobody will provide you any response,
as we learnt from many times everybody is too busy. But people will speak
if they miss something only when they query something, not another way around.
On  0, ticoann notifications@github.com wrote:

Hi Valentin,
For 1.  We can provide list of keys they can filter.  But I don't mean DAS will support flexible query from a large set of filter. I think users need to specify what they want ahead. If they want to know what information need to get providing inputdataset. They have to spell it out before this gets to add DAS.
Or if you want to get DAS more flexible. We can give the minimal set of filter with the api your suggested. But still the number of filters they can search need to be controlled, not give them all possible numbers and they can choose among them. If user really wants to do that they can directly query reqmgr2.
For 2. post method for the API is currently used by inserting the data. We could provide post call later if necessary. But for the filter, I would like to see what information they need to extract first. If we end up needing bunch of filters. We can send those information by default or one separate flag.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6700#issuecomment-197506807

Actually, that is my point. I am not sure why initially it was needed and I wonder after you removed DAS map for this due to the performance problem, did any one complain about the information they can't find?
If they don't need the information, we don't need to provide. When they complain about missing information, we will provide that information. We can go with minimal set we might think useful (maybe 0 initially :smile:), but I don't think we need to start with all possible information first.

Hi Valentin,
I hard coded what information to return when filter=DAS is set. I currently have these four keys but I will consult with Alan what to add.
["RequestName", "OutputDatasets", "InputDatasets", "PrepID"]
https://reqmgr2-dev.cern.ch/reqmgr2/data/request?status=completed&request_type=MonteCarlo&common_dict=1&filter=DAS
returns
{"result": [
 {
  "InputDatasets": [],
  "PrepID": "SUS-Summer12pLHE-00144",
  "RequestName": "sryu_MonteCarlo_LHE_Sryu_reqmgr1_160304_080530_5998",
  "OutputDatasets": [
    "/SMS-T5wg_2J_mGl-1550to1600_mLSP-25to1575_TuneZ2star_8TeV-madgraph-tauola/Integ_Test-MonteCarlo_LHE_reqmgr1_test-v1/GEN"
  ]
}]}

@amaltaro, I updated patch as you suggested. TaskChain and StepChain are handled duplicate values allowed for that which Task has the value in order (Task1, Task2, ...)
https://reqmgr2-dev.cern.ch/reqmgr2/data/request?status=ACTIVE&filter=DAS&request_type=TaskChain
{"result": [
{
"PrepID": [
"task_EXO-RunIIFall15DR76-03799",
"EXO-RunIIFall15DR76-03799",
"EXO-RunIIFall15DR76-03799",
"EXO-RunIIFall15MiniAODv1-02471"
],
"RequestName": "sryu_McMTaskChain2_Sryu_reqmgr2_160304_080733_673",
"GlobalTag": [
"76X_mcRun2_asymptotic_v12",
"76X_mcRun2_asymptotic_v12",
"76X_mcRun2_asymptotic_v12",
"76X_mcRun2_asymptotic_v12"
],
"InputDatasets": null,
"Campaign": "reqmgr2-validation",
"CMSSWVersion": [
"CMSSW_7_6_1",
"CMSSW_7_6_1",
"CMSSW_7_6_1",
"CMSSW_7_6_1"
]
}]}

And as we discussed during the meeting, the use case for DAS should be always the same, as:

user provides a dataset as input, and ReqMgr has to provide information regarding THIS dataset, not the workflow itself. The keys were already discussed. What is not provided by ReqMgr2, is provided by PhEDEx or DBS, so it should be Ok.

I will start with minimum. Actually, I didn't get about those property should not be a list.
If the prepID is different for each Task/Step, how would you organize?

We have two use cases here:
a) a DAS query: in this case the user knows exactly which dataset he wants to look at, so we do not need to provide a list of PrepID, that will only confuse him and not give a real answer. For DAS case, we need to fetch the correct PrepID, GlobalTag and etc
b) general case (anyone asking reqmgr2 for some information, a request information, etc): in this case, there is no dataset in the call (usually), so the user does not want info for a specific dataset, but instead for a whole request and thus we need some info to be in a list format.
If you prefer we can discuss over vidyo today

Thanks to Alan, I now understand more about DAS and general uses cases. However, current reqmgr document doesn't specify which task creates which outputdataset. OutputDataset is the just list outside the Task. We need to create the map in reqmgr document in the request document.
Right now only thing I can do from quest which passes output dataset. is following
https://reqmgr2-dev.cern.ch/reqmgr2/data/request?outputdataset=/ZprimeToA0hToA0chichihbb_2HDM_MZp-2500_MA0-300_13TeV-madgraph/RunIIFall15DR76-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v0/AODSIM&request_type=TaskChain&mask=DAS&common_dict=1
returns
{"result": [
 {
  "PrepID": [
    {
      "TaskName": "EXO-RunIIFall15DR76-03799_0",
      "PrepID": "EXO-RunIIFall15DR76-03799"
    },
    {
      "TaskName": "EXO-RunIIFall15DR76-03799_1",
      "PrepID": "EXO-RunIIFall15DR76-03799"
    },
    {
      "TaskName": "EXO-RunIIFall15MiniAODv1-02471_0",
      "PrepID": "EXO-RunIIFall15MiniAODv1-02471"
    }
  ],
  "RequestName": "sryu_McMTaskChain2_Sryu_reqmgr2_160318_050204_3263",
  "GlobalTag": [
    {
      "TaskName": "EXO-RunIIFall15DR76-03799_0",
      "GlobalTag": "76X_mcRun2_asymptotic_v12"
    },
    {
      "TaskName": "EXO-RunIIFall15DR76-03799_1",
      "GlobalTag": "76X_mcRun2_asymptotic_v12"
    },
    {
      "TaskName": "EXO-RunIIFall15MiniAODv1-02471_0",
      "GlobalTag": "76X_mcRun2_asymptotic_v12"
    }
  ],
  "CMSSWVersion": [
    {
      "TaskName": "EXO-RunIIFall15DR76-03799_0",
      "CMSSWVersion": "CMSSW_7_6_1"
    },
    {
      "TaskName": "EXO-RunIIFall15DR76-03799_1",
      "CMSSWVersion": "CMSSW_7_6_1"
    },
    {
      "TaskName": "EXO-RunIIFall15MiniAODv1-02471_0",
      "CMSSWVersion": "CMSSW_7_6_1"
    }
  ]
}]}

If we need, we can add the task and output dataset map in reqmgr document but we have update all old document as well.
@amaltaro, Alan, is there any other way we can map between task property and output dataset name?

@vkuznet, Valentine what is the exact reqmgr2 query in DAS which caused the problem?

Andrew Levin asked for this dataset
/SingleMu/Run2012D-ZMu-PromptSkim-v1/RAW-RECO
DAS had a call to ReqMgr2 inputdataset
https://cmsweb.cern.ch/reqmgr2/data/request?inputdataset=/SingleMu/Run2012D-ZMu-PromptSkim-v1/RAW-RECO
and ReqMgr2 returns large JSON which DAS struggles to parse in real time on web
interface.
On  0, ticoann notifications@github.com wrote:

@vkuznet, Valentine what is the exact reqmgr2 query in DAS which caused the problem?

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6700#issuecomment-198491905

Thanks Valentin, I thing we have to remove inputdataset call from DAS. DAS it doesn't distinguish whether dataset is fed as inputdataset or outputdataset right?
The same dataset can be either input or output dataset.
And the result it returns are different depending on whether it is input dataset or output dataset
I think it is more meaningful if das only queries output dataset.
https://cmsweb.cern.ch/reqmgr2/data/request?outputdataset=/SingleMu/Run2012D-ZMu-PromptSkim-v1/RAW-RECO
Anyway, we will still restrict the return values as we discussed here but I have to solve the case with Task and Step chain.
Thanks

Seangchan,
this is what I did to solve the problem, I remove inputdataset API call from
DAS and right now DAS only query outputdataset API.
V.
On  0, ticoann notifications@github.com wrote:

Thanks Valentin, I thing we have to remove inputdataset call from DAS. DAS it doesn't distinguish whether dataset is fed as inputdataset or outputdataset right?
The same dataset can be either input or output dataset.
And the result it returns are different depending on whether it is input dataset or output dataset
I think it is more meaningful if das only queries output dataset.
https://cmsweb.cern.ch/reqmgr2/data/request?outputdataset=/SingleMu/Run2012D-ZMu-PromptSkim-v1/RAW-RECO
Anyway, we will still restrict the return values as we discussed here but I have to solve the case with Task and Step chain.
Thanks

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6700#issuecomment-198507999

Yes, I know. I am just saying we shouldn't put it back even if we make this call small and consistent (no arbitrary key). You still want consistent dictionary format for output dataset result, right?

yes
On  0, ticoann notifications@github.com wrote:

Yes, I know. I am just saying we shouldn't put it back even if we make this call small and consistent (no arbitrary key). You still want consistent dictionary format for output dataset result, right?

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6700#issuecomment-198516140

merging this. If need to change the return format will do next iteration.

authors: amaltaro ticoann vkuznet

repo: dmwm/WMCore


text:
[py2py3] migration of DQMUpload.py

Fixes #9926
Status
Ready - right now i added two unittests and have not marked them as integration just to see if jenkins like those. I will add the integration attribute to the unittests as soon as i see jenkins report
Description
I restarted the work on this topic, driven by investigations necessary for #10651. A lot of time passed, and with time we have a better insight of what is going on here:

[x] #9926 is caused by python-future's backport of py3's urllib to py2. This can be avoided by simply continue using urllib2 in py2. See logs at [1]
[x] #10651 fixed a problem that arises when using py3 central services and py2 wmagent. The url argument in urllib2.Request(url) ended up being of type unicode, which mixed the types in the httplib buffer, causing a the same problem seen in #9926
[x] I added a unittest. I am making it fail because dqm.upload() is returning http code 500. Maybe it's just a matter of chaning the filename, but i have not looked for what it should be, yet  See logs for

py2 logs for WMCore_t.WMSpec_t.Steps_t.Executors_t.DQMUpload_t.DQMUpload_t.testA_Upload
py3 logs for WMCore_t.WMSpec_t.Steps_t.Executors_t.DQMUpload_t.DQMUpload_t.testA_Upload
Thanks to @amaltaro for providing the fix! :).
Since this unittest uploads a  144MB root file coming directly from Alan's eos to the DQM every time we run this new unittest, we decided that it is better not to run this often. However it is good to keep this around in case of bugs -> we mark this unittest as integration

The solution was to avoid using in py2 the backport of py3 urllib provided by future, which is broken. So the only "quick" solution is to select at runtime with a try/except clause the proper classes and functions.
An alternative would be refactoring all these machinery and start using request instead, but it may not be worth the effort.
Is it backward compatible (if not, which system it affects?)
YES
External dependencies / deployment changes
DQMUpload.py will continue to depend on python-future

@mapellidario Thanks Dario. I'm going to redeploy vocms0193, apply this patch and run some tests. Once I have any results, will report them back here.

@mapellidario as you expected, no unicode data in there:
2020-09-24 17:02:18,414:INFO:DQMUpload:[9926] header ('Content-length', '49159387') <type 'str'>
2020-09-24 17:02:18,415:INFO:DQMUpload:[9926] header ('Content-type', 'multipart/form-data; boundary=----------=_DQM_FILE_BOUNDARY_=-----------') <type 'str'>
2020-09-24 17:02:18,415:INFO:DQMUpload:[9926] header ('Accept-encoding', 'gzip') <type 'str'>
2020-09-24 17:02:18,415:INFO:DQMUpload:[9926] header ('User-agent', 'WMAgent python/2.7.13') <type 'str'>
2020-09-24 17:02:18,415:INFO:DQMUpload:[9926] Request data type: <type 'str'>
2020-09-24 17:02:18,571:ERROR:DQMUpload:HTTP upload failed! Error:
'ascii' codec can't decode byte 0xec in position 479: ordinal not in range(128)
Traceback (most recent call last):
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Executors/DQMUpload.py", line 157, in httpPost
    (headers, data) = self.upload(uploadURL, args, filename)
  File "/srv/job/WMCore.zip/WMCore/WMSpec/Steps/Executors/DQMUpload.py", line 259, in upload
    result = opener.open(datareq)
  File "/cvmfs/cms.cern.ch/COMP/slc6_amd64_gcc493/external/py2-future/0.18.2/lib/python2.7/site-packages/future/backports/urllib/request.py", line 500, in open
    response = self._open(req, data)
  File "/cvmfs/cms.cern.ch/COMP/slc6_amd64_gcc493/external/py2-future/0.18.2/lib/python2.7/site-packages/future/backports/urllib/request.py", line 518, in _open
    '_open', req)
  File "/cvmfs/cms.cern.ch/COMP/slc6_amd64_gcc493/external/py2-future/0.18.2/lib/python2.7/site-packages/future/backports/urllib/request.py", line 472, in _call_chain
    result = func(*args)
  File "/srv/job/WMCore.zip/WMCore/Services/HTTPS/HTTPSAuthHandler.py", line 58, in https_open
    return self.do_open(self.get_connection, req)
  File "/cvmfs/cms.cern.ch/COMP/slc6_amd64_gcc493/external/py2-future/0.18.2/lib/python2.7/site-packages/future/backports/urllib/request.py", line 1290, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/cvmfs/cms.cern.ch/COMP/slc6_amd64_gcc493/external/python/2.7.13/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/cvmfs/cms.cern.ch/COMP/slc6_amd64_gcc493/external/python/2.7.13/lib/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/cvmfs/cms.cern.ch/COMP/slc6_amd64_gcc493/external/python/2.7.13/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/cvmfs/cms.cern.ch/COMP/slc6_amd64_gcc493/external/python/2.7.13/lib/python2.7/httplib.py", line 880, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xec in position 479: ordinal not in range(128)

I have the suspicion that it might be coming from the marshall method. Will try to play with it tomorrow, but in case you have other suggestions, please do let me know.

I'm not sure why you said no unicode data is here, since the error message is very clear:
'ascii' codec can't decode byte 0xec in position 479: ordinal not in range(128)

It means that there is a special character which is not in ASCII table and it should be treated (decoded/encoded) properly. There are plenty of materials on internet about it.

I am suprised. I hoped that the non-ascii character was inserted by marshall and would have been in datareq.data, so I added          logging.info("[9926] Request data: %s %s", datareq.data) at line 256, but I do not see this logging in your stacktrace. Do you have any idea of why this could have been missed?

However, what puzzles me the most is that I tried to insert some unicode in datareq.data in a script here, but I have not been able to replicate this error. There is something that I am missing, and this can either be a huge mistake on my side or a subtle quirk that I missed.

Another observation, why do you run using this architecture slc6_amd64_gcc493? All cmsweb stack is now using slc7_amd64_gcc630.
And, regarding str vs unicode, may be this will help
https://stackoverflow.com/questions/18649512/unicodedecodeerror-ascii-codec-cant-decode-byte-0xe2-in-position-13-ordinal
https://stackoverflow.com/questions/21129020/how-to-fix-unicodedecodeerror-ascii-codec-cant-decode-byte/21129492

test this please

Hello @amaltaro! The status of this PR is: when we try to upload a file, we do not respect a specific filename convention and we get http code 500 back. This makes me believe that urllib is douing its job and is sending the data properly, though, and makes me believe that we fixed the issue. However, I am not sure if this is a good option, so I am making the unittest fail with a raise Exception so that you can read the logs from jenkins.
I see two ways of proceeding here

We ignore the error code 500 and we get it as a success from our side, making the new unittest mean "we only care about sending the data, we do not really care about respecting all the filename conventions"
We fix the filename and we make sure that the data we send gets actually uploaded. This means that every time we run the unittest, we upload data to the DQM gui, maybe not something we really want though.

Do you have any suggestions?

And I've only seen your question because I ran into issues importing urllib2 in python3, sorry.
Dario, it's not clear to me whether you are concerned only with the unit test, or with this code in "real" production mode. Nonetheless, not respecting the http request status code would be very confusing IMO.
Let me patch my test agent and get a test running while I have a look at this code.

Nono, i am concerned only with always uploading the file in the unittest, i hope that real production code can upload all the files we wish :)

@amaltaro using the name of the file that you provied instead of dqm.root worked indeed! i will push a new commit.
I paste below some logging of the successful request which may be useful for future-us. Notice that I am artificially making it fail just to retrieve the logs (with some debugging information) in case of a successfull request.
py3 log
#1 testA_Upload (WMCore_t.WMSpec_t.Steps_t.Executors_t.DQMUpload_t.DQMUpload_t) ... ERROR

======================================================================
ERROR: testA_Upload (WMCore_t.WMSpec_t.Steps_t.Executors_t.DQMUpload_t.DQMUpload_t)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dmwm/wmcore_unittest/WMCore/test/python/WMCore_t/WMSpec_t/Steps_t/Executors_t/DQMUpload_t.py", line 62, in testA_Upload
    raise Exception
Exception:
-------------------- >> begin captured logging << --------------------
root: INFO: Steps.Executor logging started
root: INFO: Using proxy file: /home/dmwm/certs-mount/x509up_u99307
root: INFO: Using CA certificate path: None
root: INFO: HTTP POST upload arguments:
  ==> checksum: md5:776e66d494a6eeabd52b1d7fb50e9bd4
  ==> size: 150553620

WMCore.Services.HTTPS.HTTPSAuthHandler: INFO: Found 142 default trusted CA certificates.
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO: SSL context manager created with the following settings:
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   check_hostname : True
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   options : Options.OP_ALL|OP_NO_SSLv3|OP_NO_SSLv2|OP_CIPHER_SERVER_PREFERENCE|OP_SINGLE_DH_USE|OP_SINGLE_ECDH_USE|OP_NO_COMPRESSION
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   protocol : _SSLMethod.PROTOCOL_TLS
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   verify_flags : VerifyFlags.VERIFY_X509_TRUSTED_FIRST
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   verify_mode : VerifyMode.CERT_REQUIRED
root: DEBUG: encode value - <class 'str'>, md5:776e66d494a6eeabd52b1d7fb50e9bd4
root: DEBUG: encode payload - <class 'bytes'>, b'md5:776e66d494a6eeabd52b1d7fb50e9bd4'
root: DEBUG: encode value - <class 'int'>, 150553620
root: DEBUG: encode payload - <class 'bytes'>, b'150553620'
root: DEBUG: encode body (without binary file) -<class 'bytes'>, b'------------=_DQM_FILE_BOUNDARY_=-----------\r\nContent-Disposition: form-data; name="checksum"\r\n\r\nmd5:776e66d494a6eeabd52b1d7fb50e9bd4\r\n------------=_DQM_FILE_BOUNDARY_=-----------\r\nContent-Disposition: form-data; name="size"\r\n\r\n150553620\r\n------------=_DQM_FILE_BOUNDARY_=-----------\r\nContent-Disposition: form-data; name="file"; filename="DQM_V0001_R000000001__RelValH125GGgluonfusion_13__CMSSW_8_1_0-RecoFullPU_2017PU_TaskChain_PUMCRecyc_HG1705_Validation_TEST_Alan_v67-v11__DQMIO.root"\r\nContent-Type: application/octet-stream\r\nContent-Length: 150553620\r\n'
WMCore.Services.HTTPS.HTTPSAuthHandler: DEBUG: POST method to https://cmsweb-testbed.cern.ch/dqm/dev//data/put
WMCore.Services.HTTPS.HTTPSAuthHandler: DEBUG:   with the following headers: {'Accept-encoding': 'gzip', 'User-agent': 'WMAgent python/3.8.2', b'Content-type': b'multipart/form-data; boundary=----------=_DQM_FILE_BOUNDARY_=-----------'}
root: DEBUG: headers: <class 'http.client.HTTPMessage'> Date: Fri, 13 Aug 2021 11:03:55 GMT
Server: CherryPy/5.4.0
Set-Cookie: cms-auth=a1db9d8e44eac6a056b1b6d5b0d937561bd90cef6b250f9c42967d03247917ceb85188e2ef081d14;path=/;secure;httponly;expires=Thu, 01-Jan-1970 00:00:01 GMT
Dqm-Status-Detail: Wrote 150553620 bytes
Content-Length: 376
Dqm-Status-Code: 100
Dqm-Status-Message: File saved
Content-Type: text/html;charset=utf-8
CMS-Server-Time: D=6784126 t=1628852635676700
Connection: close

root: DEBUG: data: <class 'bytes'> b'Thanks.\nThe server received your file. However this does not mean that the file was indexed successfully.\nA different server process will now check the filename to determine in which folder to archive it.\nAfter that, another process will attempt to index it. This might take some minutes.\nIn case of doubt, please check the server logs of the receive-daemon and import-daemon.'
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
XML: /home/dmwm/wmcore_unittest/WMCore/nosetests.xml
----------------------------------------------------------------------
Ran 1 test in 9.509s

FAILED (errors=1)
Testing complete, there are now 1 threads

py2 log
#1 testA_Upload (WMCore_t.WMSpec_t.Steps_t.Executors_t.DQMUpload_t.DQMUpload_t) ... ERROR

======================================================================
ERROR: testA_Upload (WMCore_t.WMSpec_t.Steps_t.Executors_t.DQMUpload_t.DQMUpload_t)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dmwm/wmcore_unittest/WMCore/test/python/WMCore_t/WMSpec_t/Steps_t/Executors_t/DQMUpload_t.py", line 62, in testA_Upload
    raise Exception
Exception:
-------------------- >> begin captured logging << --------------------
root: INFO: Steps.Executor logging started
root: INFO: Using proxy file: /home/dmwm/certs-mount/x509up_u99307
root: INFO: Using CA certificate path: None
root: INFO: HTTP POST upload arguments:
  ==> checksum: md5:776e66d494a6eeabd52b1d7fb50e9bd4
  ==> size: 150553620

WMCore.Services.HTTPS.HTTPSAuthHandler: INFO: Found 142 default trusted CA certificates.
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO: SSL context manager created with the following settings:
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   check_hostname : True
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   options : 2197947391
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   protocol : 2
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   verify_flags : 32768
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   verify_mode : 2
root: DEBUG: encode value - <type 'str'>, md5:776e66d494a6eeabd52b1d7fb50e9bd4
root: DEBUG: encode payload - <type 'str'>, md5:776e66d494a6eeabd52b1d7fb50e9bd4
root: DEBUG: encode value - <type 'int'>, 150553620
root: DEBUG: encode payload - <type 'str'>, 150553620
root: DEBUG: encode body (without binary file) -<type 'str'>, ------------=_DQM_FILE_BOUNDARY_=-----------
Content-Disposition: form-data; name="checksum"

md5:776e66d494a6eeabd52b1d7fb50e9bd4
------------=_DQM_FILE_BOUNDARY_=-----------
Content-Disposition: form-data; name="size"

150553620
------------=_DQM_FILE_BOUNDARY_=-----------
Content-Disposition: form-data; name="file"; filename="DQM_V0001_R000000001__RelValH125GGgluonfusion_13__CMSSW_8_1_0-RecoFullPU_2017PU_TaskChain_PUMCRecyc_HG1705_Validation_TEST_Alan_v67-v11__DQMIO.root"
Content-Type: application/octet-stream
Content-Length: 150553620

WMCore.Services.HTTPS.HTTPSAuthHandler: DEBUG: POST method to https://cmsweb-testbed.cern.ch/dqm/dev//data/put
WMCore.Services.HTTPS.HTTPSAuthHandler: DEBUG:   with the following headers: {'Content-length': '150554231', 'Content-type': 'multipart/form-data; boundary=----------=_DQM_FILE_BOUNDARY_=-----------', 'Accept-encoding': 'gzip', 'User-agent': 'WMAgent python/2.7.13'}
root: DEBUG: headers: <type 'instance'> Date: Fri, 13 Aug 2021 11:03:57 GMT
Server: CherryPy/5.4.0
Set-Cookie: cms-auth=c6bbcbb76af633e7876c40bb2a02b99053c8353f76dd7bcc2c9d133f68bb4236ff48ee3796edcf3f;path=/;secure;httponly;expires=Thu, 01-Jan-1970 00:00:01 GMT
Dqm-Status-Detail: Wrote 150553620 bytes
Content-Length: 376
Dqm-Status-Code: 100
Dqm-Status-Message: File saved
Content-Type: text/html;charset=utf-8
CMS-Server-Time: D=6561068 t=1628852637711378
Connection: close

root: DEBUG: data: <type 'str'> Thanks.
The server received your file. However this does not mean that the file was indexed successfully.
A different server process will now check the filename to determine in which folder to archive it.
After that, another process will attempt to index it. This might take some minutes.
In case of doubt, please check the server logs of the receive-daemon and import-daemon.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
XML: /home/dmwm/wmcore_unittest/WMCore/nosetests.xml
----------------------------------------------------------------------
Ran 1 test in 10.175s

FAILED (errors=1)
Testing complete, there are now 1 threads

I am glad to hear it works now! I was hoping to have real test results, but my test workflow was still stuck in acquired...
Regarding the unit test, I think that file is way too big for many daily uploads, Dario. I would suggest to keep that unit test around - because it's going to be extremely useful if we have to test/debug things - but mark them as @integration, such that they do not run in jenkins.

Jenkins is misbehaving and it's triggering a CI run after another. I'm temporarily closing this PR to stop it.
PS.: My tests jobs have not yet completed, they seem to keep running forever...

Thanks to @amaltaro tests, we found a bug in DQMUpload.httpPost() when running in py3, caused by the following line
https://github.com/dmwm/WMCore/blob/8b4dd2ded5f9c49d25c5672e2bb885db57687d3d/src/python/WMCore/WMSpec/Steps/Executors/DQMUpload.py#L141
the bug has been fixed in the last commit, i added a new unittest that covers that line, too.
Achtung: i removed the @attr("integration") flag from the two unittests only temporarily because i want to see those passing in jenkins. i will put back those flags as soon as we have a successfull report.

As above, I made the new unittest test_httppost fail in order to retrieve the logs. Apart from the artificial failure, a good log from the new unittest is:
path lists is ['test/python/WMCore_t/WMSpec_t/Steps_t/Executors_t/DQMUpload_t.py', 'DQMUpload_t.test_httppost']
Out of 1 cases, we will run 1
#1 test_httppost ... ERROR

======================================================================
ERROR: test_httppost
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dmwm/wmcore_unittest/WMCore/test/python/WMCore_t/WMSpec_t/Steps_t/Executors_t/DQMUpload_t.py", line 100, in test_httppost
    raise Exception
Exception:
-------------------- >> begin captured logging << --------------------
root: INFO: Steps.Executor logging started
root: INFO: HTTP Upload is about to start:
 => URL: https://cmsweb-testbed.cern.ch/dqm/dev/
 => Filename: /tmp/DQM_V0001_R000000001__RelValH125GGgluonfusion_13__CMSSW_8_1_0-RecoFullPU_2017PU_TaskChain_PUMCRecyc_HG1705_Validation_TEST_Alan_v67-v11__DQMIO.root

root: INFO: Using proxy file: /home/dmwm/certs-mount/x509up_u99307
root: INFO: Using CA certificate path: None
root: INFO: HTTP POST upload arguments:
  ==> checksum: md5:776e66d494a6eeabd52b1d7fb50e9bd4
  ==> size: 150553620

WMCore.Services.HTTPS.HTTPSAuthHandler: INFO: Found 142 default trusted CA certificates.
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO: SSL context manager created with the following settings:
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   check_hostname : True
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   options : Options.OP_ALL|OP_NO_SSLv3|OP_NO_SSLv2|OP_CIPHER_SERVER_PREFERENCE|OP_SINGLE_DH_USE|OP_SINGLE_ECDH_USE|OP_NO_COMPRESSION
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   protocol : _SSLMethod.PROTOCOL_TLS
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   verify_flags : VerifyFlags.VERIFY_X509_TRUSTED_FIRST
WMCore.Services.HTTPS.HTTPSAuthHandler: INFO:   verify_mode : VerifyMode.CERT_REQUIRED
root: DEBUG: encode value - <class 'str'>, md5:776e66d494a6eeabd52b1d7fb50e9bd4
root: DEBUG: encode payload - <class 'bytes'>, b'md5:776e66d494a6eeabd52b1d7fb50e9bd4'
root: DEBUG: encode value - <class 'int'>, 150553620
root: DEBUG: encode payload - <class 'bytes'>, b'150553620'
root: DEBUG: encode body (without binary file) -<class 'bytes'>, b'------------=_DQM_FILE_BOUNDARY_=-----------\r\nContent-Disposition: form-data; name="checksum"\r\n\r\nmd5:776e66d494a6eeabd52b1d7fb50e9bd4\r\n------------=_DQM_FILE_BOUNDARY_=-----------\r\nContent-Disposition: form-data; name="size"\r\n\r\n150553620\r\n------------=_DQM_FILE_BOUNDARY_=-----------\r\nContent-Disposition: form-data; name="file"; filename="DQM_V0001_R000000001__RelValH125GGgluonfusion_13__CMSSW_8_1_0-RecoFullPU_2017PU_TaskChain_PUMCRecyc_HG1705_Validation_TEST_Alan_v67-v11__DQMIO.root"\r\nContent-Type: application/octet-stream\r\nContent-Length: 150553620\r\n'
WMCore.Services.HTTPS.HTTPSAuthHandler: DEBUG: POST method to https://cmsweb-testbed.cern.ch/dqm/dev//data/put
WMCore.Services.HTTPS.HTTPSAuthHandler: DEBUG:   with the following headers: {'Accept-encoding': 'gzip', 'User-agent': 'WMAgent python/3.8.2', b'Content-type': b'multipart/form-data; boundary=----------=_DQM_FILE_BOUNDARY_=-----------'}
root: INFO: HTTP upload finished succesfully with response:
  Status code: 100
  Message: File saved
  Detail: Wrote 150553620 bytes
  Data: b'Thanks.\nThe server received your file. However this does not mean that the file was indexed successfully.\nA different server process will now check the filename to determine in which folder to archive it.\nAfter that, another process will attempt to index it. This might take some minutes.\nIn case of doubt, please check the server logs of the receive-daemon and import-daemon.'

--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
XML: /home/dmwm/wmcore_unittest/WMCore/nosetests.xml
----------------------------------------------------------------------
Ran 1 test in 8.386s

FAILED (errors=1)
Testing complete, there are now 1 threads

Thanks for the prompt fix, Dario. Your last commit indeed fixed the issue and harvesting jobs are now succeeding.
Feel free to squash those commits properly, I will make a new release with these changes in by tomorrow. Thanks again!

After jenkins reported that the two new unittests are passing both in py2 and py3, i added again the @attr("integration")  flag on those

Thanks for making these changes, Dario.
This has been tested with Py3 WMAgent and it works fine. We will find out in the next replay or in the coming days whether this works for Py2 as well...

authors: amaltaro mapellidario vkuznet


text:
???

in maps.w.d.b

Number of crashes: 1
Impacted devices: 1

There's a lot more information about this crash on crashlytics.com:
https://fabric.io/dmytro-karataievs-projects/android/apps/com.adkdevelopment.earthquakesurvival/issues/5816876e0aeb16625b4b33ad

Check this: http://stackoverflow.com/questions/36147370/released-unknown-imagedata-reference-on-map-clear

repo: dmytroKarataiev/EarthquakeSurvival


text:
멀티 스텝 온보딩 폼 구현

PR 타입 ( 하나 이상의 PR 타입을 선택해주세요 )

[x] 기능 추가
[ ] 버그 수정
[ ] 의존성, 환경 변수, 빌드 관련 코드 업데이트
[ ] 기타 사소한 수정

개요
저희 온보딩 UI에 따라서 일단 API가 안나온 상황에서 더미 데이터로 스텝 로직을 구현해봤습니다.
스탭 버튼 관련해서, 사실 현재 선택된 최소 갯수를 맞아야만 다음 스텝으로 갈 수 있어야하다보니까,
useForm에 mode를 onChange Validation으로 바꿔서..  컴포넌트 리렌더링은 어쩔 수 없었습니다..
혹시 의문이 가는점이 있다면.. 서슴없이 말씀해주세ㅇ..
코드 리뷰시 참고 사항
preview에서 /onboarding으로 가시면 보실 수 있으실것 같습니다.
https://fiesta-git-feature-onboarding-page-froggy1014s-projects.vercel.app/onboarding

@seokzin
@punchdrunkard
@saseungmin
리뷰 감사드립니다 ㅠ ㅠ
말씀해주신것들은 반영했구요.
클린코드적으로 다시 한번 생각해보게되는... 일단 머지하겠읍니다 🙇♂

license: MIT


text:
Samba not working

Hi
I have been trying to access local samba shares that are definitively accessible to other local Ubuntu machines. I run KDE/Kubuntu in all of them, and they show up just fine in Dolphin. In crouton, I installed everything that seemed to make sense (samba, smbclient, winbind). No charm. Any thoughts, anybody?
Cheers
Leo

Wat a minute. Are y'all seriously trying to say that Samba does not work on Crouton????!!! No wonder I've been strugling for days on end.

@MeneerJansen You can browse smb shares in a chroot see the first chapter https://github.com/dnschneid/crouton/wiki/How-to-mount-network-shares-on-Chromebook-(sshfs,-cifs,-nfs-etc)#browsing-smb-shares-in-a-chroot

You Can Now Access Samba Shares on a Chromebook (on the Chrome OS side)

@DennisLfromGA Yep that is correct, but it doesn't work well with my synology NAS.

@divx118,
Thanx for the info., I don't have anything to test it with anymore (no windoze - yay!) so I haven't tried it.
-DennisL

Thank you all for the links. I'll look into it.

The ChromeOS addon for SMB shares looks nice and easy.
Is there any way to share a folder via SMB on my Chroot or from ChromeOS? So that Windows users can assess my files?

From the Crouton Wiki:

Note: This should work on a standard chroot with no additional smb packages installed.
Mounting smb shares in a chroot is easy. It can be done with browsing with for example filemanager "pcmanfm". On LXDE this is the default filemanager on xfce we have thunar. Thunar gave me an error: Failed to open "/ on ". Still have to do research on what the problem is. Unity/gnome/kde needs to be tested.

This cannot be true. Tried it: wouldn't know what is so special about pcmanfm and it does not connect to the Windows network.
Furthermore: on my ChromeOS the afore mentioned File System for Windows from the Chrome Store indeed allows me to connect to and browse the Samba shares on my Linux machine, but I cannot get it to connect to a Windows share. And if I could: the share would  not show up in my beloved chroot anyway. I can connect to the SMB share on my Win 10 machine from my other Linux computer though.
So I tried another method from the wiki. That mounted a Samba share to ChromeOS in the 'Downloads' folder. ChromeOS cannot "see" this mount (as said on that site) but neither can my chroot. I can only browse the share from the command line in ChromeOS. And when I try to edit a text file in vi from there the shell crashes. Maybe I have to update ChromeOS, but I'm afraid that breaks other things.
Might try to use the experimental SSH support from Microsoft to try to let Window speak SSH. See:

@divx118 Thanks for your help.
About the IP tables: opening the ports via the method you posted gives no errors in ChromeOS or in my croot but saving it does:
$ sudo iptables-save > /etc/sysconfig/iptables
iptables-save > /etc/sysconfig/iptables

About mounting smb shares so thy show up in ChromeOS AND the chroot (link), I did as they say in that Wiki:

If you don't want to mount on top of an external media device, but still want to be the share visible in the chroot. You can do the following by creating a mountpoint in ~/Downloads and execute the following commands in a crosh shell.

$ sudo mount --bind ~/Downloads ~/Downloads
$ sudo mount --make-shared ~/Downloads
Didn't help either. Gonna try to mount to removable SD crd now... I'll be back.
P.S. Is there anybody who actually can access SMB shares on a Windows 10 machine from ChromeOS or Crouton? Please let me know.

@divx118  Only just a minute ago I saw that you are the author of the Wiki I am following. Mounting my workinf Linux Samba share does not work anymore, while it sorta did yesterday. Could you help me out? I think I messes u w/ the command:
 sudo mount --bind ~/Downloads ~/Downloads/sambamounts
I've been using Linux for quite some time now but I don't understand waht that does exactly. From the man page I understand that is rebinds an old dir to a new one. But I don't know what my "olddir" is anymore.
Anyway after $ sudo mountcifs start I get the following error:
mount: wrong fs type, bad option, bad superblock on /192.168.0.100/administr,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
Mount failed for /192.168.0.100/administr on /home/chronos/user/Downloads/sambamounts

This is driving me completely crazy, ha ha.

@MeneerJansen Just opening the ports in a crosh shell should be enough. You can't save them, because the chromeos rootfs is by default readonly. So you need to open them after each reboot. This method worked for me on a HP chromebook 14. The method of making a bind mount of Downloads I never tried. I always used an external media.
For the error it could be that the cifs modules are a bit outdated. Can you give me the output of uname -a So I build some new modules.

I might just be using the old kernel you used to build said module:
Linux localhost 3.8.11 #1 SMP Tue Jul 7 19:37:06 PDT 2015 x86_64 Intel(R) Celeron(R) 2955U @ 1.40GHz GenuineIntel GNU/Linux

I'm using a Toshiba Cromebook 11 (CB30-102).

@MeneerJansen Yes correct, modules are now updated. https://github.com/divx118/cifs/commit/6c61c2160289379a528c32ff22390db4eed716df
Just redo the steps below in a crosh shell to get the updated modules.
$ cd /usr/local
$ sudo wget "https://raw.github.com/divx118/cifs/master/mountcifs.tar.gz"
$ sudo tar xvf mountcifs.tar.gz

Modules are untested. If you have problems, take a look at dmesg and post the output when running the mountcifs script.

Just tested, they work for me with no problems. I have also the 3.8.11 kernel.

@divx118 So it works on your end, that means that I'm doing soething wrong. Probably some configuration error by me. This is the error:
$ sudo mountcifs start
Module cifs loaded
Module md4 loaded
mount: wrong fs type, bad option, bad superblock on //192.168.0.100/administr,
       missing codepage or helper program, or other error
       (for several filesystems (e.g. nfs, cifs) you might
       need a /sbin/mount.<type> helper program)

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
Mount failed for //192.168.0.100/administr on /media/removable/Michel_32GB/Chromebook/home/michel/sambamounts/

Dmesg, about a thousand times this::
[ 1724.461421] wlan0: cancelling probereq poll due to a received beacon
[ 1727.438500] wlan0: detected beacon loss from AP - sending probe request
[ 1727.532874] wlan0: cancelling probereq poll due to a received beacon
[ 1729.434258] wlan0: detected beacon loss from AP - sending probe request
[ 1729.478380] wlan0: cancelling probereq poll due to a received beacon
[ 1733.437510] wlan0: detected beacon loss from AP - sending probe request
[ 1733.471346] wlan0: cancelling probereq poll due to a received beacon
[ 1735.433252] wlan0: detected beacon loss from AP - sending probe request
[ 1735.519162] wlan0: cancelling probereq poll due to a received beacon
[ 1738.436753] wlan0: detected beacon loss from AP - sending probe request
[ 1738.488317] wlan0: cancelling probereq poll due to a received beacon
[ 1740.432423] wlan0: detected beacon loss from AP - sending probe request
[ 1740.536058] wlan0: cancelling probereq poll due to a received beacon
[ 1742.428127] wlan0: detected beacon loss from AP - sending probe request
[ 1742.481314] wlan0: cancelling probereq poll due to a received beacon
[ 1745.435604] wlan0: detected beacon loss from AP - sending probe request
[ 1745.450563] wlan0: cancelling probereq poll due to a received beacon
[ 1747.427266] wlan0: detected beacon loss from AP - sending probe request
[ 1747.498158] wlan0: cancelling probereq poll due to a received beacon
[ 1749.426883] wlan0: detected beacon loss from AP - sending probe request
[ 1749.443584] wlan0: cancelling probereq poll due to a received beacon
[ 1751.426629] wlan0: detected beacon loss from AP - sending probe request
[ 1751.494654] wlan0: cancelling probereq poll due to a received beacon
[ 1753.430215] wlan0: detected beacon loss from AP - sending probe request
[ 1753.444827] wlan0: cancelling probereq poll due to a received beacon
[ 1755.429882] wlan0: detected beacon loss from AP - sending probe request
[ 1755.485310] wlan0: cancelling probereq poll due to a received beacon
[ 1758.429368] wlan0: detected beacon loss from AP - sending probe request
[ 1758.453587] wlan0: cancelling probereq poll due to a received beacon
[ 1760.429076] wlan0: detected beacon loss from AP - sending probe request
[ 1760.501313] wlan0: cancelling probereq poll due to a received beacon

And:
[ 1611.208855] Chromium OS LSM: init_module locking-ignored module="/usr/local/modules/x86_64_3.8.11/cifs.ko" pid=1034 cmdline="/sbin/insmod /usr/local/modules/x86_64_3.8.11/cifs.ko"
[ 1611.228314] Chromium OS LSM: init_module locking-ignored module="/usr/local/modules/x86_64_3.8.11/md4.ko" pid=1041 cmdline="/sbin/insmod /usr/local/modules/x86_64_3.8.11/md4.ko"
[ 1611.270581] CIFS: Unknown mount option "0"

I think you messed up the etc/fstab looking at CIFS: Unknown mount option "0"

@divx118 You're right! Added "cifs" to /usr/local/etc/fstab and I can browse the Samba share on the command line shell of ChromeOS. The directory structure of said share shows up in the "Files" app of ChromeOS. I cannot readany files though. And I cannot go into a directory of my share.
In the filemanager of my chroot it takes a long time for the directory listing to show up. I can "go into" a subdirectory of my share on my chroot, but its all terribly slow. Adding a word of text in a test.txt file still hangs at this moment, but it finish successfully in about 15 minutes or so.
Anyway, it works!
Thanks very much for your help! Will try a Win 10 SMB share now. :)

Update: ha ha. Every crosh I opens hangs now. The file explorer and the text editor in my chroot has the be "xkill-ed". Gonna reboot now. @divx118  By the way, I do not have to open a port in ChromeOS every time I reboot to use your method, or do I?

@MeneerJansen No no ports need to be opened. I think you are exceeding the depth of the directory structure. Try mounting it directly in a folder in the root of your SDcard/USBdisk. I also have problems if my path depth goes beyond 8.

@divx118 Mounted it to a dir on the root of the SD card. Still very slow...

When I try to mount the Windows 10 share I get:
mount: mount //192.168.0.110/Michel on /media/removable/Michel_32GB/sambamounts failed: Connection refused

Might be because of an SMB3 issue. Win 10 uses the SMB version 3.1.1 protocol, new versions of Samba support that (SMB is the protocol from Microsoft, Samba is the software/protocol in *NIX to access it).
Giving up for now. I can connect just fine w/ Linux shares. To connect from Crouton to Windows I think I'll just run ye' ol' insecure FTP server from Win 10.
Thanks everybody for the help! :-)

authors: DennisLfromGA MeneerJansen divx118 leomilano

repo: dnschneid/crouton


text:
xiwi Frame Buffer is being changed somehow the video is horrible in most every chroot tried

Something is not correct, the video look like the update rate are super slow and the frames overlap
It is likely ChromeOS has change the task privs and or time priority for the video socket
I have no idea i did not read the code or attempt to change it

dont use xiwi

you cannot use it , it does not work  the video is not good at all

I'm assuming this is the same as #4822; please reopen if it's a separate issue

authors: TomTravis dnschneid elkawai621


text:
🛑 Dev.to is down

In 6ce2969, Dev.to (https://dev.to/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Dev.to is back up in c3563b9 after 7 minutes.


text:
fix(qdrant): fix non-class type fields #1748

Issue Resolved: Issue with Defining Index of Type QdrantDocumentIndex for Documents with Non-Class Type Fields
Description:
This pull request addresses the issue where defining an index of type QdrantDocumentIndex for documents containing fields with non-class types resulted in a TypeError. The issue has been resolved by replacing the usage of issubclass with safe_issubclass

Thanks a lot for the contribution!

authors: JoanFM TerboucheHacene

repo: docarray/docarray


text:
Older version returned instead of the latest one 😒

Analyzing scc_ccd03b001421_210922_1732, vercheck shows an older version of dbus1 than they have installed. Also looks like its displaying the shipping version instead of updates channel:
dbus-1,1.12.2-8.6.1,1.12.2-8.3.1,Basesystem Module 15 SP1 x86_64
I assume the versions reported by repositories that match the "base_regex" have more priority over the other repositories. However, it seems I forgot to check if the version is actually higher than the original version and discard it if otherwise.

I believe I got this now. I changed the regex from:
base_regex = r"Basesystem.*"
to:
base_regex = r"(SUSE Linux Enterprise.*|Basesystem.*)"
Since the regexes in this case are evaluated in order, it should prioritize updates found in the repos beginning with "SUSE Linux Enterprise", which means the LTSS ones, before deciding on a "Basesystem" repo (if available).
I also implemented a simple version comparison function, using the LooseVersion class. So, if the results contain:
a) a package that is either in the LTSS or Basesystem repos and
b) is also higher version than the package installed on the system
then we consider the result valid as an upgrade.

With this change, the dbus-1 package is now considered "up-to-date", as it matches the version 1.12-8.6.1, which is part of the 15 SP1 LTSS repository.
Also note that the "different versions" report shows that other LTSS updates were correctly picked up:
timezone,2021a-3.44.1,2021a-3.47.1,SUSE Linux Enterprise Server LTSS 15 SP1 x86_64
An interesting side-effect can be seen in the "different versions" report:
opensm,5.5.0.MLNX20190923.1c78385-0.1.47100,3.3.21-6.29,Server Applications Module 15 SP1 x86_64
opensm-devel,5.5.0.MLNX20190923.1c78385-0.1.47100,3.3.21-6.29,Server Applications Module 15 SP1 x86_64

which are also referenced in the "unsupported" report:
opensm,(none),5.5.0.MLNX20190923.1c78385-0.1.47100
opensm-devel,(none),5.5.0.MLNX20190923.1c78385-0.1.47100

In this case, these two packages are from an unsupported vendor. BUT, there are official versions are present in our repositories, and the supported versions are reported in the "different versions" report. This was not intended, but should be useful information to have.

repo: doccaz/scc-tools


text:
docker-entrypoint-initdb.d

How do I create a database in the docker-entrypoint-initdb.d directory?
Need to create an image where it already comes with some database created.

Hello,
You just have to put a file with .sql extension that contains:
CREATE DATABASE "mydb";
if you want to also insert data be sure to add a
\c "mydb"  line before the inserts because the file is run as postgres role in the postgres database by default.

I'm also interested in having a lightweight container that instantly spins up with some pre-built data. Just doing COPY db/ /docker-entrypoint-initdb.d/ works, but feels suboptimal, because the data is replicated twice and the restoring the dump takes time.
Was trying 1000's variations of the following, but with no luck so far:
FROM postgres:9.5-alpine AS data-donor
COPY etc/db/ /docker-entrypoint-initdb.d/
RUN docker-entrypoint.sh
ENTRYPOINT ["/bin/sh"]

FROM postgres:9.5-alpine
COPY --from=data-donor /var/lib/postgresql/data /var/lib/postgresql/data/

According to the logs of the donor intermediate container, the data does load, but when I RUN ls -la /var/lib/postgresql/data in any of the containers, the folder appears empty. I suspect this is because it belongs to user postgres, but I can't find a workaround.
Having an example on https://hub.docker.com/_/postgres/ would save people hundreds of hours! I guess the situations when the data is bound with the lightweight autonomous containers become more often as people move their microservices into kubernetes clusters and docker swarms.

The "/var/lib/postgresql/data" directory is a volume, so you cannot
pre-populate it.  You can set PGDATA to override the location to somewhere
that isn't a volume, or you could build your own image that does not
include the VOLUME declaration.

Awesome, thanks for the hint @tianon!
This worked:
FROM postgres:9.5-alpine AS data-donor
RUN apk add --no-cache unzip
COPY etc/db/ /docker-entrypoint-initdb.d/
ENV PGDATA=/pgdata
RUN docker-entrypoint.sh --help

FROM postgres:9.5-alpine
ENV PGDATA=/pgdata
COPY --from=data-donor /pgdata /pgdata

I'm adding --help to RUN docker-entrypoint.sh --help in the first image just to trick the script – it otherwise does not start at all or launches a foreground process, which never exits.
Problem solved! Your hint about the VOLUME was the key!

Thanks, use "COPY db/ /docker-entrypoint-initdb.d/" work.

@hunsche that's also an option, just bear in mind that your DB will be restored from a dump on each container start. Doing the build in two stages will give you faster launch times.

authors: hunsche kachkaev marema31 tianon

repo: docker-library/postgres


text:
BroadLeaf Ecomm instead of Helloworld

Hi Arun,
Can we use Broadleaf  Ecomm instead of a mvn arch type to generate a helloworld application...?
Thanks ,
Sriram Mageswaran.

Any reason why?

This does not seem relevant for Java developers.

authors: arun-gupta srirammageswaran thaJeztah

repo: docker/community


text:
remove docs files, add README w/URL to new docs repo

@dmp42 @stevvooe @johndmulhausen
Removed docs from Registry repo, added README to point to https://github.com/docker/docker.github.io/tree/master/registry
Let me know if you think we need a vnext-registry branch over there, if so I will create one.
Signed-off-by: Victoria Bialas victoria.bialas@docker.com

this looks like a duplicate of #1985 (which has a couple of outstanding comments)

Current coverage is 50.88% (diff: 100%)

Merging #1992 into master will decrease coverage by 10.12%

@@             master      #1992   diff @@
==========================================
  Files           125        125
  Lines         11132      11132
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits           6792       5665   -1127
- Misses         3452       4720   +1268
+ Partials        888        747    -141

Powered by Codecov. Last update 1921dde...8fadf50

@RichardScothern , @johndmulhausen closing this one as it is a duplicate! :-)

@codecov-io please see #1985

repo: docker/distribution


text:
Create hello.txt

Please do not send pull requests to this docker/docker-ce repository.
We do, however, take contributions gladly.
See https://github.com/docker/docker-ce/blob/master/CONTRIBUTING.md
Thanks!

OK

repo: docker/docker-ce


text:
Malformed answer accepted in step 3 (hello world)

Steps to reproduce:

Go to the step n. 3 of the tutorial
type in the following:

you@tutorial:~$ docker run learn/tutorial 'echo "hello world"
Unable to locate 'echo

(note the missing closing single quote)

while the emulator returns correctly an error the popup appears anyway saying that the command was correctly executed

I just hit this too:
you@tutorial:~$ docker run learn/tutorial 'echo hello world'
2013/07/08 23:13:30 Unable to locate 'echo
you@tutorial:~$
Great! Hellooooo World! You have just started a container and executed a program inside of it, when the program stopped, so did the container.

However, the following works fine:  docker run learn/tutorial echo hello world

I can confirm I have the same problem

+1

Same problem

docker run learn/tutorial echo hello world # w/o quote it works

authors: CristianCantoro J7mbo deltreey fedeoasi kuboj rvadim

repo: docker/docker-tutorial


text:
404 at

404 found here
Set build-time variables (–build-arg)
You can use ENV instructions in a Dockerfile to define variable values. These values persist in the built image. However, often persistence is not what you want. Users want to specify variables differently depending on which host they build an image on.

A good example is http_proxy or source versions for pulling intermediate files. The ARG instruction lets Dockerfile authors define values that users can set at build-time using the --build-arg flag:

$ docker build --build-arg HTTP_PROXY=http://10.20.30.2:1234 .
This flag allows you to pass the build-time variables that are accessed like regular environment variables in the RUN instruction of the Dockerfile. Also, these values don’t persist in the intermediate or final images like ENV values do.

Using this flag will not alter the output you see when the ARG lines from the Dockerfile are echoed during the build process.

For detailed information on using ARG and ENV instructions, see the Dockerfile reference(https://docs.docker.com/v17.09/edge/engine/reference/builder/). // <=== HERE IT IS

This issue appears to be fixed. Please reopen the issue if the problem persists. Thank you.


text:
[18.09] Ruby Ver Bump (#10040)

Proposed changes
Cherry-pick #10040
Unreleased project version (optional)

Related issues (optional)

Deploy preview for docsdocker ready!
Built with commit 3dbf1ca910d95de366cf1fb5dec71af3c0b57fdc
https://deploy-preview-10041--docsdocker.netlify.com

authors: GordonTheTurtle ollypom


text:
CLOUD-2789 Update API docs to cover namespaced commands

Describe the proposed changes
Adds information about using the docker-cloud api with namespaces.
Please take a look
@fermayo and @amegianeg

@sanscontext have you modified too the examples? I cannot find the code. For example:
{
  "deployed_datetime": "Mon, 13 Oct 2014 11:01:43 +0000",
  "destroyed_datetime": null,
  "nickname": "deployment stack",
  "name": "dockercloud-app",
  "resource_uri": "/api/app/v1/stack/7fe7ec85-58be-4904-81da-de2219098d7c/",
  "services": [
    "/api/app/v1/service/09cbcf8d-a727-40d9-b420-c8e18b7fa55b/"
  ],
  "state": "Running",
  "synchronized": true,
  "uuid": "09cbcf8d-a727-40d9-b420-c8e18b7fa55b"
}

Any resource_uri URL returned is namespaced, so in this example resource_uri and the elements of services would be namespaced too

I've updated the JSON examples, please take another look. :)

With those fixes, @fermayo I think we're g2g here?

authors: amegianeg sanscontext


text:
jenkinsfile fixes

Changes

fix docker push command
add post stage that notifies slack channel on unsuccessful builds

Deploy preview for docsdocker ready!
Built with commit ddfc403da2f91c99d53dcf092b82347499b366f9
https://deploy-preview-9399--docsdocker.netlify.com


text:
Add a known issue on WSL 2

Proposed changes
Adding a known issue on WSL 2 for Windows Insider Preview Slow Ring users who'd like to run WSL 2 on Docker Desktop 2.1.6.0
Related issues (optional)
https://github.com/docker/for-win/issues/5168

Deploy preview for docsdocker ready!
Built with commit 4c1ee6c870c52ac157af3c3079a737b9075cb1c8
https://deploy-preview-9906--docsdocker.netlify.com

authors: GordonTheTurtle usha-mandya


text:
.NetworkSettings.Networks differs depending on how a container is connected

To reproduce:

create a bridge network
create a container1
use docker connect to connect the container to the network
create another container but this type use docker run --net to connect it.

Result: An inspect of the first container returns [bridge isolated_nw] while inspecting the second container returns just  [isolated_nw]
ubuntu@ip-172-31-36-118:~$ docker inspect --format='{{.NetworkSettings.Networks}}' container3
[isolated_nw]
ubuntu@ip-172-31-36-118:~$ docker inspect --format='{{.NetworkSettings.Networks}}' container2
[bridge isolated_nw]
ubuntu@ip-172-31-36-118:~$ docker network inspect isolated_nw
[
    {
        "name": "isolated_nw",
        "id": "cfcc665b946cda94e2b6099a8a84d6f68ed89a8e0f3a746c4f5a259b0ab9cc47",
        "scope": "local",
        "driver": "bridge",
        "ipam": {
            "driver": "default",
            "config": [
                {}
            ]
        },
        "containers": {
            "6b78fea23cefac4ac977346bbd287d69696aaa57db9c226f77068c2e5c5f24be": {
                "endpoint": "cbb2f4c44633088042312dd18523922962238d9c2c27ac3540618655d278aa48",
                "mac_address": "02:42:ac:13:00:03",
                "ipv4_address": "172.19.0.3/16",
                "ipv6_address": ""
            },
            "8845961414ca452dbbff849f91203e5cc90f6b7d8cb1da58935bf87d72f59c29": {
                "endpoint": "ec7c6c247296e093e270c3d2b1fd4467a396996637e5a9ea2773e4545a33a87d",
                "mac_address": "02:42:ac:13:00:02",
                "ipv4_address": "172.19.0.2/16",
                "ipv6_address": ""
            }
        },
        "options": {}
    }
]

Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

Ohhh, I think I see what is going on.  There were two connected networks bridge because it was connected automatically at run and later attached.  (See this is why the name bridge is confusing for both the type and name.)  Umm, I'll explain it.

authors: GordonTheTurtle moxiegirl

repo: docker/docker


text:
let DOCKER_TLS_VERIFY=0 disable TLS

It's somewhat surprising that we cannot explicitly disable TLS connectivity. The docs state that the existence of DOCKER_TLS_VERIFY already enables TLS. It's actual value is ignored.
Would it be possible to change that behaviour to handle falsy values (0, false, no) like expected? If so, I'd be happy to provide a PR.
related: https://github.com/docker/docker/issues/14129

My own $0.02:
While I think empty is an acceptable method of turning TLS off, I don't think "off" should disable TLS, just the verification of the cert.
I think DOCKER_TLS_VERIFY should behave similar to the TLS verification settings of other software. For example, OpenLDAP's TLS verification settings:

TLSVerifyClient { never | allow | try | demand }
This directive specifies what checks to perform on client certificates in an incoming TLS session, if any. This option is set to never by default, in which case the server never asks the client for a certificate. With a setting of allow the server will ask for a client certificate; if none is provided the session proceeds normally. If a certificate is provided but the server is unable to verify it, the certificate is ignored and the session proceeds normally, as if no certificate had been provided. With a setting of try the certificate is requested, and if none is provided, the session proceeds normally. If a certificate is provided and it cannot be verified, the session is immediately terminated. With a setting of demand the certificate is requested and a valid certificate must be provided, otherwise the session is immediately terminated.

(Principle of least astonishment)
So I think it should either remain as is (empty turns TLS off), or we get a new variable, such as $DOCKER_TLS.

@phemmer Where do you read that an empty setting leads to actually verify anything? For me it still reads as DOCKER_TLS_VERIFY=0 is similar to TLSVerifyClient=never, with both settings having exactly those values as defaults when not set.
I'm with you that it's better to distinguish between TLS-verification and TLS-toggling, though. Yet, disabling verification leads effectively to a non-secured connection, doesn't it?

So much about least astonishment - I needed to edit my last comment. I hope I got it right now.

Where do you read that an empty setting leads to actually verify anything?

It doesn't verify anything. It disables the feature.
https://docs.docker.com/engine/reference/commandline/daemon/

Setting the DOCKER_TLS_VERIFY environment variable to any value other than the empty string is equivalent to setting the --tlsverify flag.

Alright, the surprise lies in the enabled verification when the setting is "0". "Empty" is disabling.
So this is wrong on my side:

It's somewhat surprising that we cannot explicitly disable TLS connectivity.

But again, it's at least surprising to me that it's no actual toggle and only works on presence. Compared to OpenLDAP I only miss the possibility to set a value more explicit - or "readable" - instead of relying on existence.
I'm going to close this issue now, because:

most people seem to be used to it
we have the chance to disable the feature
I'm going to think about it and maybe come up with a clearer approach

Thanks for the discussion and correcting my wrong impression!

Current behaviour is extremely annoying and is the show stopper in several environments.
There is NO EXCUSE for such dumb error.

Even if keeping DOCKER_TLS_VERIFY=0 meaning verify, the following should disable it, but doesn't:
DOCKER_TLS_VERIFY=

You actually have to use unset or similar which makes it immediately much more difficult to have a semi portable environment file.

I was surprised to see that explicitly setting
DOCKER_TLS_VERIFY=0

still performs the tls checking.  The meaning of an explicit 0 value should be no surprise.

authors: 40a exjimsk gesellix hickeng phemmer


text:
snmpd service not running on Ubuntu when installed using docker

Hi,
I am trying to install snmpd service in Ubuntu OS using docker. This is my dockerfile:
From ubuntu
RUN apt-get -y update && apt-get install -y snmpd
CMD service snmpd restart
CMD service snmpd status

When I am building the docker file, it is built successfully but when I am running the image, the container created is showing that the service is not running.
However, when I tried restarting it manually, it's working fine.
Below is the output for the file:
tcskvis@01HW735292 MINGW64 ~/ubuntu
$ docker build -t ubuntu_img .
Sending build context to Docker daemon 2.048 kB
Step 1/4 : FROM ubuntu
 ---> 0ef2e08ed3fa
Step 2/4 : RUN apt-get -y update && apt-get install -y snmpd
 ---> Using cache
 ---> 5f33942a2f8c
Step 3/4 : CMD service snmpd restart
 ---> Using cache
 ---> 6f3cf32978f8
Step 4/4 : CMD service snmpd status
 ---> Using cache
 ---> 56936e6a81a0
Successfully built 56936e6a81a0
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x'
permissions. It is recommended to double check and reset permissions for sensitive files and directories.

tcskvis@01HW735292 MINGW64 ~/ubuntu
$ docker run -it ubuntu_img /bin/bash
root@e36726bff9a6:/# service snmpd status
 * snmpd is not running
root@e36726bff9a6:/# service snmpd restart
 * Restarting SNMP services:                                                                                snmpd[30]:
 Created directory: /var/lib/snmp/mib_indexes

snmpd[30]: /etc/snmp/snmpd.conf: line 145: Warning: Unknown token: defaultMonitors.

snmpd[30]: /etc/snmp/snmpd.conf: line 147: Warning: Unknown token: linkUpDownNotifications.

snmpd[30]: Turning on AgentX master support.

snmpd[30]: Created directory: /var/agentx

root@e36726bff9a6:/# service snmpd status
 * snmpd is running
root@e36726bff9a6:/#

Am I doing something wrong here? Can someone please help me out?
Thanks in advance...!!!

containers are not VM's; a container is a "sandbox" to run a single main process. That process is defined by the CMD, and is executed when the container is started.
In your case, the Dockerfile has multiple CMD lines, and the latter will overwrite the former, so when starting the container, service snmpd status is executed, and the container stops when that command completes.
If you want to run multiple services in a single container (think if this is really what you want, please read this blog https://blog.docker.com/2016/03/containers-are-not-vms/), you need to run an init / supervisor system in your container that acts as the main process, and starts sub-processes (e.g. https://docs.docker.com/engine/admin/using_supervisord/), if you want to run systemd as init system inside the container, you need a lot more configuration, because systemd requires additional privileges (which is possible, but make your container less secure)
Please keep in mind that the GitHub issue tracker is not intended as a general support forum,
but for reporting bugs and feature requests. For other type of questions, consider using one of;

https://forums.docker.com
the Docker community Slack channel
StackOverflow

I'm going to close this issue because this is not a bug, but feel free to continue the conversation 👍

Hi @thaJeztah
But when I remove the last line (i.e. service snmpd status) and keep service snmpd restart as my last command, from what you said, service should be restarted and be in running state. But even in that case, snmpd is showing not running.
this is the output I get when running with service snmpd restart command:
tcskvis@01HW735292 MINGW64 ~/ubuntu
$ cat Dockerfile
From ubuntu
RUN apt-get -y update && apt-get install -y snmpd
CMD service snmpd restart
#CMD service snmpd status

tcskvis@01HW735292 MINGW64 ~/ubuntu
$ docker build -t ubuntu_img .
Sending build context to Docker daemon 2.048 kB
Step 1/3 : FROM ubuntu
 ---> 0ef2e08ed3fa
Step 2/3 : RUN apt-get -y update && apt-get install -y snmpd
 ---> Using cache
 ---> 5f33942a2f8c
Step 3/3 : CMD service snmpd restart
 ---> Using cache
 ---> 6f3cf32978f8
Successfully built 6f3cf32978f8
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x'
permissions. It is recommended to double check and reset permissions for sensitive files and directories.

tcskvis@01HW735292 MINGW64 ~/ubuntu
$ docker run -it ubuntu_img /bin/bash
root@284f1e2093e6:/# service snmpd status
 * snmpd is not running
root@284f1e2093e6:/#

authors: HarshaYeluru thaJeztah


text:
Fix issue #15939: Hyperlink to MongoDB Dockerfile not found.

Replaced with link to file in Github.
Signed-off-by: Charles Chan charleswhchan@users.noreply.github.com

Sorry about the EOL changes.

LGTM - @moxiegirl @thaJeztah

LGTM, thanks!

authors: SvenDowideit charleswhchan thaJeztah


text:
Update ubuntulinux.md

Remove '*' for purge

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "patch-1" git@github.com:lrkwz/docker.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Ammending updates the existing PR. You DO NOT need to open a new one.

LGTM

ping @moxiegirl

LGTM @crosbymichael sorry for the delay

authors: GordonTheTurtle crosbymichael lrkwz moxiegirl


text:
bugfix: report "destroy" after all volumes of container destroy

fixes #25766
If a container's AutoRemove is enabled, client will wait until it's
removed after container exits, this is implemented based on "destroy"
event.
Currently an "AutoRemove" container will report "destroy" event to
notify a hanging client to exit before all volumes are removed, this is
wrong, we should wait container until everything is cleaned up.
Signed-off-by: Zhang Wei zhangwei555@huawei.com

To reproduce the flaky test easily, we can add one line time.Sleep(1*time.Second) after https://github.com/docker/docker/blob/master/daemon/delete.go#L42
E.g.
	err = daemon.cleanupContainer(container, config.ForceRemove)
+      time.Sleep(1*time.Second)
	if err == nil || config.ForceRemove {
		if e := daemon.removeMountPoints(container, config.RemoveVolume); e != nil {
			logrus.Error(e)
		}
	}

	return err

LGTM. Thanks!

LGTM :cat:

authors: WeiZhang555 tonistiigi vdemeester


text:
[19.03 backport] Update containerd to v1.2.9

backport of

https://github.com/moby/moby/pull/39364 Update containerd to v1.2.7
https://github.com/moby/moby/pull/39786 Update containerd to v1.2.8
https://github.com/moby/moby/pull/39874 Update containerd to v1.2.9

Update containerd to v1.2.9. https://github.com/containerd/containerd/releases/tag/v1.2.9

The ninth patch release for containerd 1.2 provides a handful of bug fixes and an
update to the gRPC vendored codebase to include 3 CVE fixes provided in the upstream
v1.23.0 release of gRPC. Note that updating gRPC to the current release required small
changes to our core containerd codebase to match the upstream changes since gRPC v1.12.0.
These changes have been backported from containerd's master branch, as well as a
similar small change in ttrpc, requiring that package's vendoring to be updated.

In addition to the gRPC update to include CVE fixes, fixes were made to correct a
container's default Unix environment (introduced in 1.2.8), a small list of CRI plugin
fixes, as well as fixes for registry interactions where Docker-Content-Digest is not
returned (e.g. GitHub Package Registry), and a tar archive modification time bug found
by the buildkit maintainers. A fix to the zfs snapshotter was also included via a
re-vendoring of containerd's zfs import. More notes on these fixes are found below.

Update containerd to v1.2.8
https://github.com/containerd/containerd/releases/tag/v1.2.8
full diff: https://github.com/containerd/containerd/compare/85f6aa58b8a3170aec9824568f7a31832878b603...a4bc1d432a2c33aa2eed37f338dceabb93641310
The eighth patch release for containerd 1.2 provides a series of bug fixes, many
of them backported from the master branch to correct several known issues around
manifest lists/indexes and pulling multi-arch, CVEs related to Golang/http2,
fd leakage in the Golang runtime, a shim hang, process and image environment config
handling, and finally mount cleanup related to Cloud Foundry's use of containerd
with rootless containers.
From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7

Welcome to the v1.2.7 release of containerd!
The seventh patch release for containerd 1.2 introduces OCI image
descriptor annotation support and contains fixes for containerd shim logs,
container stop/deletion, cri plugin and selinux.
It also contains several important bug fixes for goroutine and file
descriptor leakage in containerd and containerd shims.
Notable Updates

Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254
Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31
Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37
Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264
Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273
Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby/moby#38978
Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282
Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283
cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177
cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159
Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50
Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316
Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330

marked as WIP because https://github.com/moby/moby/pull/39874 was not yet merged
first two commits are also in https://github.com/docker/engine/pull/282

PowerPC failure, looks like it was caused by Jenkins killing randomly

Everything else was green; I'll kick CI one more time to see if we can get green, otherwise I think we should merge

Is this a new flaky?

[2019-09-09T22:46:49.775Z] --- FAIL: TestServiceRemoveKeepsIngressNetwork (31.95s)
[2019-09-09T22:46:49.775Z]     service_test.go:232: Creating a new daemon at: /go/src/github.com/docker/docker/bundles/test-integration/1/TestServiceRemoveKeepsIngressNetwork
[2019-09-09T22:46:49.775Z]     daemon.go:321: [dd9ed34db0a08] waiting for daemon to start
[2019-09-09T22:46:49.775Z]     daemon.go:339: [dd9ed34db0a08] daemon started
[2019-09-09T22:46:49.775Z]     service_test.go:237: timeout hit after 30s: ingress not ready: 0 peers and 0 containers
[2019-09-09T22:46:49.775Z]     daemon.go:449: [dd9ed34db0a08] Stopping daemon
[2019-09-09T22:46:49.775Z]     daemon.go:288: [dd9ed34db0a08] exiting daemon
[2019-09-09T22:46:49.775Z]     daemon.go:436: [dd9ed34db0a08] Daemon stopped

No, it's not new; https://github.com/moby/moby/issues/39426. Moby currently skips that test; https://github.com/moby/moby/pull/39453 (and it was just backported to this repository, but not yet in this PR's branch: https://github.com/docker/engine/pull/350)

The other failure is TestSwarmLockUnlockCluster which is a known flaky test (https://github.com/moby/moby/issues/34051). (perhaps https://github.com/docker/engine/pull/345 might help a bit, not sure)

authors: kolyshkin thaJeztah

repo: docker/engine


text:
Better socket support

Updated some of the socket support along with changes to the event listeners. Unused logic was removed to clean up the code

Taking a look! Sorry for the delay

A couple comments!^

I'll change the if/else on the display message - good call

authors: FrenchBen jeffdm

repo: docker/kitematic


text:
fix some typos

Signed-off-by: bin liu liubin0329@gmail.com

LGTM

LGTM, @liubin can you please rebase?

Hi, @dave-tucker , I have rebased it, please check again.
Thanks,
Bin

@mrjana @mavenugo ready to merge this one?

Thanks. LGTM.

authors: aboch dave-tucker liubin mavenugo

repo: docker/libnetwork


text:
Delete endpoint from network map for macvlan driver upon endpoint deletion

Endpoint delete from network map in macvlan driver.
Signed-off-by: Sainath Grandhi sainath.grandhi@intel.com

ping @mavenugo  PTAL

LGTM

Can you please pass the proper parameter, the endpoint id.

Hi@sainath14 if you wouldn't mind changing the patch to n.deleteEndpoint(ep.id) as the method is looking for a string. Thanks!

sorry, missed it... please check now

Thanks, LGTM

I am seeing a rebase conflict. d.storeDelete(ep) is being added an upstream change. Adding deleteEndpoing after the storeDelete sounds correct to me.

If its from PR #1196 don't worry about that. That is blocked until v1.13. That what you were referring to?

It was from PR #1244

LGTM

authors: aboch nerdalert sainath14


text:
Remove extra unused fields

DeleteNeighbor does not need the extra boolean for the cleanup of the OS
The mask of the peer was not really used, also the assumption is that networks
are never overlapping
Signed-off-by: Flavio Crisciani flavio.crisciani@docker.com

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@659480c). Click here to learn what that means.
The diff coverage is 8.45%.

@@            Coverage Diff            @@
##             master    #1980   +/-   ##
=========================================
  Coverage          ?   37.99%
=========================================
  Files             ?      137
  Lines             ?    27333
  Branches          ?        0
=========================================
  Hits              ?    10386
  Misses            ?    15675
  Partials          ?     1272

Impacted Files
Coverage Δ

drivers/overlay/ov_network.go
0.26% <0%> (ø)

drivers/overlay/joinleave.go
0% <0%> (ø)

osl/neigh_linux.go
0% <0%> (ø)

drivers/overlay/overlay.go
27.52% <0%> (ø)

drivers/overlay/ov_endpoint.go
0% <0%> (ø)

drivers/overlay/ov_serf.go
30.86% <0%> (ø)

drivers/overlay/peerdb.go
7.57% <25%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 659480c...f2ecb67. Read the comment docs.

ping @fcrisciani - needs a rebase, because we removed solaris support

authors: codecov-io fcrisciani thaJeztah


text:
Change RSA key size to at least 1024 bits

OpenSSL-FIPS does not allow RSA keys to be generated if the key size is less than 1024. So the RSA key sizes in tests have been updated to prevent them from failing if Notary is made FIPS compliant.
Signed-off-by: Ali Yousuf aly.yousuf7@gmail.com

This LGTM.
Non-blocking:  we disallow key sizes < 2048 for certicates, but we may want to disallow any key size, certificate or not, < 2048.

repo: docker/notary


text:
Network List API Name Filters

When requesting a list of networks from the Swarm API endpoint with a name filter, overlay networks that match the filter are not returned. The network is returned in an unfiltered list request.

I'll take a look

I'd love to help testing. @tacchino which version of Swarm are you using? Is it 1.1-rc2?

@chanwit I was running 1.0.1

I've just tested with a master be96293. This bug is still be there but seems to be a bit different behaviour.
debian-1gb-sgp1-01:~
# docker version
Client:
 Version:      1.10.0-rc1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   677c593
 Built:        Fri Jan 15 18:21:13 2016
 OS/Arch:      linux/amd64

Server:
 Version:      swarm/1.1.0
 API version:  1.21
 Go version:   go1.5.2
 Git commit:   be96293
 Built:        Tue Jan 26 00:55:57 UTC 2016
 OS/Arch:      linux/amd64
debian-1gb-sgp1-01:~
# docker network ls
NETWORK ID          NAME                  DRIVER
2c7820ee92f5        multihost             overlay
5f05cb1d5141        ocean-master/host     host
3fbd594d606a        ocean-master/bridge   bridge
d554f3464e5a        ocean-master/none     null
a9aca4b8667f        ocean-1/bridge        bridge
9551a4f79d6b        ocean-1/host          host
abcaaec00b35        ocean-2/host          host
ca3884c2df02        ocean-1/none          null
422019490fe9        ocean-2/bridge        bridge
e7cb3add6dcf        ocean-2/none          null
debian-1gb-sgp1-01:~
# docker network ls --filter name=multihost
NETWORK ID          NAME                DRIVER

Thanks for the fast turn around!

authors: chanwit tacchino vieux

repo: docker/swarm