text:
XGBoostError save_raw() on mac M1

Hi,
I've installed xgboost from source using pip install -U --no-cache xgboost --no-binary xgboost -v and I am having weird crashes, actually, what I noticed is that each time a command issues some warning (it can be as simple as a deprecation message), the function being called crashes.
An example:
dump = model.get_booster().save_raw()

[17:32:17] WARNING: [/Users/runner/work/xgboost/xgboost/src/c_api/c_api.cc:1240](https://file+.vscode-resource.vscode-cdn.net/Users/runner/work/xgboost/xgboost/src/c_api/c_api.cc:1240): Saving into deprecated binary model format, please consider using `json` or `ubj`. Model format will default to JSON in XGBoost 2.2 if not specified.

XGBoostError                              Traceback (most recent call last)
Cell In[12], line 1
-> 20     dump = model.get_booster().save_raw()

...
  [bt] (6) 7   libffi.8.dylib                      0x000000010567174c ffi_call_int + 1208
  [bt] (7) 8   _ctypes.cpython-310-darwin.so       0x0000000105650c88 _ctypes_callproc + 1260
  [bt] (8) 9   _ctypes.cpython-310-darwin.so       0x000000010564af58 PyCFuncPtr_call + 1132

This example comes from a practical usage that currently makes the shap library unusable on mac m1.
Another one (this is a dummy one, where I tell to save the model in a dir, rather than a file):
model.save("./path_that_should_be_a_file/")

XGBoostError                              Traceback (most recent call last)
Cell In[13], line 1
----> 1 model.save_model

File ... in XGBModel.save_model(self, fname)
    834 meta_str = json.dumps(meta)
    835 self.get_booster().set_attr(scikit_learn=meta_str)
--> 836 self.get_booster().save_model(fname)
    837 self.get_booster().set_attr(scikit_learn=None)

File .. in Booster.save_model(self, fname)
   2530 if isinstance(fname, (str, os.PathLike)):  # assume file name
   2531     fname = os.fspath(os.path.expanduser(fname))
-> 2532     _check_call(_LIB.XGBoosterSaveModel(self.handle, c_str(fname)))
   2533 else:
   2534     raise TypeError("fname must be a string or os PathLike")

File [~/Library/Caches/pypoetry/virtualenvs/churn-prediction-Pm6D_DmO-py3.10/lib/python3.10/site-packages/xgboost/core.py:281](https://file+.vscode-resource.vscode-cdn.net/Users/../projects/../notebooks/train_xgboost/~/Library/Caches/pypoetry/virtualenvs/churn-prediction-Pm6D_DmO-py3.10/lib/python3.10/site-packages/xgboost/core.py:281), in _check_call(ret)
    270 """Check the return value of C API call
    271
    272 This function will raise exception when error occurs.
   (...)
    278     return value from API calls
    279 """
...
  [bt] (6) 7   _ctypes.cpython-310-darwin.so       0x0000000105650c88 _ctypes_callproc + 1260
  [bt] (7) 8   _ctypes.cpython-310-darwin.so       0x000000010564af58 PyCFuncPtr_call + 1132
  [bt] (8) 9   python3.10                          0x0000000104d9faec _PyObject_MakeTpCall + 612

If I input a correct file path, the command works just fine.
Any ideas?

What happens if you install xgboost using pip from PyPI?
cc @hcho3

Already tested it, but same result:
dump = model.get_booster().save_raw()

UserWarning: [18:12:53] WARNING: [/Users/runner/work/xgboost/xgboost/src/c_api/c_api.cc:1240](https://file+.vscode-resource.vscode-cdn.net/Users/runner/work/xgboost/xgboost/src/c_api/c_api.cc:1240): Saving into deprecated binary model format, please consider using `json` or `ubj`. Model format will default to JSON in XGBoost 2.2 if not specified.
  warnings.warn(smsg, UserWarning)

XGBoostError                              Traceback (most recent call last)
Cell In[10], line 1
----> 1 d = model.get_booster().save_raw()#raw_format="json")

File [~/Library/Caches/pypoetry/virtualenvs/churn-prediction-Pm6D_DmO-py3.10/lib/python3.10/site-packages/xgboost/core.py:2553](https://file+.vscode-resource.vscode-cdn.net/Users/../projects/churn-prediction/notebooks/train_xgboost/~/Library/Caches/pypoetry/virtualenvs/../lib/python3.10/site-packages/xgboost/core.py:2553), in Booster.save_raw(self, raw_format)
   2551 cptr = ctypes.POINTER(ctypes.c_char)()
   2552 config = make_jcargs(format=raw_format)
-> 2553 _check_call(
   2554     _LIB.XGBoosterSaveModelToBuffer(
   2555         self.handle, config, ctypes.byref(length), ctypes.byref(cptr)
   2556     )
   2557 )
   2558 return ctypes2buffer(cptr, length.value)

File [~/Library/Caches/pypoetry/virtualenvs/../lib/python3.10/site-packages/xgboost/core.py:281](https://file+.vscode-resource.vscode-cdn.net/Users/../projects/churn-prediction/notebooks/train_xgboost/~/Library/Caches/pypoetry/virtualenvs/../lib/python3.10/site-packages/xgboost/core.py:281), in _check_call(ret)
    270 """Check the return value of C API call
    271
    272 This function will raise exception when error occurs.
   (...)
    278     return value from API calls
    279 """
    280 if ret != 0:
--> 281     raise XGBoostError(py_str(_LIB.XGBGetLastError()))
...
  [bt] (6) 7   libffi.8.dylib                      0x00000001035a974c ffi_call_int + 1208
  [bt] (7) 8   _ctypes.cpython-310-darwin.so       0x0000000103588c88 _ctypes_callproc + 1260
  [bt] (8) 9   _ctypes.cpython-310-darwin.so       0x0000000103582f58 PyCFuncPtr_call + 1132

Then I tried to comment out that exception check in core.py: what happens in this particular case is that the dump variable contains an empty bytearray. So something breaks during the process, but it's hidden from the final stack trace.

My initial guess is the warning call in Python is throwing exception somehow. Will try to get a Mac device from AWS tomorrow.

Can you provide more details about your Mac M1 machine?
I just tried the same command on AWS's Mac instance and could not reproduce the error.

Macbook Pro, M1 Pro, Ventura 13.0.1.
Python 3.10.12 via poetry. Tried the same also with a conda venv, but same result.

Here's my M1 machine.
$ sw_vers -productVersion
13.5.2

$  system_profiler SPSoftwareDataType
Software:

    System Software Overview:

      System Version: macOS 13.5.2 (22G91)
      Kernel Version: Darwin 22.6.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      User Name: ec2-user (ec2-user)
      Secure Virtual Memory: Enabled
      System Integrity Protection: Enabled
      Time since boot: 12 hours, 1 minute

❯ sw_vers -productVersion
13.0.1
❯ system_profiler SPSoftwareDataType
Software:

    System Software Overview:

      System Version: macOS 13.0.1 (22A400)
      Kernel Version: Darwin 22.1.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      Computer Name: ...
      User Name: ...
      Secure Virtual Memory: Enabled
      System Integrity Protection: Enabled
      Time since boot: 8 days, 23 hours, 9 minutes

❯ python --version
Python 3.10.12

So both of us are using MacOS Ventura, with different patch versions. I don't think this should make much difference.
Unless we can reproduce the bug on our end, it will be difficult for us to fix the bug unfortunately.

I am trying to purge pip cache and retry. Can you tell me which version of clang, gcc and cmake do you have?
Mine:
❯ clang --version
Homebrew clang version 15.0.4
Target: arm64-apple-darwin22.1.0
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin
❯ cmake --version
cmake version 3.24.3

CMake suite maintained and supported by Kitware (kitware.com/cmake).
❯ gcc --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

$ clang --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

$ cmake --version
cmake version 3.27.7

CMake suite maintained and supported by Kitware (kitware.com/cmake).

$ gcc --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

I've tried to uninstall llvm so that clang and gcc are by apple, create an env from scratch, pip cache purge, install requirements.. same story. This is a weird one, for sure something wrong on my system. I'll try to replicate it with someone at work.
Thank you for your time!

Since you have already gone to length to debug this and have familiarity with compiling from the source, let's try to debug the code directly. During warning log:
https://github.com/dmlc/xgboost/blob/ea9f09716bd426f235ce785d6286e0ad26350ce1/src/logging.cc#L20
calls:
https://github.com/dmlc/xgboost/blob/ea9f09716bd426f235ce785d6286e0ad26350ce1/python-package/xgboost/core.py#L156
through Python ctypes callback. See if there's anything wrong with the callback.
This is the implementation of save raw: https://github.com/dmlc/xgboost/blob/ea9f09716bd426f235ce785d6286e0ad26350ce1/src/c_api/c_api.cc#L1321 , it returns inside the API_END macro if there's a c++ exception, which is caught here https://github.com/dmlc/xgboost/blob/ea9f09716bd426f235ce785d6286e0ad26350ce1/src/c_api/c_api_error.h#L31 .

You can easily try to reproduce that with this notebook:
_reproduce_bug_xgboost.ipynb.zip

I can reproduce the error, but it has full error trace with the correct message:
/home/jiamingy/workspace/xgboost_dev/xgboost/python-package/xgboost/core.py:160: UserWarning: [00:23:29] WARNING: /home/jiamingy/workspace/xgboost_dev/xgboost/src/c_api/c_api.cc:1273: Saving into deprecated binary model format, please consider using `json` or `ubj`. Model format will default to JSON in XGBoost 2.2 if not specified.
  warnings.warn(smsg, UserWarning)
Traceback (most recent call last):
  File "/home/jiamingy/workspace/xgboost_dev/XGBoostUtils/issues/9683/reproduce.py", line 68, in <module>
    dump = model.get_booster().save_raw()
  File "/home/jiamingy/workspace/xgboost_dev/xgboost/python-package/xgboost/core.py", line 2593, in save_raw
    _check_call(
  File "/home/jiamingy/workspace/xgboost_dev/xgboost/python-package/xgboost/core.py", line 281, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [00:23:29] /home/jiamingy/workspace/xgboost_dev/xgboost/src/tree/tree_model.cc:899: Check failed: !HasCategoricalSplit(): Please use JSON/UBJSON for saving models with categorical splits.
Stack trace:
  [bt] (0) /home/jiamingy/workspace/xgboost_dev/xgboost/python-package/xgboost/../../lib/libxgboost.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x80) [0x7f5ef7d80480]
  [bt] (1) /home/jiamingy/workspace/xgboost_dev/xgboost/python-package/xgboost/../../lib/libxgboost.so(xgboost::RegTree::Save(dmlc::Stream*) const+0x4d2) [0x7f5ef81e3f02]
...

emphasis on:
model.cc:899: Check failed: !HasCategoricalSplit(): Please use JSON/UBJSON for saving models with categorical splits.

You should use json/ubjson during save_raw:
dump = model.get_booster().save_raw(raw_format="json")
dump = model.get_booster().save_raw(raw_format="ubj")

I don't know why the message is missing in your error trace, something is wrong with logging.

authors: hcho3 masc-it trivialfis


text:
Passing updater parameter produces warning about ignored tree_method even if the latter is not passed

If one passes parameter updater, xgboost will throw a warning about tree_method being ignored, even if said parameter is not passed. The warning occurs in both python and R, and happens only the first time that xgb.train with said parameter is executed.
import numpy as np, xgboost as xgb
rng = np.random.default_rng(seed=123)
y = rng.integers(2, size=100)
X = rng.standard_normal(size=(100,10))
dm = xgb.DMatrix(X, y)
model = xgb.train(
    dtrain=dm,
    num_boost_round=10,
    params={
        "objective" : "binary:logistic",
        "max_depth" : 2,
        "eta" : 0.05,
        "updater" : "grow_colmaker,prune",
    }
)

WARNING: You have manually specified the `updater` parameter. The `tree_method` parameter will be ignored. Incorrect sequence of updaters will produce undefined behavior. For common uses, we recommend using `tree_method` parameter instead.

Subsequent executions of the same call to xgb.train do not produce the warning twice.

Thank you for raising the issue.
It's a really old warning that predates my involvement. From my understanding, initially, xgboost tried to support having a sequence of updaters as you shared in the snippet, but in practice, it's not quite working for most of the cases since the updater/optimizer has internal states to keep track of things like prediction cache. Then, the prune updater is not quite useful during training, since not growing the tree is the same as (if not better than) pruning a tree.
We kept the parameter for gbtree, but recommend people to use tree_method instead.
I will try to elevate the rest of the updaters into tree methods.

source: gharchive/issue

authors: david-cortes trivialfis

repo: dmlc/xgboost

license_type: permissive

license_source: github-api


text:
Fatal error compiling: CompilerException: NullPointerException

Operating system: Arch linux
Java version: openjdk 12.0.2
Maven version: 3.6.2
~/Downloads/Elegit(master) » mvn clean compile jfx:build-jar assembly:single

[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for elegit:Elegit:jar:1.1.1
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-surefire-plugin is missing. @ line 41, column 21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
Downloading from central: https://repo.maven.apache.org/maven2/com/zenjava/javafx-maven-plugin/8.5.0/javafx-maven-plugin-8.5.0.pom
Downloaded from central: https://repo.maven.apache.org/maven2/com/zenjava/javafx-maven-plugin/8.5.0/javafx-maven-plugin-8.5.0.pom (15 kB at 27 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/com/zenjava/javafx-maven-plugin/8.5.0/javafx-maven-plugin-8.5.0.jar
Downloaded from central: https://repo.maven.apache.org/maven2/com/zenjava/javafx-maven-plugin/8.5.0/javafx-maven-plugin-8.5.0.jar (84 kB at 428 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-assembly-plugin/3.1.0/maven-assembly-plugin-3.1.0.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-assembly-plugin/3.1.0/maven-assembly-plugin-3.1.0.pom (16 kB at 304 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-assembly-plugin/3.1.0/maven-assembly-plugin-3.1.0.jar
Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-assembly-plugin/3.1.0/maven-assembly-plugin-3.1.0.jar (241 kB at 988 kB/s)
[INFO]
[INFO] ---------------------------< elegit:Elegit >----------------------------
[INFO] Building Elegit 1.1.1
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ Elegit ---
[INFO] Deleting ~/Downloads/Elegit/target
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ Elegit ---
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO] Copying 32 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ Elegit ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 108 source files to ~/Downloads/Elegit/target/classes
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.607 s
[INFO] Finished at: 2019-09-27T13:28:29+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project Elegit: Fatal error compiling: CompilerException: NullPointerException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

Thanks. This is almost certainly related to issue #637.

authors: dmusican piegamesde

repo: dmusican/Elegit

license: mit

license_source: bigquery


text:
Changes for transfer IDs logic

Fixes #<GH_Issue_Number>
Status
In development
Description
It implements logic for MS based on transfer IDs. Here is how it is done:

transferor API obtain request info and perform transferRequest
the transferRequest performs phedex subscription (via subscribe API which
so far is commented out) and obtain transfer ids via getTransferIds API
It does this for every dataset in request spec. At the end transfer API
received status (transfer) records which its store to couchdb via
updateTransferInfo API (which calls apis from 9275)
monit API get requests and fetch status (transfer) records from couchdb
via getStatusRecords API (which by itself calls getTransferIds one).
Then I check status records in checkStatusRecords API to obtain
new set of records and its completed value. If completed value is
100 (percent) we are done, otherwise we update new set of records
in couchdb via updateTransferInfo API.

Is it backward compatible (if not, which system it affects?)
YES
Related PRs
depends on #9198 , #9275, #9307
External dependencies / deployment changes
no

As mentioned in the meeting, please create an issue reporting the problem and what's the expected behavior.

Alan, this PR already listed dependencies:  #9198 , #9275. The only missing bit is you need to decide which DB name to use and which structure (transfer IDs or suscription) we'll deal with. Once settle and we both issues will be merged I can move with this.

Valentin, I'm having a hard time reviewing this PR. Can you please describe in the first comment box what exactly are these changes? What is the flow of actions executed by MS here? and what has to be discussed/defined/fix ahead.

let's adjust it once you'll clarify which Phedex APIs to use.

ok, then I'll rebase once you'll merge https://github.com/dmwm/WMCore/pull/9307 and continue working on this

Done, feel free to proceed with this one. I'm going to ask a new testbed release tomorrow morning, so we might have to postpone it until the next week (if safe changes).

Alan, do you mind to explain why did you re-factor MS code and what exactly you did.
I see SO MANY changes that I want to understand the entire logic before proceeding. For instance, to preserve previous work I need clear understanding which branch to use to resolve the conflicts, the head or the one I had before. I need to understand which methods/functions to keep, etc.
And, for the future, please elaborate and discuss the changes up-front. I already rewrote this piece so many times that I start having a feeling that I'm wasting my time on this codebase.

Because the transferor code was looking an all-in-one, where we were piling up methods and more methods without really thinking where things belong to, without reusing code.
I have all that explained in the first comment of this PR:
https://github.com/dmwm/WMCore/pull/9296
where you were tagged. I also asked you to have a closer look once you get back from vacation (it was merged in the past week). Let me know if you still have questions, but we better then follow up on that PR to keep the discussion in the same context.

I have started the basic description/behaviour of the Monitor thread here:
https://github.com/dmwm/WMCore/wiki/ReqMgr2-MicroService-Monitor
but as you know, we might have to touch it once we fully understand the "black box" logic of the transferor service and all the parameters it depends on. Which made me create this option D here:
https://gist.github.com/amaltaro/72599f995b37a6e33566f3c749143154#file-transferor_format-json-L32
and add an extra field campaignName, such that we don;t need to look into a request detail once again in the monitor thread.

I performed rebase and added/fixed the following:

I re-factored all mine and Alan's changes to separate MS logic into the following classes:

MSCore.py represents core module for MSTransferor, MSMonitor, RequestInfo classes
I separated transferor, monitoring and request info into separate classes
I cleaned and implemented consistent logic in MSManager class

I refactored Alan's changes to make code thread safe (especially introduced in RequestInfo and MSManager classes)
I implemented logic based on option D from transfer records document (see https://gist.github.com/amaltaro/72599f995b37a6e33566f3c749143154)

I did not (yet) made any effort to complete full logic of transferor/monitoring MS services and suggest that it should be handled in next PR once we agree on these changes and merge them. I also didn't touch yet unified logic and suggest that it should be done after full MS logic will be in place.

@vkuznet you probably have a broken import here, that's why this avalanche of unit tests failing.
In addition to that, please do not disable pylint checks blindly, otherwise we never know what needs attention in the future or not.

Alan, I doubt it is broken import since pylint/pycodestyle will capture this. I also checked manually imports and they work fine. Plus I have doubt that such unit test:
WMCore_t.Services_t.DBS_t.DBSReader_t.DBSReaderTest:testListDatatiers changed from success to error

will be related to MS code. I think failure of unit tests is related by something else. Do you have any log to look what is exactly fails?
Regarding pylint checks, I don't exclude blindly. I excluded those which I believe irrelevant. For instance, here is a typical codes:
pylint --help-msg=C0103,R0914,R0912,R1702,R0201,W0703
No config file found, using default configuration
:invalid-name (C0103): *%s name "%s" doesn't conform to %s*
  Used when the name doesn't conform to naming rules associated to its type
  (constant, variable, class...). This message belongs to the basic checker.

:too-many-locals (R0914): *Too many local variables (%s/%s)*
  Used when a function or method has too many local variables. This message
  belongs to the design checker.

:too-many-branches (R0912): *Too many branches (%s/%s)*
  Used when a function or method has too many branches, making it hard to
  follow. This message belongs to the design checker.

:too-many-nested-blocks (R1702): *Too many nested blocks (%s/%s)*
  Used when a function or a method has too many nested blocks. This makes the
  code less understandable and maintainable. This message belongs to the
  refactoring checker.

:no-self-use (R0201): *Method could be a function*
  Used when a method doesn't use its bound instance, and so could be written as
  a function. This message belongs to the classes checker.

:broad-except (W0703): *Catching too general exception %s*
  Used when an except catches a too general exception, possibly burying
  unrelated errors. This message belongs to the exceptions checker.

If you want I can remove all pylints, but it is trivial to check, as shown, what codes means.

Besides C0103, I'd say all the others have a more meaningful impact on the code and can eventually be made better (even C0103 could, after all most of it is new code...). If you disable them, we never get it reported (unless we remember to check the pylints that were disabled).
I'm just saying we don't need to have a perfect score, but if possible not to make it worse...
About the error, if you follow the default Details link, it brings you to a summary of all the unit tests failing, e.g.:
https://cmssdt.cern.ch/dmwm-jenkins/job/DMWM-WMCore-PR-test/9231/testReport/junit/WMCore_t.WorkQueue_t.Policy_t.Start_t.Block_t/BlockTestCase/testLumiMask/
so the problem is not an import indeed, but the PhEDex module being mocked at the wrong place (you reshuffled things, so we don't need to mock PhEDEx there anymore). Ping me on slack if you need help walking through those reports.

Alan, I removed pylint exceptions, I fixed unit test. Please review since I don't to waste time fixing either pylint or irrelevant unit tests (Rucio).

Has it been tested? I need to request another WMCore tag tomorrow early in the morning and probably won't manage to have a thorough review and test.

Alan, I changed code according to your suggestions, but I didn't put slicing. Please provide a pointer and I can added it back. But once again, don't try to get ideal code until we understand its flow. Don't over optimize it until it is working, etc. I DO UNDERSTAND issue with memory consumption but we're still battling on codebase flow.
For the record code is not tested since I'm awaiting your ok with its logic.

Alan, I put back the slicer, but I really doubt it is proper solution. Instead, we should convert RequestInfo into generator, then it will not consume memory and neither being prompt to amount of records it process. The loop will be one pass across passed records where it will enhance them (if necessary) and yield back new set of records. I can do that code-refactoring later.
So far I suggest to finish this code refactoring and put it in place that we can work on top of it.

Alan, regarding your comment:
Besides those comments, I think we should fetch the transfer records in bulk such that we can avoid hundreds/thousands of calls, so in the MSMonitor, you fetch all the requests in a given status, then you fetch all the transfer. Amount of data should be pretty small (I foresee <20MB).

which bulk operation are you referring to? Please be explicit. The code now uses getTransferInfo API from ReqMgrAux. This API has workflow name as an input parameter. If you want to get all records from CouchDB you need to provide appropriate API for that. Otherwise I don't see ANY way to get all records (or do bulk operation). If you're talking about another topic please provide pointer to API(s) which does this bulk operation.

Valentin, about getting the transfer docs in bulk, it can be done using the same getTransferInfo API from ReqMgrAux module, by providing ALL_DOCS as the document name.
I updated this information here as well:
https://github.com/dmwm/WMCore/wiki/ReqMgr2-apis#retrieve-a-workflow-transfer-document

ok, here is what I've done:

use bulk call via ALL_DOCS, I'm not convinced that it will be faster and I put proper comment in a code
I renamed checkRecordsStatus to completion method, and make later accept transfer records to find out completion value
I renamed statusRecords into transferRecords for better code clarify
I implemented basic pileup logic based on wiki description (further tweaking may be required)
I left as is subscriptions call since Phedex API does not provide call via requests only, at the moment I don't see need to change anything until it is clarified
added completion threshold , read it from configuration

I think at this moment all requests are implemented. Feel free to review again.

Thanks, Valentin. I'll look into it tomorrow.
Just one more comment. The completion threshold does not come from the service configuration. It will either come from the unified config or from the campaign one.
Last but not least, can you please make a nice description in the first comment of this PR explaining what are the changes and so on? Thanks

Valentin, just fix those problems such that nothing crashes, and we can proceed and merge it.
I'll then work on their logic in another PR.

Alan, I fixed issues you pointed out and it require additional code refactoring (I was need to move common methods from MSMonitoring/MSTransferor to MSCore).

I also fixed reading threshold from unified config and updated description of this issue.

Alan, I put thr = 100 in a code and put in comments that we should fix it and provide example.
Feel free to proceed.

Alan, I submitted the fix for unifiedConfig.

Thanks, it seems to be working fine now. Once we hear back from Jenkins, I'll merge it and put in a new tag for Lina for testbed tomorrow.

source: gharchive/pull-request

authors: amaltaro vkuznet

repo: dmwm/WMCore


text:
During deployment, activate agent after applying PR patches

Status
Tested
Description
If you have a PR that updates the agent base config (WMAgentConfig.py) and you try to apply it during agent deployment, it never updates. That is because the "activate-agent" function of the manage script copies the default config to /data/srv/wmagent/current/config before the PR patches are applied.
https://github.com/dmwm/deployment/blob/2536c3673535d1e19515f3ca06c92db7007bf1a4/wmagent/manage#L102
This PR reverses the order so you can at least apply a WMAgentConfig.py patch during deployment by specifying the pull path to the file like this:
|diff --git a/etc/WMAgentConfig.py b/etc/WMAgentConfig.py
|index 6a0087de07..34049fcd79 100644
|--- a/etc/WMAgentConfig.py
|+++ b/etc/WMAgentConfig.py
--------------------------
File to patch: /data/srv/wmagent/current/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/etc/WMAgentConfig.py
patching file /data/srv/wmagent/current/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/etc/WMAgentConfig.py

Then you don't have to manually patch the agent config later (after wmagent-mod-config has already run), which is pretty annoying in some cases.
Is it backward compatible (if not, which system it affects?)
YES
Related PRs
NA
External dependencies / deployment changes
NA

@todor-ivanov Todor, can you please get this PR tested while deploying the backfill agent on vocms0285?

I just tested this patch and it works for normal src/python patches. I didn't test for configuration like patches though. @goughes have you tested that use case? If so, then it's good to go from my side.

I tested this. The config patches won't apply automatically since they are out of the src/python directory, but you can specify the absolute path to patch the files. Like the example in the PR description.
File to patch: /data/srv/wmagent/current/sw/slc7_amd64_gcc630/cms/wmagent/1.2.6.patch1/etc/WMAgentConfig.py

Is this patch good to go from your side? If so let me know and remove the "do not merge" label please

It's good from my side.

Thanks

authors: amaltaro goughes


text:
License Signature Validation Error

I wrote the below console application trying to test the implementaton of this. I have not been successful as i have been getting the "InvalidSignatureValidationErrorFailure" exception thrown.
I have pasted my code below. I appreciate any help you can give

     static void Main(string[] args)
    {
        string path = @"C:\Development\Licenses\";
        string fileName = "Licence.lic";

        Console.WriteLine("Enter the application name:");
        string appName = Console.ReadLine();

        Console.WriteLine("Do you wish to create a new License? (Y/N):");
        var createLicense = Console.ReadLine();

        if (createLicense.Equals("Y"))
        {
            Console.WriteLine("Enter the customer company:");
            var company = Console.ReadLine();

            Console.WriteLine("Enter the customer full name:");
            var fullname = Console.ReadLine();

            Console.WriteLine("Enter the customer email:");
            var email = Console.ReadLine();

            Console.WriteLine("Enter the validity days before expiry (number of days):");
            var days = Console.ReadLine();

            Console.WriteLine("Enter the number of allowed uses of this license (numbers only):");
            var count = Console.ReadLine();

            var license = License.New();
            license.LicensedTo(fullname, email);

            if (days != null)
            {
                license = license.ExpiresAt(DateTime.Now.AddDays(Int32.Parse(days)));
            }

            var productFeatures = new Dictionary<string, string>
            {
                {"Version", "1.0"}
            };

            string passphrase = GeneratePassPhraseForProduct();
            var keyPair = GenerateKeysForProduct(passphrase);

            string privateKey = keyPair.ToEncryptedPrivateKeyString(passphrase);
            string publicKey = keyPair.ToPublicKeyString();

            var finalLicense = license.WithUniqueIdentifier(Guid.NewGuid())
           .As(LicenseType.Standard)
           .WithMaximumUtilization(1)
           .WithProductFeatures(productFeatures)
           .CreateAndSignWithPrivateKey(privateKey, passphrase);

            Console.WriteLine("Private key: {0}", privateKey);
            Console.WriteLine("Public key: {0}", publicKey);

            System.IO.File.WriteAllText(path + string.Format("{0}-private-key.txt", appName), privateKey);
            System.IO.File.WriteAllText(path + string.Format("{0}-public-key.txt", appName), publicKey);

            File.WriteAllText(path + fileName, finalLicense.ToString(), Encoding.UTF8);

            Console.WriteLine("License file successfully created");

            Console.WriteLine("Do you wish to validate the licence? (Y/N)");
            var validate = Console.ReadLine();

            if (validate.Equals("Y"))
            {
                using (XmlReader reader = XmlReader.Create(path + fileName))
                {
                    License readLicence = License.Load(reader);

                    var validationFailures = readLicence.Validate()
                      .ExpirationDate()
                           .When(lic => lic.Type == LicenseType.Trial)
                      .And()
                      .Signature(publicKey)
                      .AssertValidLicense().ToList();

                    foreach (var failure in validationFailures)
                        Console.WriteLine(failure.GetType().Name + ": " + failure.Message + " - " + failure.HowToResolve);
                }
            }
        }

        Console.WriteLine("Press any key to exit");

        Console.ReadLine();
    }

    public static KeyPair GenerateKeysForProduct(string PassPhrase)
    {
        if (string.IsNullOrWhiteSpace(PassPhrase))
        {
            throw new Exception("No passphrase was given. Cannot generate keys.");
        }

        var keyGenerator = Portable.Licensing.Security.Cryptography.KeyGenerator.Create();
        var keyPair = keyGenerator.GenerateKeyPair();

        return keyPair;
    }

    public static string GeneratePassPhraseForProduct()
    {
        return Guid.NewGuid().ToString();
    }`

Have anyone find out the solution? i am also facing same error!!!

I think this is to do with using the XmlReader to load the file after generation.  If you use File.ReadAllText(using encoding) it works fine

authors: blackchineykh newky2k pankajmcait


text:
REST API 설계

DB 모델링과 더불어  API 설계까지 해야 합니다.
설계 이후에 목데이터를 프론트 개발자들에게 넘겨주어 개발이 원활히 진행되도록 해야합니다
이를 위해서는 기능 정리, ERD작성, DB와 관련된 작업이 선행되어야 합니다.

중복 이슈인 것 같아서 닫을게용

license: MIT


text:
DNN-30483: fix several bugs around export/import area.

Fix #2880

@zyhfish Dnn 9.4.0 is code frozen, can you please re-target development. Thanks.

authors: valadas zyhfish


text:
DNN-6818 - Added logic to check is DisplaySearch setting has a valid boo...

...lean value or not

merging into 7.4.2 and cherrypicking into release/7.4.1

authors: agrueso cathalconnolly


text:
Introduce Globals.GetCurrentServiceProvider

Summary
This PR introduces Globals.GetCurrentServiceProvider. This allows our fallbacks to use the correct scope for the vast majority
of cases where objects lifetimes follow the request scope.

Thanks @valadas, I agree and have marked those constructors as Obsolete

Thanks @valadas, I agree and have marked those constructors as Obsolete

Awesome, love it! Great work!

authors: bdukes valadas


text:
Error in installation script

running
$ curl -s https://raw.githubusercontent.com/dnote-io/cli/master/install.sh | sh
Error fetching latest version. Please try again.

I think the regex to parse version tag is broken
LATEST=$(curl -s https://api.github.com/repos/dnote-io/cli/tags | grep -Eo '"name": "v\d*\.\d*\.\d*",'  | head -n 1 | sed 's/[," ]//g' | cut -d ':' -f 2)

where my PLATFORM=linux-amd64, means I should get
https://github.com/dnote-io/cli/releases/download/v0.2.0/dnote-linux-amd64

I am currently unable to reproduce it.
Could you please share the output of curl -s https://api.github.com/repos/dnote-io/cli/tags. And let me know if trying again resolves the issue.

Same issue here. The output you requested:
$ curl -s https://api.github.com/repos/dnote-io/cli/tags
[
  {
    "name": "v0.2.0",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0",
    "commit": {
      "sha": "922bb4ba04cafff0669be1acf555413be979f882",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/922bb4ba04cafff0669be1acf555413be979f882"
    }
  },
  {
    "name": "v0.2.0-beta.1",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-beta.1",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-beta.1",
    "commit": {
      "sha": "5456ccecf55bf7c7b6ab18ccc8a76834fa499533",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/5456ccecf55bf7c7b6ab18ccc8a76834fa499533"
    }
  },
  {
    "name": "v0.2.0-alpha.4",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-alpha.4",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-alpha.4",
    "commit": {
      "sha": "3f57d446e8143abcfd0ade778d204edc82322d4f",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/3f57d446e8143abcfd0ade778d204edc82322d4f"
    }
  },
  {
    "name": "v0.2.0-alpha.3",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-alpha.3",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-alpha.3",
    "commit": {
      "sha": "41ee8eba0f2a8bed8eca8d31fac4b4c4bd0cb4f6",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/41ee8eba0f2a8bed8eca8d31fac4b4c4bd0cb4f6"
    }
  },
  {
    "name": "v0.2.0-alpha.2",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-alpha.2",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-alpha.2",
    "commit": {
      "sha": "d31b22aed05e3acca1dd2c076cc2f082b53c785f",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/d31b22aed05e3acca1dd2c076cc2f082b53c785f"
    }
  },
  {
    "name": "v0.2.0-alpha",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-alpha",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-alpha",
    "commit": {
      "sha": "9043dbe4953d10fd9aa91e947de434a9fbcaf89b",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/9043dbe4953d10fd9aa91e947de434a9fbcaf89b"
    }
  },
  {
    "name": "v0.1.3",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.1.3",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.1.3",
    "commit": {
      "sha": "9f554c40c0a974b409fe4cba8a814fb4dd1e4495",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/9f554c40c0a974b409fe4cba8a814fb4dd1e4495"
    }
  },
  {
    "name": "v0.1.2",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.1.2",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.1.2",
    "commit": {
      "sha": "9d6334c9d732f844ee906f802749235904147682",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/9d6334c9d732f844ee906f802749235904147682"
    }
  },
  {
    "name": "v0.1.1",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.1.1",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.1.1",
    "commit": {
      "sha": "156372e319b70316aa4ca3f270009aaaf0776063",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/156372e319b70316aa4ca3f270009aaaf0776063"
    }
  },
  {
    "name": "v0.1.0",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.1.0",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.1.0",
    "commit": {
      "sha": "65a9e1e916ec31f346df732f8d0827a7c1e81075",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/65a9e1e916ec31f346df732f8d0827a7c1e81075"
    }
  },
  {
    "name": "v0.1.0-beta.2",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.1.0-beta.2",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.1.0-beta.2",
    "commit": {
      "sha": "80ea5a1e88c7384880a6bfa659a69ebf66845b44",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/80ea5a1e88c7384880a6bfa659a69ebf66845b44"
    }
  },
  {
    "name": "v0.1.0-beta.1",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.1.0-beta.1",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.1.0-beta.1",
    "commit": {
      "sha": "f67c29be05b95da5b3823c3e314e0ce44f8e77f4",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/f67c29be05b95da5b3823c3e314e0ce44f8e77f4"
    }
  },
  {
    "name": "v0.0.3",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.0.3",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.0.3",
    "commit": {
      "sha": "bb388062e3e3e77fd3c1c86bf1a08cca47006a81",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/bb388062e3e3e77fd3c1c86bf1a08cca47006a81"
    }
  },
  {
    "name": "v0.0.2",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.0.2",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.0.2",
    "commit": {
      "sha": "d623488da7648b753126a52290966410463668cc",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/d623488da7648b753126a52290966410463668cc"
    }
  },
  {
    "name": "v0.0.1",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.0.1",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.0.1",
    "commit": {
      "sha": "d623488da7648b753126a52290966410463668cc",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/d623488da7648b753126a52290966410463668cc"
    }
  }
]

@sungwoncho output of curl -s https://api.github.com/repos/dnote-io/cli/tags
I'm running Linux Mint Sarah (Ubuntu 16.04 LTS)
[
  {
    "name": "v0.2.0",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0",
    "commit": {
      "sha": "922bb4ba04cafff0669be1acf555413be979f882",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/922bb4ba04cafff0669be1acf555413be979f882"
    }
  },
  {
    "name": "v0.2.0-beta.1",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-beta.1",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-beta.1",
    "commit": {
      "sha": "5456ccecf55bf7c7b6ab18ccc8a76834fa499533",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/5456ccecf55bf7c7b6ab18ccc8a76834fa499533"
    }
  },
  {
    "name": "v0.2.0-alpha.4",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-alpha.4",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-alpha.4",
    "commit": {
      "sha": "3f57d446e8143abcfd0ade778d204edc82322d4f",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/3f57d446e8143abcfd0ade778d204edc82322d4f"
    }
  },
  {
    "name": "v0.2.0-alpha.3",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-alpha.3",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-alpha.3",
    "commit": {
      "sha": "41ee8eba0f2a8bed8eca8d31fac4b4c4bd0cb4f6",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/41ee8eba0f2a8bed8eca8d31fac4b4c4bd0cb4f6"
    }
  },
  {
    "name": "v0.2.0-alpha.2",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-alpha.2",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-alpha.2",
    "commit": {
      "sha": "d31b22aed05e3acca1dd2c076cc2f082b53c785f",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/d31b22aed05e3acca1dd2c076cc2f082b53c785f"
    }
  },
  {
    "name": "v0.2.0-alpha",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-alpha",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-alpha",
    "commit": {
      "sha": "9043dbe4953d10fd9aa91e947de434a9fbcaf89b",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/9043dbe4953d10fd9aa91e947de434a9fbcaf89b"
    }
  },
  {
    "name": "v0.1.3",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.1.3",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.1.3",
    "commit": {
      "sha": "9f554c40c0a974b409fe4cba8a814fb4dd1e4495",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/9f554c40c0a974b409fe4cba8a814fb4dd1e4495"
    }
  },
  {
    "name": "v0.1.2",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.1.2",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.1.2",
    "commit": {
      "sha": "9d6334c9d732f844ee906f802749235904147682",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/9d6334c9d732f844ee906f802749235904147682"
    }
  },
  {
    "name": "v0.1.1",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.1.1",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.1.1",
    "commit": {
      "sha": "156372e319b70316aa4ca3f270009aaaf0776063",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/156372e319b70316aa4ca3f270009aaaf0776063"
    }
  },
  {
    "name": "v0.1.0",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.1.0",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.1.0",
    "commit": {
      "sha": "65a9e1e916ec31f346df732f8d0827a7c1e81075",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/65a9e1e916ec31f346df732f8d0827a7c1e81075"
    }
  },
  {
    "name": "v0.1.0-beta.2",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.1.0-beta.2",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.1.0-beta.2",
    "commit": {
      "sha": "80ea5a1e88c7384880a6bfa659a69ebf66845b44",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/80ea5a1e88c7384880a6bfa659a69ebf66845b44"
    }
  },
  {
    "name": "v0.1.0-beta.1",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.1.0-beta.1",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.1.0-beta.1",
    "commit": {
      "sha": "f67c29be05b95da5b3823c3e314e0ce44f8e77f4",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/f67c29be05b95da5b3823c3e314e0ce44f8e77f4"
    }
  },
  {
    "name": "v0.0.3",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.0.3",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.0.3",
    "commit": {
      "sha": "bb388062e3e3e77fd3c1c86bf1a08cca47006a81",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/bb388062e3e3e77fd3c1c86bf1a08cca47006a81"
    }
  },
  {
    "name": "v0.0.2",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.0.2",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.0.2",
    "commit": {
      "sha": "d623488da7648b753126a52290966410463668cc",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/d623488da7648b753126a52290966410463668cc"
    }
  },
  {
    "name": "v0.0.1",
    "zipball_url": "https://api.github.com/repos/dnote-io/cli/zipball/v0.0.1",
    "tarball_url": "https://api.github.com/repos/dnote-io/cli/tarball/v0.0.1",
    "commit": {
      "sha": "d623488da7648b753126a52290966410463668cc",
      "url": "https://api.github.com/repos/dnote-io/cli/commits/d623488da7648b753126a52290966410463668cc"
    }
  }
]

Same for me.
Output of the tags curl:
[{"name":"v0.2.0","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0","commit":{"sha":"922bb4ba04cafff0669be1acf555413be979f882","url":"https://api.github.com/repos/dnote-io/cli/commits/922bb4ba04cafff0669be1acf555413be979f882"}},{"name":"v0.2.0-beta.1","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-beta.1","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-beta.1","commit":{"sha":"5456ccecf55bf7c7b6ab18ccc8a76834fa499533","url":"https://api.github.com/repos/dnote-io/cli/commits/5456ccecf55bf7c7b6ab18ccc8a76834fa499533"}},{"name":"v0.2.0-alpha.4","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-alpha.4","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-alpha.4","commit":{"sha":"3f57d446e8143abcfd0ade778d204edc82322d4f","url":"https://api.github.com/repos/dnote-io/cli/commits/3f57d446e8143abcfd0ade778d204edc82322d4f"}},{"name":"v0.2.0-alpha.3","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-alpha.3","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-alpha.3","commit":{"sha":"41ee8eba0f2a8bed8eca8d31fac4b4c4bd0cb4f6","url":"https://api.github.com/repos/dnote-io/cli/commits/41ee8eba0f2a8bed8eca8d31fac4b4c4bd0cb4f6"}},{"name":"v0.2.0-alpha.2","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-alpha.2","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-alpha.2","commit":{"sha":"d31b22aed05e3acca1dd2c076cc2f082b53c785f","url":"https://api.github.com/repos/dnote-io/cli/commits/d31b22aed05e3acca1dd2c076cc2f082b53c785f"}},{"name":"v0.2.0-alpha","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.2.0-alpha","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.2.0-alpha","commit":{"sha":"9043dbe4953d10fd9aa91e947de434a9fbcaf89b","url":"https://api.github.com/repos/dnote-io/cli/commits/9043dbe4953d10fd9aa91e947de434a9fbcaf89b"}},{"name":"v0.1.3","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.1.3","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.1.3","commit":{"sha":"9f554c40c0a974b409fe4cba8a814fb4dd1e4495","url":"https://api.github.com/repos/dnote-io/cli/commits/9f554c40c0a974b409fe4cba8a814fb4dd1e4495"}},{"name":"v0.1.2","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.1.2","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.1.2","commit":{"sha":"9d6334c9d732f844ee906f802749235904147682","url":"https://api.github.com/repos/dnote-io/cli/commits/9d6334c9d732f844ee906f802749235904147682"}},{"name":"v0.1.1","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.1.1","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.1.1","commit":{"sha":"156372e319b70316aa4ca3f270009aaaf0776063","url":"https://api.github.com/repos/dnote-io/cli/commits/156372e319b70316aa4ca3f270009aaaf0776063"}},{"name":"v0.1.0","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.1.0","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.1.0","commit":{"sha":"65a9e1e916ec31f346df732f8d0827a7c1e81075","url":"https://api.github.com/repos/dnote-io/cli/commits/65a9e1e916ec31f346df732f8d0827a7c1e81075"}},{"name":"v0.1.0-beta.2","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.1.0-beta.2","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.1.0-beta.2","commit":{"sha":"80ea5a1e88c7384880a6bfa659a69ebf66845b44","url":"https://api.github.com/repos/dnote-io/cli/commits/80ea5a1e88c7384880a6bfa659a69ebf66845b44"}},{"name":"v0.1.0-beta.1","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.1.0-beta.1","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.1.0-beta.1","commit":{"sha":"f67c29be05b95da5b3823c3e314e0ce44f8e77f4","url":"https://api.github.com/repos/dnote-io/cli/commits/f67c29be05b95da5b3823c3e314e0ce44f8e77f4"}},{"name":"v0.0.3","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.0.3","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.0.3","commit":{"sha":"bb388062e3e3e77fd3c1c86bf1a08cca47006a81","url":"https://api.github.com/repos/dnote-io/cli/commits/bb388062e3e3e77fd3c1c86bf1a08cca47006a81"}},{"name":"v0.0.2","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.0.2","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.0.2","commit":{"sha":"d623488da7648b753126a52290966410463668cc","url":"https://api.github.com/repos/dnote-io/cli/commits/d623488da7648b753126a52290966410463668cc"}},{"name":"v0.0.1","zipball_url":"https://api.github.com/repos/dnote-io/cli/zipball/v0.0.1","tarball_url":"https://api.github.com/repos/dnote-io/cli/tarball/v0.0.1","commit":{"sha":"d623488da7648b753126a52290966410463668cc","url":"https://api.github.com/repos/dnote-io/cli/commits/d623488da7648b753126a52290966410463668cc"}}]

I changed the regex from
grep -Eo '"name": "v\d*\.\d*\.\d*",'

to
grep -Eo '"name":"v\d*\.\d*\.\d*",'

and it works.

The install does not work on Linux.
On Linux, the problem is that you're using a non-POSIX regex. \d is not supported on Linux by default. Use [0-9] (technically, the analogous term is [[:digit:]] but that's cumbersome). You could also use the -P flag to enable Perl regexes, but I do not know how portable that is.
Combined with the problem of spaces mentioned by @michaelcontento, the fix is:
grep -Eo '"name":[ ]*"v[0-9]*\.[0-9]*\.[0-9]*",'

authors: expz hemache lbrito1 michaelcontento sungwoncho

repo: dnote-io/cli


text:
Tentative: Explicitely support device instead of using use_cuda (see http://pytorch.org/docs/stable/tensor_attributes.html#torch-device).

Currently, the user indicates use_cuda=True/False. With pytorch 0.4, it is suggested to use device instead.
It could be worth to make a breaking change (or transition) that removes use_cuda and adds a device parameter. If device=None, we use:
self.device_ = torch.device("cuda" if use_cuda else "cpu")

as per the suggestion of the migration guide.
We should also allow a string to be passed, e.g. 'cpu' or 'cuda'.
The change from use_cuda to device will probably allow for more diverse backends in the future.

I suggest to change the name of location in forward to device as well so everything's consistent.

authors: benjamin-work ottonemo

repo: dnouri/skorch


text:
how to  presisting data between different states in stateprovider

Please paste the output of the following command here: sudo edit-chroot -all

// Code goes here
angular.module('main', ['main.guides', 'main.projects', 'ui.router']).config(function ($stateProvider, $urlRouterProvider) {
$stateProvider.
state('main', {
url: '/',
template: 'index',
controller: 'IndexCtrl'
}).
state('login', {
url: '/login',
template: 'login',
controller: 'LoginCtrl'
}).
state('guides', {
url: '/guides',
template: 'guides',
controller: 'GuidesCtrl'
}).
state('projects', {
url: '/projects',
templateUrl: 'project.html',
controller: 'ProjectsCtrl'
});
    $urlRouterProvider.otherwise('/main');

})
.controller('IndexCtrl', function() {
console.log('Index');
})
.controller('LoginCtrl', function() {
console.log('Login');
})
.controller('GuidesCtrl', function() {
console.log('GuidesCtrl');
})
.controller('ProjectsCtrl', function() {
console.log('Projects');
})
.controller('ListCtrl', function() {
console.log('List');
})
angular.module('main.guides', ['ui.router']).config(function ($stateProvider) {
$stateProvider.
state('guides.login', {
url: '/login',
template: 'guides login',
controller: 'LoginCtrl'
}).
state('guides.menu', {
url: '/login',
template: 'guides-menu',
controller: 'LoginCtrl'
}).
state('guides.web', {
url: '/web',
template: 'guides web',
controller: 'ListCtrl'
}).
state('guides.mobile', {
url: '/web',
template: 'guildes mobile',
controller: 'ListCtrl'
});
});
angular.module('main.projects', ['ui.router']).config(function ($stateProvider) {
$stateProvider.
state('projects.login', {
url: '/login',
template: 'projects login',
controller: 'LoginCtrl'
}).
state('projects.business', {
url: '/business',
templateUrl:'project.html',
controller: 'ListCtrl'
}).
state('projects.menu', {
url: '/menu',
template: 'projects menu',
controller: 'ListCtrl'
}).
state('projects.community', {
url: '/community',
template: 'projects-community',
controller: 'ListCtrl'
})
});
Please describe your issue:
When  navigating through different states i want persist the data.I attached the punker for that example  let say i entered some data in the projects(project.html) page and navigate to index(index.html) page and come back to projects page the data should persist on the text box but it is refreshing and making blank.i want to cache the whole view when i coming back to that view.
If known, describe the steps to reproduce the issue:
https://plnkr.co/edit/XtfZtDPeWSuMlURZHXbS?p=preview
Please advise.

...thus proving #2454 was totally worth it.
See #262.

It's the thought that counts... :+1:

authors: DennisLfromGA dnschneid saanviboyapati

repo: dnschneid/crouton


text:
⚠ Site ANTS has degraded performance

In 421de31, Site ANTS (https://ants.gouv.fr) experienced degraded performance:

HTTP code: 200
Response time: 7289 ms

Resolved: Site ANTS performance has improved in 05b057d.

repo: dnum-mi/stats-sites-api


text:
🛑 Site telepoint is down

In 0eb16fe, Site telepoint (https://tele7.interieur.gouv.fr) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Site telepoint is back up in 8fd939b after 2 hours, 4 minutes.


text:
🛑 Site Histovec is down

In bfd291e, Site Histovec (https://histovec.interieur.gouv.fr/histovec) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Site Histovec is back up in 7692993.


text:
tensor_type argument for all DocVec deserializations

DocVec.from_protobuf(tensor_type=...) already exists, but this needs to be the case for all deserializations:

proto
json
pandas
bytes
binary
base64

Otherwise there is no way of knowing if the deserialized DocVec should use torch, np, or tf

Can you elaborate more? does it need to be added, removed, commented?

@JoanFM you need to slow down a bit, you're faster at commenting than I am at filling in the description of my own issue ;)

haha sorry, I just happened to be watching at that time

authors: JoanFM JohannesMessner

repo: docarray/docarray


text:
Can't use a volume mounted from an NFS

We are currently running a postgres 9.4 container. We map /path/to/mount from the host to /var/lib/postgres/data in the container successfully, and our postgres container is running.
For a specific environment, we need the container to use a volume which is mounted as an NFS storage.
Unfortunately, postgres does not start.
The error I'm currently getting is from gosu postgres initdb command:
gosu postgres initdb
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... initdb: could not fsync file "/var/lib/postgresql/data/base/1": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/base/12136": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/base/12141": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/base": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/global": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_clog": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_dynshmem": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_logical/mappings": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_logical/snapshots": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_logical": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_multixact/members": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_multixact/offsets": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_multixact": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_notify": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_replslot": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_serial": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_snapshots": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_stat": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_stat_tmp": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_subtrans": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_tblspc": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_twophase": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_xlog/archive_status": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_xlog": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data": Invalid argument
initdb: could not fsync file "/var/lib/postgresql/data/pg_tblspc": Invalid argument
ok

WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    postgres -D /var/lib/postgresql/data
or
    pg_ctl -D /var/lib/postgresql/data -l logfile start

System Information
docker version 1.91 host machine is running ubuntu 14.04
Steps to Reproduce

Mount the NFS with the following command:
sudo mount.cifs //my-machine/my-shared-folder /path/to/mount -o username=my-user,password=my-password,gid=1000,uid=1000,dir_mode=0700,file_mode=0700

Dockerfile:
I know that there are all kinds of issues with mounting NFS volumes.
To overcome issues, I've changed the uid and gid of postgres to 1000, which is the owner of the mounted folder in the host.

FROM postgres:9.4
RUN groupmod --gid=1000 postgres && usermod --gid=1000 --uid=1000 postgres
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["postgres"]

build the image:
docker build -t my_postgres my_postgres/

Start the container with /bin/bash so I can run and debug
docker run -it -p 5432:5432 -v /path/to/mount:/var/lib/postgresql/data -e DB_USER='my-db-user' -e DB_PASSWORD='my-db-password' -e DB_NAME='dev' -e RUN_MODE='production' my_postgres /bin/bash

Run the first commands of docker-entrypoint.sh
chown -R postgres "$PGDATA" && chmod g+s /run/postgresql && chown -R postgres:postgres /run/postgresql && gosu postgres initdb

Things I've verified

User postgres has write access from the container to the mounted volume, for example, running the command:  gosu postgres echo "hello" > /var/lib/postgres/data/test.txt` creates a file. also the initdb command succeeds in writing file to this location.

Reading and trying various things suggested in these related issues: https://github.com/docker/docker/issues/4213, https://github.com/docker/docker/issues/2259,  https://github.com/docker/docker/issues/7198

Sorry for the long tail :disappointed:
My guess is that NFS probably does not support some filesystem requirement that PostgreSQL server requires which is why fsync is returning "invalid argument".
Possibly related issues/threads:

https://www.postgresql.org/message-id/CABUevEzRd6TT89dx5AG_ceLgrvg7KTpCFmM8%3DiKvVL-HRHoM6w%40mail.gmail.com
https://forums.docker.com/t/trying-to-get-postgres-to-work-on-persistent-windows-mount-two-issues/12456
https://github.com/docker/for-win/issues/445
https://www.postgresql.org/docs/9.6/static/creating-cluster.html#CREATING-CLUSTER-NFS

Closing given that this is a PostgreSQL limitation, not something we can really solve in the image itself.

@liricooli Did you fix the issue, i also got the problem when deploying postgres with the NFS volume. It stuck on performing post-bootstrap initialization

@Itfly nope, sorry. we just worked-around the issue by not using postgres on NFS. also, I didn't spend more time on this issue as I switched jobs.

@liricooli  Thanks.

Any update with this issue guys?

@wlisesrivas https://github.com/docker-library/postgres/issues/213#issuecomment-329908750 :wink:

authors: Itfly liricooli tianon wlisesrivas yosifkit

repo: docker-library/postgres


text:
Error from V2 registry: EOF

I have a registry:2.0.1 running on a docker-machine host (provider=virtualbox with port forwarding).
On a remote host, When I run docker pull <remote_host>:5000/<my_image> I got the following output:
latest: Pulling from <remote_host>:5000/<my_image>
8ebbde154a62: Pulling fs layer
c12da83ee7c1: Pulling fs layer
a16207605ac6: Pulling fs layer
ea994adb8f5e: Pulling fs layer
b9a671861279: Pulling fs layer
a58de78a6a7f: Pulling fs layer
12724ce38086: Pulling fs layer
6bafb24bd6c1: Pulling fs layer
4d217892bcfe: Pulling fs layer
b741ec2037b8: Pulling fs layer
983765c3d97c: Pulling fs layer
96f8e8f069ed: Pulling fs layer
Pulling repository <remote_host>:5000/<my_image>
FATA[0000] Error: image <my_image> not found.

I checked the docker daemon log and found the following error message:
level=error msg="Error from V2 registry: Get http://<remote_host>:5000/v2/<my_image>/blobs/sha256:59ea07ffd330c17c154ca58866abd061a2a661cfe56fef997c5525e4d4ff50eb: EOF"
This is the output of my docker info:
Containers: 0
Images: 0
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 4.0.3-boot2docker
Operating System: Boot2Docker 1.6.2 (TCL 5.4); master : 4534e65 - Wed May 13 21:24:28 UTC 2015
CPUs: 1
Total Memory: 998.1 MiB
Name: lab
ID: SQBO:NVC7:L2JS:4DQF:JFW3:SDLS:MXZM:UYHL:DD3X:D4ZE:22Q5:XRH2
Debug mode (server): true
Debug mode (client): false
Fds: 266
Goroutines: 422
System Time: Wed Jun 17 05:41:57 UTC 2015
EventsListeners: 0
Init SHA1: 7f9c6798b022e64f04d2aff8c75cbf38a2779493
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Registry: [https://index.docker.io/v1/]
Labels:
 provider=virtualbox

@rafabene thanks for reporting this.  We need a bit more information to debug this.
What is your registry storage backend?  S3/filesystem etc
Can you get the logs from the registry around the time the error happened and paste them here.

I'm using filesystem. (default config file from registry:2.0.1).
Since the log (at the time of the error) is huge, I paste it to here: http://pastebin.com/8dCtBUps

There is nothing in that log file that helps diagnose the problem.  I cannot see a reference to the digest from your error message, nor any lines relating to error conditions.
When the registry fails on an operation it prints a line with error details (err.code err.detail etc).  Can you look for those and paste them here?

@rafabene Does this happen every time on the same image?

It happen's every time. and not on the same image. It doesn't happen locally. I can't find any error message in the registry's log but just on docker daemon's log. I was able once or twice to pull a image for the first time and then it started to fail on next images.
I'll try again now and paste the whole logs

I pasted it to gist because it provides separated files information. I hope this helps:
https://gist.github.com/rafabene/969b087f41f7c5081dcb

I tried with registry 0.9.1 and the issue doesn't happen.

@rafabene Are you sure the image is being pushed to the same registry instance? The daemon 404's on pulling "sha256:fdd04e6fcc43523cb5d1f8f09467d7b30925b7a2c06e86956fe0ba85a9dced6b" causing the pull to fallback to v1. From the registry logs and the daemon logs, this digest was never pushed or pulled.
How are you running the registry? Have you setup persistent storage?

Indeed! That's really odd. I'm really sure the there's just a single registry instance running. My env at this moment is a isolated network with both hosts only (Titanium resolved as classroom.example.com and Colosus). The thing to consider is that I see the [registry log running] (https://gist.github.com/rafabene/969b087f41f7c5081dcb#file-registry-log) when I execute the docker pull from remote host.
I'm running it through the following docker-compose.yml
registryv2:
  image: registry:2.0.1
  ports:
    - "5000:5000"
  environment:
    - REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY=/var/lib/registry
    - REGISTRY_LOG_LEVEL=debug
  volumes:
    - /srv/docker/registry:/var/lib/registry
  restart: always
<other docker containers removed>
```

@rafabene The next step is to start poking around the backend and confirming the existence of the blobs. The path layout, which should be rooted at /srv/docker/registry with your configuration, is documented in registry/storage/paths.go. Check the primary blob store and see if you can confirm existence of the suspect digest. If it is there, validate the "data" file with a sha tool (sha256sum on most linux systems).

Yeap.
The data file is there and the checksum matches:
sha256sum data
fdd04e6fcc43523cb5d1f8f09467d7b30925b7a2c06e86956fe0ba85a9dced6b  data

The funny part is that I can't see any reference to those blobs when pulling or pushing:
..v2/blobs/sha256$ ls
0b/ 46/ 63/ 6b/ 6e/ 8a/ 9f/ a3/ d9/ ef/ fd/

@rafabene V1 image IDs are not hashes. We should be seeing some the hashes in the registry storage in the logs.
Can you download https://<reghost>/v2/mysql/blobs/sha256:fdd04e6fcc43523cb5d1f8f09467d7b30925b7a2c06e86956fe0ba85a9dced6b directly with curl? (note you may need to disable or provide some authentication)

It worked through direct download.
[rafael@Colosus ~]$ docker pull classroom.example.com:5000/mysql
latest: Pulling from classroom.example.com:5000/mysql
104de4492b99: Pulling fs layer
065218d54d7d: Pulling fs layer
9525a21bc96b: Pulling fs layer
88b078de9380: Pulling fs layer
857e8088dd20: Pulling fs layer
02a18be0147e: Pulling fs layer
02a18be0147e: Download complete
7c0571f90444: Download complete
8709a819fa6b: Download complete
86a40114dce2: Pulling fs layer
86a40114dce2: Download complete
d7f9d4cf73af: Download complete
b2f80b2caf89: Download complete
3ce454bd8801: Download complete
569ac853d7f5: Download complete
d63d4723d715: Download complete
FATA[0000] Error: image mysql:latest not found
[rafael@Colosus ~]$ wget http://classroom.example.com:5000/v2/mysql/blobs/sha256:fdd04e6fcc43523cb5d1f8f09467d7b30925b7a2c06e86956fe0ba85a9dced6b
--2015-06-19 19:47:46--  http://classroom.example.com:5000/v2/mysql/blobs/sha256:fdd04e6fcc43523cb5d1f8f09467d7b30925b7a2c06e86956fe0ba85a9dced6b
Resolving classroom.example.com (classroom.example.com)... 192.168.0.11
Connecting to classroom.example.com (classroom.example.com)|192.168.0.11|:5000... connected.
HTTP request sent, awaiting response... 200 OK
Length: 37211143 (35M) [application/x-gzip]
Saving to: 'sha256:fdd04e6fcc43523cb5d1f8f09467d7b30925b7a2c06e86956fe0ba85a9dced6b.2'
sha256:fdd04e6fcc43523cb5d1f8f09467 100%[==================================================================>]  35.49M  3.28MB/s   in 11s
2015-06-19 19:47:57 (3.24 MB/s) - 'sha256:fdd04e6fcc43523cb5d1f8f09467d7b30925b7a2c06e86956fe0ba85a9dced6b.2' saved [37211143/37211143]
Now I'm updating my env to bootdocker/docker-machine/docker to 1.7 and see what happens.

With Docker 1.7 the error persists but a more detailed error message is presented at docker daemon log:  https://gist.github.com/rafabene/64296e23cfd8ab572587

@rafabene Ah, now we've got some information. This is a problem we've been hunting down for some time in the daemon. Please file a bug in docker/docker and reference this issue. We will close this one when we find a workaround.
Are you running the registry behind a proxy, such as nginx?

I tried to use nginx but it changed the expected result, so I'm not using nginx any more.

@rafabene Ok, so we've eliminated the possibility of a proxy timeout. Let's continue troubleshooting on the docker side.
I am going to close this issue. If that doesn't work for you, please let me know.

authors: RichardScothern rafabene stevvooe

repo: docker/distribution


text:
Running security bench: Cannot reach trust server at this time

I am running the security bench per the documentation for Ubuntu
docker run -it --net host --pid host --userns host --cap-add audit_control \
    -e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
    -v /etc:/etc:ro \
    -v /lib/systemd/system:/lib/systemd/system:ro \
    -v /usr/bin/containerd:/usr/bin/containerd:ro \
    -v /usr/bin/runc:/usr/bin/runc:ro \
    -v /usr/lib/systemd:/usr/lib/systemd:ro \
    -v /var/lib:/var/lib:ro \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    --label docker_bench_security \
    docker/docker-bench-security

I get this output:
docker: unable to reach trust server at this time: 500.
See 'docker run --help'.

Is there something I am missing? Do I need to setup the DTR per here
I'd truly appreciate any help or suggestions or guidance.

Hi @rirlswift, are you running the officail Docker image, or have you built your own?
If you're using the official image, could you please try running the script on the host or build your own image?

Hey  konstruktoid,
Thanks for helping me get this to work. I am now running the script on the host via
git clone https://github.com/docker/docker-bench-security.git
cd docker-bench-security
sudo sh docker-bench-security.sh

Looks great, even on CE.

ref https://github.com/docker/docker-bench-security/issues/405

authors: konstruktoid rirlswift

repo: docker/docker-bench-security


text:
Section "Docker for Mac and Docker for Windows" needs clarification

Problem description
The sentence "Modifying the storage driver on these platforms is not supported" can be confusing.
Are we talking about the commercial support or the technical ability to change the storage driver?
Problem location
At the section Docker for Mac and Docker for Windows

Docker for Mac and Docker for Windows are intended for development, rather than production. Modifying the storage driver on these platforms is not supported.

Project version(s) affected
Docker for Mac and Docker for Windows
Suggestions for a fix
Make some clarification.

What it means is that it is not possible. :)

@mistyhacks, if it means that it's not possible, then the statement is wrong according to this discussion.
To be sure, I just tested and I have been able to change from overlay2 to aufs and from aufs to overlay2, using Server Version: 17.09.0-ce.

OK, even if it is technically possible, we do not test it at all. There should be no reason to adjust the storage driver on Docker for Mac or Docker for Windows, as the storage driver is a deployment-time concern, and Docker for Mac and Docker for Windows are for local development only.
cc/ @dsheets who I remember talking to about this before.

authors: Kwadz mistyhacks


text:
Add info about backing up swarms

Proposed changes
Add more explicit info about backup and recovery as outlined in #1072
Some clean-up and reorganization along the way
Fixes #1072

Addressed feedback so far. @cyli @aaronlehmann PTAL and let me know if more changes are needed, or if we need to go over anything f2f.

Addressed the new (awesome) feedback. I think this is looking good. I talked about the need to rotate the unlock key after the restoration too.

Note to myself. Read this when merged.

LGTM

LGTM!  Thank you for adding this and for the super quick response times!

authors: aaronlehmann cyli mstanleyjones pascalandy


text:
Correct Debian for ARMHF installation steps

Proposed changes
My goal is to correct the installation instruction for Debian. The current document mixes up Debian for ARM and Raspbian.
Related issues (optional)
Fixes #2098
I was not entirely satisfied by the changes in the PR which fixed the issue. So here is my first contribution which clarifies the Debian part of this document.

I meant convenience script

Hi @NonCreature0714
I have two Raspberry Pi 2, a Raspbian and Ubuntu variant.
For both, I have been installing the Debian armhf packages. And it works fine.
I could try the convenience script on Ubuntu armhf. Do you think it makes sense? Sadly, I do not have a free SD card at the moment to try and test installing Docker with the convenience script on Debian armhf.

@andrewhsu PTAL

authors: NonCreature0714 jcberthon mstanleyjones


text:
Update content_trust.md

Proposed changes

Unreleased project version (optional)

Related issues (optional)

Deploy preview for docsdocker ready!
Built with commit af6ad4fd9959644e6c1546b42db74c694a00b50c
https://deploy-preview-8489--docsdocker.netlify.com

authors: GordonTheTurtle scjane


text:
Add terms for cluster and persistent storage

Proposed changes
Add terms for cluster and persistent storage
Related issues (optional)
Jira: https://docker.atlassian.net/browse/ENGDOCS-52
GitHub: https://github.com/docker/docker.github.io/issues/7691

@joaofnfernandes PTAL. This is related to the Jira issue that you reported here: https://docker.atlassian.net/browse/ENGDOCS-52

Deploy preview for docsdocker ready!
Built with commit acd7ffb37a94666e78bc99a83d4162cc73ffba1c
https://deploy-preview-9294--docsdocker.netlify.com

Deploy preview for docsdocker ready!
Built with commit 6819fac1eee58a021648a672f14d8c61989608f8
https://deploy-preview-9294--docsdocker.netlify.com

I lost the changes on this branch while trying to reset, so I am closing this PR in favor of https://github.com/docker/docker.github.io/pull/9333.

authors: GordonTheTurtle traci-morrison


text:
Docker 1.7.0-rc1 did not compile on ARM

Trying to compile Docker-1.7.0-rc1 on an ARMv7l machine (Scaleway Cloud), I get the following error:
---> Making bundle: dynbinary (in bundles/1.7.0-rc1/dynbinary)
# github.com/vishvananda/netns
vendor/src/github.com/vishvananda/netns/netns_linux_arm.go:4: SYS_SETNS redeclared in this block
	previous declaration at vendor/src/github.com/vishvananda/netns/netns_linux_amd.go:4

Seems to be that the dependency github.com/vishvananda/netns breaks it.
Workaround:
Removing ./vendor/src/github.com/vishvananda/netns/netns_linux_amd.go gets me around this error. Now it's compiling successfully.

Hi!
If you are reporting a new issue, PLEASE make sure that does not have any duplicates already open.
We would like to take this time to remind you of the information we need to debug the problem you are seeing. This is an automated response so if this ticket is not about a bug, do not fret.
If you fail to provide this information within 7 days, we will close this because we cannot debug your issue. We can reopen whenever the information is provided.
Thank you.
Please see:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues
Description of problem:

`docker version`:

`docker info`:

`uname -a`:

Environment details (AWS, VirtualBox, physical, etc.):

How reproducible:

Steps to Reproduce:
1.
2.
3.

Actual Results:

Expected Results:

Additional info:

#ENEEDMOREINFO

Hi, you can add // +build linux,amd to vendor/src/github.com/vishvananda/netns/netns_linux_amd.go to work around this, the latest netns has fixed this.

@hqhq, thanks for this info. Now I know, we're just waiting to include support for arm64 to fix this issue as well! Good to know.

I'm going to close this issue, because this looks to be an issue in a vendored package, not in docker. I suggest to open an issue in the repo for that package.
Feel free to continue the discussion after closing

I think we should update netns dependency if we care about building on arm

Oops, I overlooked this;

the latest netns has fixed this.

@hqhq @DieterReuter feel free to create a PR to update the netns version if they have fixed this already, I'll re-open

@hqhq @thaJeztah @DieterReuter This came in via libnetwork upgrade. And there was a recent discussion around this very same topic. @dave-tucker & @hqhq were involved working on the netns library. They might have more info to share. In short, this update must happen in libnetwork and must get into the 1.7.0 bump branch through vendor-in process.

@mavenugo ah! Thanks for the reminder, I though I've seen a similar discussion, but couldn't find it, because I was looking in the wrong repo.
Here's what I'll do; I'll keep this issue open here, if one of you can update the vendored package in libnetwork, then bump the version here; please mention a closes #13534
If this is needed in the 1.7 release; we're in feature-freeze, but perhaps it could be cherry-picked if needed.

@thaJeztah We are planning on a vendor-in of libnetwork shortly in the 1.7.0 bump branch shortly with selected bug-fixes that were marked for the 1.7.0 release. We can accomodate this netns library upgrade once @dave-tucker and @hqhq solve the issues around this.

@mavenugo thanks for the update, please remind me to close this after that, or feel free to close now if you think that's appropriate :smile:

@thaJeztah @mavenugo there is no issue with armv6/armv7 in the updated netns. The patch I'm working on with @hqhq it to address armv8 and I'm currently blocked on getting some hardware (or working qemu) to recreate the issue

@dave-tucker Perhaps @DieterReuter can help providing access to the hardware? I might still have my scaleway test account active, but it may be worth a try contacting them directly (as it's in their interest too)

Oh, realize I might have misread your comment; Scaleway is on Armv7 I think?

Yep :) armv7 and armv6 should be fine if we just bump the netns dependency. The issue Madhu mentioned was related to armv8 and that's where I'm currently blocked ;)

@dave-tucker, you're absolute right, armv6l and armv7l is just fine with new version of netns.
@thaJeztah, ScaleWay is armv7l.

I just tested again with 1.7.0-rc4, this issue is fixed now.

Thanks for reporting back, @DieterReuter! And good to hear :)

Issue fixed by #13784.

authors: DieterReuter GordonTheTurtle dave-tucker hqhq mavenugo runcom thaJeztah

repo: docker/docker


text:
Potential docker service "--dns" flag for custom service discovery resolution?

For internal network applications I like to be able to namespace internal services with DNS (such as front-end.project.service). Of course due to the limitations of service names (namely that they can't contain .s) this ends up either being impossible or inelegant, such as using hyphens instead.
I'm sure there are other use cases where the option to specify a custom DNS address would be useful. I have searched the issues and docs so my apologises if this has been suggested before, looking forward to your thoughts!

I think what you're looking for is --network-alias (which currently isn't available for services), there's an open issue for that here: https://github.com/docker/docker/issues/24787
Basically, similar to docker run --network=my-network --network-alias=foo.bar.com --name foo ....
I'll close this issue, because I think this is a duplicate of https://github.com/docker/docker/issues/24787, but feel free to comment if you think I closed this by mistake. Thanks for reporting!

Ah wonderful, thank you very much this is exactly what I was looking for!


text:
docker images reports same image in differing order

Running sudo docker images results in an unpredictable and changing listing order for the multiply tagged docker image.
Output of docker version:
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:00:36 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:00:36 2016
 OS/Arch:      linux/amd64

Output of docker info:
Containers: 25
 Running: 1
 Paused: 0
 Stopped: 24
Images: 80
Server Version: 1.12.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 177
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-92-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.62 GiB
Name: myhost
ID: WS4V:2MWJ:IWFT:AV5C:74U2:PPJB:G36B:5E3N:E3NR:OAW6:EERM:CKNW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
physical
Steps to reproduce the issue:
1. create an empty directory for context
mkdir /tmp/my-docker

2. create file /tmp/my-docker/Dockerfile with contents
FROM ubuntu:trusty
# add a layer for sake of example
RUN apt-get update && apt-get install -y apt-utils

3. build a new docker instance with multiple tags
sudo docker build -t tag1 -t tag2 .

4. watch the output of docker images
sudo -- watch -d -- docker images

Describe the results you received:
Running sudo docker images results in an unpredictable and changing listing order for the same docker image.  Sometimes the output was
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
tag1                            latest              e8cb1537535d        3 minutes ago       210.1 MB
tag2                            latest              e8cb1537535d        3 minutes ago       210.1 MB

then other times
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
tag2                            latest              e8cb1537535d        3 minutes ago       210.1 MB
tag1                            latest              e8cb1537535d        3 minutes ago       210.1 MB

Describe the results you expected:
Running sudo docker images results in a predictable and consistent listing order.  By Created time and then tagging order and then ...
Additional information you deem important (e.g. issue happens only occasionally):
This did not happen with the prior version of docker which I believe was version 1.10.

Actually this is not only about images command, if we look at the stats command, if has similar problem.
For example, if you run docker stats -a, the output order looks consistent to docker ps -a results. Then if you start a new container, the new container will be appended at tail of the output list, it's not consistent to docker stats -a results any more.
I am working on a new flag "--sort-by" for images and ps, but I hope I can expand it to other commands.


text:
Failed to pull docker image on RHEL 7 with docker 1.13

Description
Failed to pull docker image on RHEL 7 after upgrading to 1.13.rc3
Environment
Red Hat Enterprise Linux Server release 7.2 (Maipo)
docker version:
Client:
 Version:      1.13.0-rc3
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   4d92237
 Built:
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.0-rc3
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   4d92237
 Built:
 OS/Arch:      linux/amd64
 Experimental: false

Docker service configuration:
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target firewalld.service

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --log-opt max-size=50m  --insecure-registry ourregistry:5000 --label=role=app --storage-driver=devicemapper --storage-opt dm.datadev=/dev/ttt/docker-data --storage-opt dm.metadatadev=/dev/ttt/docker-metadata  --storage-opt dm.use_deferred_removal=true --storage-opt dm.basesize=30G
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process

[Install]
WantedBy=multi-user.target

Filesystem:
lsblk
NAME                           MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda                           202:0    0   20G  0 disk
|-xvda1                        202:1    0    1M  0 part
`-xvda2                        202:2    0   20G  0 part /
xvdf                           202:80   0  300G  0 disk
|-ttt-docker--data        253:0    0  200G  0 lvm
| `-docker-202:2-17191269-pool 253:3    0  200G  0 dm
|-ttt-docker--metadata    253:1    0    3G  0 lvm
| `-docker-202:2-17191269-pool 253:3    0  200G  0 dm
`-ttt-data                253:2    0   93G  0 lvm  /Data

Commands I run:
docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
c04b14da8d14: Pull complete
Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Status: Downloaded newer image for hello-world:latest

Works fine. But then trying for a bigger image:
docker pull portainer/portainer:1.10.2
1.10.2: Pulling from portainer/portainer
a3ed95caeb02: Already exists
802d894958a2: Extracting [==================================================>] 143.3 kB/143.3 kB
c6e381376096: Download complete
failed to register layer: devmapper: Unknown device 262ad7c056621dda8e3bd916da56eb213e9e2cfaebbab802bcab3035734bbb71

What can be the problem?

Could you add the output of docker info ? Is it possible device mapper ran out of space?

docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 14
Server Version: 1.13.0-rc3
Storage Driver: devicemapper
 Pool Name: docker-202:2-17191269-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 32.21 GB
 Backing Filesystem: xfs
 Data file: /dev/thetaray/docker-data
 Metadata file: /dev/thetaray/docker-metadata
 Data Space Used: 12.32 GB
 Data Space Total: 214.7 GB
 Data Space Available: 202.4 GB
 Metadata Space Used: 17.56 MB
 Metadata Space Total: 3.221 GB
 Metadata Space Available: 3.204 GB
 Thin Pool Minimum Free Space: 21.47 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: wu51b5zyqyv41vgw55jhg52s4
 Is Manager: true
 ClusterID: yb522mycxt13m2enefikncdvs
 Managers: 1
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 172.16.1.75
 Manager Addresses:
  172.16.1.75:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 51371867a01c467f08af739783b8beafc154c4d7
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-327.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 14.28 GiB
Name: ip-172-16-1-75.us-west-2.compute.internal
ID: RLGZ:DDZ5:S7SG:APF4:BM2A:PTQN:OAEY:FX34:JSVA:MQVN:2UFD:UR2E
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Labels:
 role=app
Experimental: false
Insecure Registries:
 ourregistry:5000
 127.0.0.0/8
Live Restore Enabled: false

ping @rhvgoyal @tonistiigi any ideas?

Unknown device message comes from lookupDevice() in devicemapper.
That means a file with details to this device is not there. I suspect that somehow that device has been removed in the past.
Looking at messages
a3ed95caeb02: Already exists
I suspect this device might be belonging to this layer. You can do docker inspect a3ed95caeb02 and see what's the storage device associated with it.
Try removing image layer image a3ed95caeb02 and pull again and see if it works.

There is no such a layer..

@thaJeztah  If there is no such layer and docker pull says layer already exists, I think there is some bug in generic layer code and its not a graph driver issue.

btw, everything worked fine with 1.12.x on the same machine, so it looks like a bug in 1.13

Any workaround you can suggest?

@rhvgoyal You can't inspect a layer or even more, a hash of a compressed distribution layer blob that a3ed95caeb02 is.

@tonistiigi not sure why we will not be able to inspect a3ed95caeb02. docker pull says this layer is already present and if that's the case we should be able to inspect it.
If its associated graph driver Id (I guess calculated using hash of contents), is missing, then I agree inspect will fail as graph driver will refuse to return data associated with it.
But as I understand, core of the problem seems to be that docker images -a does not list the image. Is that right @liyaka . If that's the not the problem, then issue is how graph driver data got deleted while image data is still around.

@rhvgoyal a3ed95caeb02 is a hash of a blob not an image. Even if it was a layer(the ones you see in docker inspect --format '{{.RootFS}}') it's not an image and can't be inspected.
Most likely thing that is happening here is that after the upgrade devicemapper's backing storage got switched out. Afaik docker does not detect any changes to --storage-opt and metadata about existing layers is only stored by the driver name. So docker expects layers to exist because it has metadata about it but when they are looked up from the driver they appear to be missing. Other way how this would show is that old images pulled with v1.12 would show up in docker images but when you create a container from them you would get the same device lookup error. If this is the case and switch to a fresh devicemapper devices was intentional then the docker storage dir /var/lib/docker should be cleared as well.

@tonistiigi, ok, it is hash of blob and not image id. did not know it.
Iven if data/metadata device was switched out, this error should not have happened. It can't find the file with same name in /var/lib/docker/devicemapper/... dir.
Also if data/metadata device was switched out, devicemapper will detect it. When it activates thin pool, it queries the UUID of base device file system and matches it with the stored UUID. If it does not match, docker fails saying wrong thin pool. That's the mechanism we introduced to catch these kind of errors.

@rhvgoyal If I do
> dockerd -s devicemapper --storage-opt dm.override_udev_sync_check=true -D 2>/daemon.log &
> docker pull busybox
> kill %1 && wait %1
> rm -rf /var/lib/docker/devicemapper/
> dockerd -s devicemapper --storage-opt dm.override_udev_sync_check=true -D 2>/daemon.log &
> docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
busybox             latest              e02e811dd08f        2 months ago        1.09 MB
> docker run busybox ls
docker: Error response from daemon: devmapper: Unknown device 35ab199fb7ac37e94ee072de54e6a41562fcca6f6f5c2a4e554699e0b2757eed.
See 'docker run --help'.

@rhvgoyal For the "metadata" I meant the data docker keeps about layers /var/lib/docker/image/devicemapper/ etc. not the devicemapper's own metadata.

@tonistiigi ok, got it. I was referring to switching thin pool data or metadata device. Agreed, if you do something with /var/lib/docker, then things will go wrong for sure.

I got the same problem with clean docker 1.13 rc3 install on RHEL 7.2, so it doesn't seem to be an upgrade issue..
@rhvgoyal image is indeed not listed

@liyaka Does it show a3ed95caeb02: Already exists for a fresh install on first pull?

@tonistiigi not on the first pull

I found a bug in my automatic installation - thank to your questions!
Please close this issue

authors: liyaka rhvgoyal thaJeztah tonistiigi


text:
Update Centos links

Signed-off-by: Sabin Basyal sabin.basyal@gmail.com
Centos links were not pointing to right section, but were pointing to the same page.

LGTM

authors: moxiegirl sbasyal


text:
Add test coverage for pkg/tarsum

Add some coverage on pkg/tarsum.
# From
+ go test -test.timeout=30m github.com/docker/docker/pkg/tarsum
PASS
coverage: 70.0% of statements
# To
+ go test -test.timeout=30m github.com/docker/docker/pkg/tarsum
PASS
coverage: 90.3% of statements

And removing unused code. I was also wondering about the following :

tarsum.go :
NewTarSumHash could be non exported (for now)
NewTarSumForLabel is never used, except for the tests
fileinfosums.go:
SortByPos is never used, except for the tests
versionning.go:
GetVersions is never used, expect for the tests

Signed-off-by: Vincent Demeester vincent@sbr.pm

ping @vbatts

thanks. ./pkg/tarsum will slowly get phased out, but these tests are fine.
As for your questions, usually the libraries in ./pkg/... API are a bit fixed, because we allow for outside tooling to use them. So, while those may not be used in docker's source, we'll leave them.

overall, LGTM

LGTM thanks!

I'll wait for the couple of questions and suggestions before merging.

hum DockerSuite.TestCliStatsNoStreamGetCpu seems to fail randomly :( (failed on janky here, on windows there https://github.com/docker/docker/pull/13501)

thanks @vdemeester

authors: duglin jfrazelle vbatts vdemeester


text:
Fix ignore -q flag in docker ps when there is a default format.

Docker ps default format should not take precedence over cli flags.
This happens effectively for other flags except -q.
We need to let the cli to set the format as table to print the
expected output with -q.
Signed-off-by: David Calavera david.calavera@gmail.com

LGTM

LGTM

LGTM
Ignoring Window's issues :-(

Cherry picked into #15809

authors: calavera cpuguy83 duglin tonistiigi