text:
How to interpret the content of model after dump

After dump i notice that all the conditions to split nodes is feature value less than 2. while all the feature values in my feature vectors are all zero or one. I do not know why because all the values are less than 2. But the predictions seem to be right.
Did i mistake the meaning of the model content
The first tree of model after dump is as follow:
booster[0]:
0:[f1750<2] yes=1,no=2,missing=2
1:[f1601<2] yes=3,no=4,missing=4
3:[f82919<2] yes=7,no=8,missing=8
7:[f17<2] yes=15,no=16,missing=16
15:leaf=0.986014
16:leaf=-0.902439
8:[f70286<2] yes=17,no=18,missing=18
17:leaf=0.259962
18:leaf=-0.875333
4:[f1650<2] yes=9,no=10,missing=10
9:[f1251<2] yes=19,no=20,missing=20
19:leaf=0.504145
20:leaf=0.743942
10:[f1549<2] yes=21,no=22,missing=22
21:leaf=0.853494
22:leaf=0.241296
2:[f1601<2] yes=5,no=6,missing=6
5:[f80324<2] yes=11,no=12,missing=12
11:[f51410<2] yes=23,no=24,missing=24
23:leaf=-0.2
24:leaf=0.241573
12:[f1782<2] yes=25,no=26,missing=26
25:leaf=-0.531034
26:leaf=-0.983747
6:[f1701<2] yes=13,no=14,missing=14
13:[f70574<2] yes=27,no=28,missing=28
27:leaf=0.145435
28:leaf=-0.757687
14:[f1702<2] yes=29,no=30,missing=30
29:leaf=-0.663374
30:leaf=-0.395844

Besides my feature vector is sparse and only the the feature position with 1 value stored

repo: dmlc/xgboost


text:
Convert trees to a specific format

Trying to enable the inTrees package for the xgboost R package, for rule extraction and analysis.
My question is that, how to convert an xgboost tree to the following format (which randomForest uses)? thanks!

library(randomForest)
rf <- randomForest(Species ~ ., data=iris)
getTree(rf, 1, labelVar=TRUE)
node   left daughter right daughter    split var split point status prediction
1              2              3 Petal.Length        4.75      1
2              4              5 Sepal.Length        5.45      1
3              6              7  Sepal.Width        3.15      1
4              8              9  Petal.Width        0.80      1
5             10             11  Sepal.Width        3.60      1
6              0              0                 0.00     -1  virginica
7             12             13  Petal.Width        1.90      1
8              0              0                 0.00     -1     setosa
9             14             15  Petal.Width        1.55      1
10             0              0                 0.00     -1 versicolor
11             0              0                 0.00     -1     setosa
12            16             17 Petal.Length        5.40      1
13             0              0                 0.00     -1  virginica
14             0              0                 0.00     -1 versicolor
15             0              0                 0.00     -1  virginica
16             0              0                 0.00     -1 versicolor
17             0              0                 0.00     -1  virginica

You can execute xgb_model.clf._Booster.get_dump() to get a string representation of all trees in the ensemble. Unfortunately, xgb inherently works with leaf values instead of class probabilities at each leaf, thus you will need to convert those leaf values to probabilities in some hacky way (https://github.com/dmlc/xgboost/issues/1746#issuecomment-290130695)
I once wrote code that converts this output from R randomForest to my own decision tree object (https://github.com/IBCNServices/GENESIM/blob/master/constructors/inTrees.py). Lately, I did the same for the xgb trees.
`
from sklearn.cross_validation import StratifiedKFold
from xgboost import plot_tree
import matplotlib.pyplot as plt
from constructors.ensemble import XGBClassification
from data.load_all_datasets import load_all_datasets
from sortedcontainers import SortedList, SortedSet
import numpy as np
from collections import Counter
import time
from decisiontree import DecisionTree
def parse_xgb_tree_string(tree_string, training_data, feature_cols, label_col, the_class):
# Get class distribution
_classes = np.unique(training_data[label_col].values)
class_distributions = {}
for _class in _classes:
data = training_data[training_data[label_col] != _class]
class_counts = Counter(data[label_col].values)
total = sum(class_counts.values(), 0.0)
for key in class_counts:
class_counts[key] /= total
class_distributions[_class] = class_counts
# Get the unique values per feature
unique_values_per_feature = {}
for feature_col in feature_cols:
    # Just use a simple sorted list of np.unique (faster than SortedList or a set to get index of elements after testing)
    unique_values_per_feature[feature_col] = sorted(np.unique(training_data[feature_col].values))

return parse_xgb_tree(tree_string, _class=the_class, class_distributions=class_distributions,
                      unique_values_per_feature=unique_values_per_feature, n_samples=len(training_data))

def get_closest_value(x, values):
for i in range(len(values)-1):
if values[i+1] > x:
return float(values[i])
return x
def parse_xgb_tree(tree_string, _class=0, class_distributions={}, unique_values_per_feature={}, n_samples=0):
# There is some magic involved! The leaf values need to be converted to class distributions somehow!
# For binary classification problems: convert to probability by calculating 1/(1+exp(-value))
# For multi_class: the tree_string contains n_estimators * n_classes decision trees
# WARNING: Classes are sorted according to output of np.unique
# Ordered as follows [tree_1-class_1, ..., tree_1-class_k, tree_2-class_1, ....]
# The problem is: tree_i is different for each class...
# One possibility is to assign the probability to that class by calculating logistic function
# And dividing the rest of the probability (sum to 1) according to the distribution of the remaining classes

# Next problem is: everything is expressed as "feature < threshold" instead "feature <= threshold"
# Solution is: take the infimum of those feature values

decision_trees = {}
# Binary classification
binary_classification = len(class_distributions.keys()) == 2
for line in tree_string.split('\n'):
    if line != '':
        _id, rest = line.split(':')
        _id = _id.lstrip()
        if rest[:4] != 'leaf':
            feature = rest.split('<')[0][1:]
            highest_lower_threshold = get_closest_value(float(rest.split('<')[1].split(']')[0]),
                                                        unique_values_per_feature[feature])
            decision_trees[_id] = DecisionTree(right=None, left=None,
                                               label=feature, value=highest_lower_threshold,
                                               parent=None)
        else:
            leaf_value = float(rest.split('=')[1])
            if binary_classification:
                probability = 1/(1+np.exp(leaf_value))
                other_class = class_distributions[_class].keys()[0]
                class_probs = {_class: int(n_samples*probability),
                               other_class: int(n_samples*(1-probability))}
                if probability > 0.5: most_probable_class = _class
                else: most_probable_class = other_class
            else:
                probability = 1/(1+np.exp(-leaf_value))
                class_probs = {}
                remainder_samples = int(n_samples - probability*n_samples)
                class_probs[_class] = int(probability*n_samples)
                most_probable_class, most_samples = _class, class_probs[_class]
                for other_class in class_distributions[_class]:
                    amount_samples = int(remainder_samples*class_distributions[_class][other_class])
                    class_probs[other_class] = amount_samples
                    if amount_samples > most_samples:
                        most_probable_class = other_class
                        most_samples = amount_samples

            decision_trees[_id] = DecisionTree(right=None, left=None,
                                               label=most_probable_class, value=None,
                                               parent=None)
            decision_trees[_id].class_probabilities = class_probs

# Make another pass to link the different decision trees together
for line in tree_string.split('\n'):
    if line != '':
        _id, rest = line.split(':')
        _id = _id.lstrip()
        tree = decision_trees[_id]
        if rest[:4] != 'leaf':
            rest = rest.split(']')[1].lstrip()
            links = rest.split(',')
            for link in links:
                word, link_id = link.split('=')
                if word == 'yes' or word == 'missing':
                    tree.left = decision_trees[link_id]
                else:
                    tree.right = decision_trees[link_id]

return decision_trees['0']

NR_FOLDS = 5
xgb = XGBClassification()
for dataset in load_all_datasets():
df = dataset['dataframe']
label_col = dataset['label_col']
feature_cols = dataset['feature_cols']
skf = StratifiedKFold(df[label_col], n_folds=NR_FOLDS, shuffle=True, random_state=1337)

for fold, (train_idx, test_idx) in enumerate(skf):
    print 'Fold', fold + 1, '/', NR_FOLDS, 'for dataset', dataset['name']
    train = df.iloc[train_idx, :].reset_index(drop=True)
    X_train = train.drop(label_col, axis=1)
    y_train = train[label_col]
    test = df.iloc[test_idx, :].reset_index(drop=True)
    X_test = test.drop(label_col, axis=1)
    y_test = test[label_col]

    xgb_model = xgb.construct_classifier(train, feature_cols, label_col)

    n_classes = len(np.unique(y_train.values))
    if n_classes > 2:
        for idx, tree_string in enumerate(xgb_model.clf._Booster.get_dump()):
            tree = parse_xgb_tree_string(tree_string, train, feature_cols, label_col, np.unique(y_train.values)[idx % n_classes])
            tree.visualise('xgbtree')
            plot_tree(xgb_model.clf, num_trees=idx)
            plt.show()
            raw_input()
    else:
        for idx, tree_string in enumerate(xgb_model.clf._Booster.get_dump()):
            tree = parse_xgb_tree_string(tree_string, train, feature_cols, label_col, 0)
            tree.visualise('xgbtree')
            plot_tree(xgb_model.clf, num_trees=idx)
            plt.show()
            raw_input()

`

thanks @GillesVandewiele! Will take a further look.

More than welcome! Love your work on inTrees btw! Made a python wrapper around it :)

@softwaredeng here's an example (after peeking at your GBM2List and formatGBM functions):
library(xgboost)
library(data.table)

dtrain <- xgb.DMatrix(as.matrix(iris[, -5]), label = as.numeric(iris[, "Species"]) - 1)
watchlist <- list( train = dtrain)
param <- list(max_depth = 5, eta = 0.2, nthread = 2, objective = "multi:softmax",
    num_class = 3, eval_metric = "mlogloss")
set.seed(1)
bst <- xgb.train(param, dtrain, nrounds = 10, watchlist)

xt <- xgb.model.dt.tree(feature_names = as.character(1:ncol(dtrain)), model=bst)
xt[Feature == 'Leaf', Feature := '-1']
xt[, 'split var' := as.integer(Feature)]
xt[, 'split point' := ifelse(`split var` > -1, Split, Quality)]
xt[, 'left daughter' := as.integer(tstrsplit(Yes, '-')[[2]]) + 1]
xt[, 'right daughter' := as.integer(tstrsplit(No, '-')[[2]]) + 1]
xt[, MissingNode := as.integer(tstrsplit(Missing, '-')[[2]]) + 1]
xt[, Weight := Cover]
xt[, Prediction := Quality]
xt[, Node := Node + 1]
xt[, c('ID', 'Yes', 'No', 'Split','Missing', 'Quality', 'Cover', 'Feature') := NULL]
for (f in c('left daughter', 'right daughter', 'MissingNode'))
  set(xt, which(is.na(xt[[f]])), f, -1)
xt[Tree<3]

    Tree Node split var   split point left daughter right daughter MissingNode    Weight    Prediction
 1:    0    1         3  2.450000e+00             2              3           2 66.666664  7.229677e+01
 2:    0    2        -1  2.870813e-01            -1             -1          -1 22.222221  2.870813e-01
 3:    0    3        -1 -1.466993e-01            -1             -1          -1 44.444443 -1.466993e-01
 4:    1    1         3  2.450000e+00             2              3           2 66.666664  1.807419e+01
 5:    1    2        -1 -1.435407e-01            -1             -1          -1 22.222221 -1.435407e-01
 6:    1    3         4  1.750000e+00             4              5           4 44.444443  4.190784e+01
 7:    1    4         3  4.950000e+00             6              7           6 23.999998  4.589849e+00
 8:    1    5         3  4.850000e+00             8              9           8 20.444443  6.351590e-01
 9:    1    6         4  1.550000e+00            10             11          10 21.333332  2.558654e-01
10:    1    7         4  1.550000e+00            12             13          12  2.666667  8.571428e-01
11:    1    8        -1 -5.108970e-09            -1             -1          -1  1.333333 -5.108970e-09
12:    1    9        -1 -1.425415e-01            -1             -1          -1 19.111111 -1.425415e-01
13:    1   10        -1  2.857143e-01            -1             -1          -1 19.999998  2.857143e-01
14:    1   11        -1  8.571428e-02            -1             -1          -1  1.333333  8.571428e-02
15:    1   12        -1 -8.571430e-02            -1             -1          -1  1.333333 -8.571430e-02
16:    1   13        -1  8.571428e-02            -1             -1          -1  1.333333  8.571428e-02
17:    2    1         4  1.650000e+00             2              3           2 66.666664  5.972297e+01
18:    2    2         3  4.950000e+00             4              5           4 45.333332  5.965446e+00
19:    2    3        -1  2.686567e-01            -1             -1          -1 21.333332  2.686567e-01
20:    2    4        -1 -1.465995e-01            -1             -1          -1 43.111107 -1.465995e-01
21:    2    5        -1  1.448276e-01            -1             -1          -1  2.222222  1.448276e-01

I assume you can take it from here and define the status column as you see appropriate. Keep in mind that unlike gbm, xgboost passes missing values either to the right or to the left (whichever is more appropriate according to data).

@GillesVandewiele That's great! I'll link to your project.

@khotilov nice, thank you for providing an elegant way producing the matrix!

@softwaredeng  I'm trying to hook up your InTrees with xgboost too, have you made any progress on that? or are you available to answer some questions I'm running into?

@Zephyr1213 where are you having troubles with? Extracting decision trees from XGBoost or feeding the trees to inTrees? Maybe I can help...

@Zephyr1213 @GillesVandewiele I was distracted by other stuff but it is on my agenda. Yes, let me know if there are any questions.

@GillesVandewiele  I used the codes @khotilov posted above to extract the trees from xgboost, it seems I need to edit @softwaredeng 's RF2List or GBM2List function to make everything click, right? I was trying to do that, but haven't got time to test it out yet.

@Zephyr1213  correct - create a function something like XGB2List. I have added you to the inTrees github project. feel free to push your code there, and I can take a look.
@GillesVandewiele added the python wrapper link to the project.

@GillesVandewiele Do your python wrapper support xgboost now??

@qingyuanxingsi it does :). Make sure to take a look at https://github.com/IBCNServices/GENESIM/blob/master/constructors/genesim.py#L427 etc (it's been quite a while that I actively used/developed GENESIM)
Make an issue if you have any troubles or whatsoever :).

close this issue. xgboost is now added to inTrees, should be on cran soon. https://github.com/softwaredeng/inTrees/blob/master/inTrees_1.2.tar.gz

source: gharchive/issue

authors: GillesVandewiele Zephyr1213 khotilov qingyuanxingsi softwaredeng

license_type: permissive

license_source: github-api


text:
Does Spark4j-Spark latest build support reg:pseudohubererror as objective?

Does Spark4j-Spark latest build support reg:pseudohubererror as objective?
I am trying to use XGBoost4j-Spark to build a regression model with pseudohubererror as the objective.
Environment:

AWS EMR release version 5.30.0
I am using the jars downloaded from s3://xgboost-maven-repo/release/ml/dmlc/xgboost4j_2.11/1.1.0/

xgboost4j-spark_2.11-1.1.0.jar
xgboost4j_2.11-1.1.0.jar

I launched the spark-shell on EMR master node as:
spark-shell \
    --jars hdfs:///user/lib/xgboost4j_2.11-1.1.0.jar,hdfs:///user/lib/xgboost4j-spark_2.11-1.1.0.jar,hdfs:///user/lib/MyProjectPoc-1.0.jar \
    --conf spark.dynamicAllocation.enabled=false \
    --conf spark.executors.cores=5 \
    --conf spark.task.cpus=4 \
    --conf spark.executors.memory=16G \
    --conf spark.executor.memoryOverhead=4G \
    --conf spark.driver.memoryOverhead=4G \
    --conf spark.driver.memory=16G \
    --conf spark.driver.cores=5 \
    --conf spark.executor.instances=20 \
    --conf spark.default.parallelism=200 \
    --conf spark.sql.shuffle.partitions=200 \
    --conf spark.network.timeout=800s \
    --conf spark.executor.heartbeatInterval=60s \
    --conf spark.memory.fraction=0.80 \
    --conf spark.memory.storageFraction=0.30 \
    --conf spark.yarn.scheduler.reporterThread.maxFailures=5 \
    --conf spark.storage.level=MEMORY_AND_DISK_SER \
    --conf spark.rdd.compress=true \
    --conf spark.shuffle.compress=true \
    --conf spark.shuffle.spill.compress=true

The following is the scala code to fit model (Sorry I cannot share the dataset):
val xgbParam = Map(
    "objective" -> "reg:pseudohubererror",
    "num_round" -> 1,
    "eta" -> 0.05,
    "max_depth" -> 6,
    "missing" -> MISSING_VALUE,
    "num_workers" -> 20,
    "nthread" -> 4,
    "num_early_stopping_rounds" -> 10,
    "maximize_evaluation_metrics" -> false,
    "verbosity" -> 2
)

val xgbRegressor = {
    new XGBoostRegressor(xgbParam)
        .setLabelCol(labelColName)
        .setFeaturesCol("features")
}

val xgb_huber_model = xgbRegressor.fit(trainingInput)

The following is the error message from one of the executors:
20/05/22 16:14:38 WARN TaskSetManager: Lost task 19.1 in stage 2.0 (TID 231, ip-10-0-1-165.ec2.internal, executor 10): ml.dmlc.xgboost4j.java.XGBoostError: [16:13:44] /workspace/src/objective/objective.cc:26: Unknown objective function: `reg:pseudohubererror`
Objective candidate: survival:aft
Objective candidate: binary:hinge
Objective candidate: multi:softmax
Objective candidate: multi:softprob
Objective candidate: rank:pairwise
Objective candidate: rank:ndcg
Objective candidate: rank:map
Objective candidate: reg:squarederror
Objective candidate: reg:squaredlogerror
Objective candidate: reg:logistic
Objective candidate: binary:logistic
Objective candidate: binary:logitraw
Objective candidate: reg:linear
Objective candidate: count:poisson
Objective candidate: survival:cox
Objective candidate: reg:gamma
Objective candidate: reg:tweedie

Stack trace:
  [bt] (0) /mnt/yarn/usercache/hadoop/appcache/application_1589774373197_0126/container_1589774373197_0126_01_000015/tmp/libxgboost4j1915509404376697341.so(xgboost::ObjFunction::Create(std::string const&, xgboost::GenericParameter const*)+0x85a) [0x7f47cefafb0a]
  [bt] (1) /mnt/yarn/usercache/hadoop/appcache/application_1589774373197_0126/container_1589774373197_0126_01_000015/tmp/libxgboost4j1915509404376697341.so(xgboost::LearnerConfiguration::ConfigureObjective(xgboost::LearnerTrainParam const&, std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > >*)+0x3bc) [0x7f47cef2e69c]
  [bt] (2) /mnt/yarn/usercache/hadoop/appcache/application_1589774373197_0126/container_1589774373197_0126_01_000015/tmp/libxgboost4j1915509404376697341.so(xgboost::LearnerConfiguration::Configure()+0x4c3) [0x7f47cef3af13]
  [bt] (3) /mnt/yarn/usercache/hadoop/appcache/application_1589774373197_0126/container_1589774373197_0126_01_000015/tmp/libxgboost4j1915509404376697341.so(xgboost::LearnerImpl::UpdateOneIter(int, std::shared_ptr<xgboost::DMatrix>)+0x69) [0x7f47cef22649]
  [bt] (4) /mnt/yarn/usercache/hadoop/appcache/application_1589774373197_0126/container_1589774373197_0126_01_000015/tmp/libxgboost4j1915509404376697341.so(XGBoosterUpdateOneIter+0x59) [0x7f47cee0d1b9]
  [bt] (5) [0x7f480d018427]

	at ml.dmlc.xgboost4j.java.Rabit.checkCall(Rabit.java:53)
	at ml.dmlc.xgboost4j.java.Rabit.shutdown(Rabit.java:83)
	at ml.dmlc.xgboost4j.scala.spark.XGBoost$.ml$dmlc$xgboost4j$scala$spark$XGBoost$$buildDistributedBooster(XGBoost.scala:381)
	at ml.dmlc.xgboost4j.scala.spark.XGBoost$$anonfun$trainForNonRanking$1.apply(XGBoost.scala:455)
	at ml.dmlc.xgboost4j.scala.spark.XGBoost$$anonfun$trainForNonRanking$1.apply(XGBoost.scala:450)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
	at org.apache.spark.rdd.RDD$$anonfun$7.apply(RDD.scala:359)
	at org.apache.spark.rdd.RDD$$anonfun$7.apply(RDD.scala:357)
	at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1164)
	at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1155)
	at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1090)
	at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1155)
	at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:881)
	at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:357)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:308)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:123)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

No, the pseudo Huber loss was added after the 1.1.0 release. You should use the SNAPSHOT version instead.

Thanks for the lightening fast reply.
Is there a SNAPSHOT build for Scala 2.11 that I can download?

Yes, you can use either 2.11 and 2.12. Follow instructions in https://xgboost.readthedocs.io/en/latest/jvm/index.html#access-snapshot-version.
FYI, you should consider upgrading Scala to 2.12 in the near future, since XGBoost will be moving to Spark 3.x.

Thanks
Thank you for the reminder. Since my current setup uses Spark 2.4.5 and Scala 2.11, so I have to use the build for 2.11. If my model works fine, we will probably use 2.12 build in production.

Is there a Snapshot build that has DMLC_USE_S3=1 enabled? Do I have to compile by myself?

No, the snapshot builds don’t support HDFS or S3. You should compile it yourself.

Thanks. Is there a up to date guide on how to build the snapshot? Or some sort of reference. Some guides I found online are out dated.
--
Pan

On May 29, 2020, at 15:20, Philip Hyunsu Cho notifications@github.com wrote:

No, the snapshot builds don’t support HDFS or S3. You should compile it yourself.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Take a look at https://xgboost.readthedocs.io/en/latest/jvm/index.html#installation-from-source


text:
N_jobs and nthreads not respected when training with scikit RandomizedSearchCV

I am trying to tune my parameters on a spark cluster. I want to set the number of jobs to 16, but XGBClassifier is using all available threads. Why is xgb not respecting the n_jobs/nthread parameters?
x_grid = {
        'n_estimators': xn_estimators,
        'scale_pos_weight': scale_pos_weight,
        'learning_rate': eta,
        'max_depth': max_depth,
        'min_child_weight': min_child_weight,
        'gamma': gamma,
        'subsample': subsample,
        'colsample_bytree': colsample_bytree,
        'reg_alpha': reg_alpha,
        'n_jobs': [16]
    }

model = XGBClassifier()
params = x_grid

X_train, X_test, y_train, y_test = prepro.scipre(filepath)

grid = RandomizedSearchCV(estimator=model, param_distributions=params, n_iter = 100,
            scoring='f1_weighted', n_jobs=16, cv=3, verbose=2, random_state=1)

grid_result = grid.fit(X_train, y_train)

Anyone able to reproduce?

Just ran it on master, works pretty well ...

Which version of XGBoost are you using?

I'm on 1.2.0 and just two classes. I'll try running XGBoost without scikit to see if it makes a difference

Ok the issue is that I was using grid search's n_jobs in conjunction with XGBClassifier's n_jobs. So the n_jobs were multiplicative. If I don't use grid search's n_jobs, XGB works as expected

authors: davidfstein trivialfis


text:
Fix divide by 0 in feature importance when no split is found.

Close https://github.com/dmlc/xgboost/issues/6639

@wphicks Thanks for the review!

source: gharchive/pull-request


text:
Clarify effect of enable_categorical

As a follow up from comment: https://github.com/dmlc/xgboost/pull/9828#discussion_r1423263091
This PR expands the docs about parameter enable_categorical in the R and Python interfaces so as to make it more clear what it actually is for.

@david-cortes For future reference, the XGBoost's document site is built with docs in https://github.com/dmlc/xgboost/tree/master/doc . You can view the newly built doc for PR in the readthedocs test. For this PR: https://xgboost--9877.org.readthedocs.build/en/9877/python/python_api.html#xgboost.DMatrix
I will followup with some changes for sklearn interface.

authors: david-cortes trivialfis


text:
Ignore option TypeError

I've been testing this LIB, but the ignore option is not working, it always gives this error:
TypeError: options.ignore.test is not a function

Are you passing an array of strings as the ignore option?

authors: apenasigordev dmotz

repo: dmotz/express-spa-router

license: mit

license_source: bigquery


text:
Having trouble gettings started with namespaces

Hi,
I want to thank you for taking the time to write such an awesome looking library. Unfortunately I am having a bit of trouble getting started. I am trying to build an ePUB viewer in javascript. ePUB documents contain a lot of xml manifest information and a nice standardized way for me to convert this to JSON would really help the project along. Below is an example of a manifest pulled from a real world ePUB
<?xml version="1.0" encoding="UTF-8"?>
<package xmlns="http://www.idpf.org/2007/opf" unique-identifier="ean" version="2.0">
  <metadata xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns="http://www.idpf.org/2007/opf">
      <dc:title>L'espagnol dans votre poche</dc:title>
      <dc:creator></dc:creator>
      <dc:publisher>Larousse</dc:publisher>
      <dc:rights>© Éditions Larousse, </dc:rights>
      <dc:identifier id="ean">9782035862464</dc:identifier>
      <dc:language>fr</dc:language>
  </metadata>
  <manifest>
    <item id="css" href="styles.css" media-type="text/css" />
    <item id="ncx" href="toc.ncx" media-type="application/x-dtbncx+xml" />
    <item id="cover" href="images/cover.jpg" media-type="image/jpeg"/>
    <item id="font2" href="Fonts/HighlanderStd-Medium.otf" media-type="application/x-font-otf"/>
    <item id="Page_1"  href="Page_1.html"  media-type="application/xhtml+xml"/>
    <item id="Page_1_jpg" href="images/Page_1.jpg"  media-type="image/jpeg"/>
    <item id="Page_2"  href="Page_2.html"  media-type="application/xhtml+xml"/>
    <item id="Page_2_jpg" href="images/Page_2.jpg"  media-type="image/jpeg"/>
  </manifest>

  <spine toc="ncx">
    <itemref idref="Page_1"/>
    <itemref idref="Page_2"/>
  </spine>
  <guide>
  </guide>
</package>

For now lets say I am only interested in the itemhref items under the spine node. What would be the right way to pull them out. I have tried multiple things but the only way I can get it work it by ripping out all the namespaces from the document. Is this a  bug in Jath or am I missing something?
BTW sorry for bugging you with such a n00b plea for help but if you can help me I would be happy to compensate by contributing a bit of documentation on how this namespace resolving thing works in your wiki.

XPath doesn't support the idea of a default namespace, so you have to map the url of your default namespace to some binding or use the local-name() function in order to refer to elements within the that namespace.
There are some examples of this if you look in samples.html. Probably the 'correct' way to approach it is to give Jath a resolver function:
Jath.resolver = function( prefix ) {
    var mappings = { def: "http://www.idpf.org/2007/opf" };
    return mappings[ prefix ];
}

Then use something like this as the template:
var template = [ "//def:spine", { idref: "@idref" } ];

Or you could alter the xpath selectors to use local-name():
var template = [ //*[local-name()='spine'], { idref: "@idref" } ];

I haven't tested these code bits. I might be able to do something a little later though.

Ok so obviously the problem is not Jath but my poor of knowledge of XPath I managed to have a bit of luck using the following template:
var template0 = {
  manifest: [ "//*[namespace-uri()='http://www.idpf.org/2007/opf' and local-name()='item']", {
    id: "@id",
    href: "@href",
    media_type: "@media-type"
  } ],
  spine: [ "//*[namespace-uri()='http://www.idpf.org/2007/opf' and local-name()='itemref']", {
    idref: "@idref"
  } ],
};

ideally I would like to tidy this up by passing Jath a resolver, I tried a few things but no luck so far. Here is one thing I thought would work:
Jath.resolver = packageDoc.createNSResolver(packageDoc)

and then use this as my template:
var template0 = {
  manifest: [ "//item", {
    id: "@id",
    href: "@href",
    media_type: "@media-type"
  } ],
  spine: [ "//itemref'", {
    idref: "@idref"
  } ],
};

By the way, thanks for your help!

No prob.
The reason your createNSResolver isn't working is because (ironically) only bound namespaces will be resolved. So probably your "dc:" namespaces will resolve this way but not the elements in the default namespace.

Ok looks like I am all sorted now. I get it, I missed the prefix part of the template you suggested. Once I added it in it is working perfectly :)

cool

authors: dnewcome matthewrobertson

repo: dnewcome/jath


text:
DNN-21973: retry the search when there have exception throw during search process.

Fix #2231.

Hey Team, any update on this? thx


text:
It's not an issue, but where does KXSrudio fit into crouton ?

I was reading this thread: https://lists.ubuntu.com/archives/ubuntu-studio-users/2015-March/thread.html#10319
And wondered where KXStudio fitted into crouton ?
I'know I'm gonna be slated, bt your gonna have to cut me some slack for this.
Of course I googled it and found it was a huge source of plug-ins
http://kxstudio.sourceforge.net/
Any chance this could be accesssed in future ?

See the wiki.

repo: dnschneid/crouton


text:
startxfce4 not working on freshly installed precise chroot

When trying to enter a fresh precise install in xfce4 I get a black screen, here is the output in crosh when I attempt sudo startxfce4
 Entering

  /mnt/stateful_partition/crouton/chroots/precise...
 /usr/bin/startxfce4: Starting X server

 _XSERVTransmkdir: Owner of /tmp/.X11-unix should be set to root

 X.Org X Server 1.15.1
 Release Date: 2014-04-13
 X Protocol Version 11, Revision 0
 Build Operating System: Linux 2.6.42-75-generic x86_64 Ubuntu
 Current Operating System: Linux localhost 3.10.18 #1 SMP Thu Dec 1 21:56:24 PST 2016 x86_64
 Kernel command line: cros_secure console= loglevel=7 init=/sbin/init cros_secure oops=panic panic=-1 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="1 vroot none ro 1,0 2539520 verity payload=PARTUUID=53f1dd78-d512-d54e-ab32-e6551aaac164/PARTNROFF=1 hashtree=PARTUUID=53f1dd78-d512-d54e-ab32-e6551aaac164/PARTNROFF=1 hashstart=2539520 alg=sha1 root_hexdigest=d11335a2b65d5fa5456e63a1e27d147fbf1da7ab salt=40337c851571888b038a5c0d6272ca30c848358b20f78d3bf4e5035ae5691e16" noinitrd vt.global_cursor_default=0 kern_guid=53f1dd78-d512-d54e-ab32-e6551aaac164 add_efi_memmap boot=local noresume noswap i915.modeset=1 tpm_tis.force=1 tpm_tis.interrupts=0 nmi_watchdog=panic,lapic
 Build Date: 12 February 2015  03:37:52PM
 xorg-server 2:1.15.1-0ubuntu2~precise5 (For technical support please see http://www.ubuntu.com/support)
 Current version of pixman: 0.30.2
         Before reporting problems, check http://wiki.x.org
         to make sure that you have the latest version.
 Markers: (--) probed, (**) from config file, (==) default setting,
         (++) from command line, (!!) notice, (II) informational,
         (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
 (++) Log file: "/tmp/Xorg.crouton.2.log", Time: Wed Dec  7 17:04:53 2016
 (==) Using system config directory "/usr/share/X11/xorg.conf.d"
 setversion 1.4 failed: Permission denied
 Initializing built-in extension Generic Event Extension
 Initializing built-in extension SHAPE
 Initializing built-in extension MIT-SHM
 Initializing built-in extension XInputExtension
 Initializing built-in extension XTEST
 Initializing built-in extension BIG-REQUESTS
 Initializing built-in extension SYNC
 Initializing built-in extension XKEYBOARD
 Initializing built-in extension XC-MISC
 Initializing built-in extension SECURITY
 Initializing built-in extension XINERAMA
 Initializing built-in extension XFIXES
 Initializing built-in extension RENDER
 Initializing built-in extension RANDR
 Initializing built-in extension COMPOSITE
 Initializing built-in extension DAMAGE
 Initializing built-in extension MIT-SCREEN-SAVER
 Initializing built-in extension DOUBLE-BUFFER
 Initializing built-in extension RECORD
 Initializing built-in extension DPMS
 Initializing built-in extension X-Resource
 Initializing built-in extension XVideo
 Initializing built-in extension XVideo-MotionCompensation
 Initializing built-in extension XFree86-VidModeExtension
 Initializing built-in extension XFree86-DGA
 Initializing built-in extension XFree86-DRI
 Initializing built-in extension DRI2
 Loading extension GLX
 xf86EnableIOPorts: failed to set IOPL for I/O (Operation not permitted)
 setversion 1.4 failed: Permission denied
 crouton: version 1-20161129162558~master:5f2f11f9
 release: precise
 architecture: amd64
 xmethod: xorg
 targets: xfce
 host: version 9000.15.0 (Official Build) dev-channel swanky
 kernel: Linux localhost 3.10.18 #1 SMP Thu Dec 1 21:56:24 PST 2016 x86_64 x86_64 x86_64 GNU/Linux
 freon: yes
 Another instance of croutontriggerd running, waiting...
 /etc/xdg/xfce4/xinitrc: 1: /etc/xdg/xfce4/xinitrc: ssh-agent: not found
 _IceTransmkdir: Owner of /tmp/.ICE-unix should be set to root

 (xfwm4:2577): Gtk-WARNING **: Unable to locate theme engine in module_path: "pixmap",

 (xfsettingsd:2582): Gtk-WARNING **: Unable to locate theme engine in module_path: "pixmap",

 (xfce4-session:2571): Gtk-WARNING **: Unable to locate theme engine in module_path: "pixmap",
 xfdesktop[2584]: starting up

 (xfwm4:2577): GLib-CRITICAL **: g_str_has_prefix: assertion `prefix != NULL' failed

 (xfdesktop:2584): Gtk-WARNING **: Unable to locate theme engine in module_path: "pixmap",

 (polkit-gnome-authentication-agent-1:2598): GLib-CRITICAL **: g_variant_new_string: assertion `string != NULL' failed

 (polkit-gnome-authentication-agent-1:2598): polkit-gnome-1-WARNING **: Failed to register client: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.gnome.SessionManager was not provided by any .service files

 (migrate:2610): Gtk-WARNING **: Unable to locate theme engine in module_path: "pixmap",

 (xfce4-settings-helper:2611): xfce4-settings-helper-WARNING **: Failed to get the _NET_NUMBER_OF_DESKTOPS property.

Any advice anyone can offer?

Hi everyone and ddllaagg,
Same experience here. I got a black screen with with standard xfce install on my Acer Chromebook CB5-571. I have since tried to create some more chroots with different enviroments.
tried -r jessie -t gnome ... it says something's wrong with one button to press. logout button
tried -t xbmc ... just blank screen
tried -t xfce ... something with unable to connect do d-bus
The only one working is -t unity -r trusty. But I am deleting it, since it occupies almost 4GB of space alone. As of now I'm trying to install xfce and trusty. See If it can works

@aleifuu,

tried -r jessie -t gnome ... it says something's wrong with one button showing ie one button to logout

I haven't used 'jessie' in ages so I can't help there, sorry.

tried -t xbmc ... just blank screen. I forgot what was it complaining for

Take a look at/track this issue here: Black screen #2923

tried -t xfce ... something with unable to connect do d-bus. some dialog window pop-up with similar error message

Take a look at/track this issue here: chroots will no longer boot to gui after updating chrome OS to version 56.0.2924.9 dev (64-bit) #2917

An alternative is to use the 'xiwi' xmethod.
Hope this helps,
-DennisL

Hi @DennisLfromGA, thanks for the reply. Sure I will look at those threads you mentioned
As for now. Well, I can confirm  -t xfce -r trusty is working and occupies around 900MB instead of unity/trusty that can go up to 4GB ( and slow ! ). So I'm good for now
On the other hand I'd just like to have separate xbmc chroots up and running and still having black screen. Will do some tweaking and see how it goes

I am very new to all this only having got my Pixel yesterday but I was a bit disappointing when I was just greeted with a black screen after following the default instructions.
Having found this issue I have managed to get it working with trusty and unity but trusty and xfce still fails.

@Roaders,
Have you tried using the 'xiwi' target?
Hope this helps,
-DennisL

Yes, it does work with xiwi thanks Dennis. Thanks for the help, I much prefer xfce to unity :-)

Black screen has been fixed with drinkcat's workarounds.

My daughter is using the ASUS Chromebook for Minecraft. When trying to startxfce4 recently, the screen went immediately blank. I am new to this and could use some help I tried to follow work arounds, no luck. Here is the script when typing sudo edit-chroot -all
name: precise
encrypted: no
Entering /mnt/stateful_partition/crouton/chroots/precise...
crouton: version 1-20161129162558~master:5f2f11f9
release: precise
architecture: amd64
xmethod: xorg
targets: xfce
host: version 9202.60.0 (Official Build) stable-channel quawks
kernel: Linux localhost 4.4.35-06796-g0d8baba #1 SMP PREEMPT Tue Mar 28 23:35:35 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux
freon: yes
Unmounting /mnt/stateful_partition/crouton/chroots/precise...
Any help would be appreciated...

@sgt7fathom,
This issue is closed but see this issue and comment for a probable cause: https://github.com/dnschneid/crouton/issues/3184#issuecomment-293057721
Hope this helps,
-DennisL

authors: DennisLfromGA Roaders aleifuu ddllaagg dnschneid sgt7fathom


text:
crouton must be run as a root

I have deleted a chroot, now I want to install a new one. I enter the exact same code I did before, and get this message:

Does anyone know if, or where I screwed up? And how can I make crouton run as a root?
Thanks in advance.
Edit:
Before that line of code I wrote this line, does it have anything to do with it?

@hylkevdveen,

I enter the exact same code I did before, and get this message:

Before that line of code I wrote this line, does it have anything to do with it?

Maybe not quite the exact same code, you're missing the 'sudo' prefix -
sudosh ~/Downloads/crouton -t kde
Hope this helps,
-DennisL

authors: DennisLfromGA hylkevdveen


text:
Error: Use of const in strict mode

const JARNAME = 'DynamoDBLocal.jar';
^^^^^
SyntaxError: Use of const in strict mode.
at Module._compile (module.js:439:25)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Module.require (module.js:364:17)
at require (module.js:380:17)
at Object. (/home/jdunlap/github/vidroll/vidroll-portal/gulpfile.js:9:14)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
OS: Debian Jessie
node --version v0.10.38

Thats because the const keyword was introduced in ES6.  Your version of node does not support ES6 constants.  I developed this module with some ES6 support required (const, let, Promise).  I'll update the docs. https://iojs.org/en/es6.html

authors: doapp-ryanp jadbox

repo: doapp-ryanp/dynamodb-local


text:
Dynamic class creation by "guessing" types

We already have support for dynamic class creation, but it assumes that the types for all fields are known in advance: https://github.com/docarray/docarray/pull/1179
However, it would be nice to be able to dynamically create classes from data alone. We would have to "guess" the types by looking at the data itself (e.g. in the dict).
This would be useful, for example, for loading data from csv without having a schema beforehand. Reference: https://github.com/docarray/docarray/pull/1144

I'd like to work on this issue

Hi @h4shk4t, I will take over this ticket because we want to merge it quickly. I hope you don't mind.

authors: JohannesMessner h4shk4t jupyterjazz

repo: docarray/docarray


text:
db-footnote="#####" attribute of <sub> tag does not validate xhtml.

db-footnote="#####" attribute of <sub> tag does not validate xhtml. I have not been able to find dependencies proposing to replace  with property="db-footnote" content="####"

This one is a little harder. The HTML5 rule is that element names and attribute names with hyphens in them are allowed for extensibility. I'll have to investigate. It's possible that the db-footnote attribute is actually only supposed to be used internally and it's "leaking" out into the final result by mistake. If that's the case, I'll just drop it.
Replacing it with property isn't really ideal IMHO because I might have more than one db- extension attribute.

authors: g-vidal ndw

repo: docbook/xslTNG

license: MIT


text:
Add "--build-order" to cat (and a couple minor bugfixes)

fix parsing of empty BASHBREW_ARCH_NAMESPACES
better (more contextual) error messages from bashbrew remote arches

(This is something we've emulated in a few places with bashbrew list --build-order ... | xargs bashbrew cat but that provides a result that's different in kind of subtle ways and I don't see a good reason not to support this here :sweat_smile:)

To be explicitly clear: this only sorts the top-level arguments to cat in build order - any deeper entry-level sorting would be accomplished in the format string.

repo: docker-library/bashbrew


text:
Questions: Disk usage and network connection

Sorry if the repo is not ready for public consumption yet, but I'd have two burning questions.

Does this address https://github.com/jpetazzo/dind#important-warning-about-disk-usage and is /var/lib/docker reused from the "outer-docker"?
How to connect to the containers running in the container?

PS: I tried a build and ran a nginx container, which worked quite well so far.

Yeah, definitely not ready for public consumption yet.  Really only created it to start soliciting comments from people who've expressed interest in it being a thing, such as @jpetazzo and @jfrazelle.
Functionally, it probably has all the exact same shortcomings as jpetazzo/dind.

repo: docker-library/docker


text:
Dockerfile using Multi-build

Hey folks!
I was looking to slim the Ghost image using multi-build and it looks someone made it happen. Wondering if there is a plan to update the official dockerfile and use multi-build?
I understand that some folks would prefer the actual image for developing while most of us want to use the image no-question-ask. Maybe having two versions would make sense?
Cheers!
Pascal

Official images do not support multi-stage builds; see https://github.com/docker-library/official-images/issues/3383
The image does not install anything that is not required by ghost-cli and ghost

perhaps a minimal version that installs ghost-cli to install ghost and then removes ghost-cli, all in one layer, but that would leave out the database initialization as well (knex-migrator-migrate --init --mgpath "$GHOST_INSTALL/current").

@acburdine, what do you think about an image without the ghost-cli or knex-migrator?  Would there be value in that?

Closing as the maintainer will take action once Docker EE support multi-build.

Thinking about it, it should be possible to remove the ghost-cli dependency. There was a change in Ghost-CLI recently too that removed the need to globally install knex-migrator - it now uses the one that's installed with Ghost itself.
tl;dr it should be too difficult.

Meanwhile, I forked the official project here. It supports the multi-stage build :
https://github.com/firepress-org/ghostfire
There are other features as well. Check out the README.
Cheers!
P

authors: acburdine pascalandy yosifkit

repo: docker-library/ghost


text:
updating docker and docker-compose

is there a plan to update docker and docker-compose to a more recent version?
docker seems to be at version 20.10.6 while docker-compose seems to be at 1.29.1 as opposed to docker version 19.03.15 and docker-compose version 1.25.5 in the snap

as i  press the create issue button i noticed the pull requests :see_no_evil:

Build is in progress, 20.10.6 should be hitting edge shortly!

awesome @tianon thanks!
do you have any thoughts on docker-compose 1.29.1?

Yeah, it's probably a good idea to update that as well, however I'm a bit more hesitant on that because I don't have any real testing of Compose here like I do Actions -- I believe Canonical has some spread tests (seen in this repository) that do some really basic testing of it, but I don't think those are going to be easy for us to run directly and would probably make sense to try and duplicate (at least the absolute basics) so we can bump without worrying as much about breaking some of the basic functionality (and then rely on the post-build testing that happens before we get all the way to the stable channel).

ah i see uh so it comes down to testing if docker-compose works as expected hmmm

I was looking for anything that would have a version of docker-compose that is compatible with compose file version 3. For that, docker-compose version 1.27+ is required. All instructions I find on the internet, including the official Docker documentation tell us to user curl to install docker-compose. However, that is a bad practice as it never updates and does not keep track of security patches.
I tried to find anything (snap channels, or a ppa repository for apt) that would satisfy these requirements, but such a thing does not yet exist. Even the latest/edge channel of this snap has a very old docker-compose 1.25.5.

hi @augusto-herrmann as mentioned earlier in this very thread the main blocker for the maintainer is a lack of means to ensure that docker-compose doesn't break whatsover under snap
if a rigous test could be run directly that would open the way

Docker compose has tests here.
To run them, just clone the repository and type
make test

so I don't understand what is really keeping docker-compose from being updated.

Docker compose has tests here.
To run them, just clone the repository and type
make test

so I don't understand what is really keeping docker-compose from being updated.

Refer to https://github.com/docker-snap/docker-snap/issues/37#issuecomment-845558871
If you think it can be done 'easily' i'm sure a PR would be appreciated

@Fuseteam that comment is from May. I believe those tests have already been implemented, see this file.

@augusto-herrmann whether it is from may or january is not important that was the current situation at the time i wrote that, you now have an new answer from tianon confirming the situation. again if it's "easy" and you consider it worth the effort i'm sure a PR is appreciated.

There's a PR up now at https://github.com/docker-snap/docker-snap/pull/48, and the CI generated a build at https://github.com/docker-snap/docker-snap/suites/3696078821/artifacts/93212286 -- it would help for someone to download/install that and do some basic testing to validate that at least some basic level of functionality is working as expected (up, down, ps, etc, then comment on https://github.com/docker-snap/docker-snap/pull/48 with the results).

authors: Fuseteam augusto-herrmann tianon

repo: docker-snap/docker-snap


text:
Docker pull & ping issue

I have a job running every 5 min which does a docker pull against the registry.
The command fails on the client with the error:
time="2015-04-29T06:05:06Z" level="fatal" msg="Error: v1 ping attempt failed with error: Get http://<.....>:5000/v1/_ping: dial tcp: i/o timeout"

the docker registry logs state:
2015/04/29 15:37:12 http: TLS handshake error from x.x.x.x:46333: tls: first record does not look like a TLS handshake
time="2015-04-29T15:40:06.642213348Z" level=info msg="response completed" environment=production http.request.host="registry.corp.snips.net:5000" http.request.id=80f1eb19-c274-4b36-91fd-cef68bfcb45e http.request.method=GET http.request.remoteaddr="y.y.y.y:56963" http.request.uri="/v1/_ping" http.request.useragent="docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64" http.response.contenttype="text/plain; charset=utf-8" http.response.duration="69.584µs" http.response.written=0 instance.id=6503ae81-1ab5-46d1-a178-1e3fb83b9caf service=registry version=v2.0.0
y.y.y.y - - [29/Apr/2015:15:40:06 +0000] "GET /v1/_ping HTTP/1.1" 404 19 "" "docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64"
time="2015-04-29T15:40:06.684952582Z" level=info msg="response completed" environment=production http.request.host="registry.corp.snips.net:5000" http.request.id=3ac40bdc-1f8b-4e1b-b3b0-10114af962f3 http.request.method=GET http.request.remoteaddr="y.y.y.y:56964" http.request.uri="/v1/_ping" http.request.useragent="docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64" http.response.contenttype="text/plain; charset=utf-8" http.response.duration="108.326µs" http.response.written=0 instance.id=6503ae81-1ab5-46d1-a178-1e3fb83b9caf service=registry version=v2.0.0
y.y.y.y - - [29/Apr/2015:15:40:06 +0000] "GET /v1/_ping HTTP/1.1" 404 19 "" "docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64"
time="2015-04-29T15:40:06.722088486Z" level=info msg="response completed" environment=production http.request.host="registry.corp.snips.net:5000" http.request.id=26c53527-a325-4cc2-a5df-db6a3e3b4d3c http.request.method=GET http.request.remoteaddr="y.y.y.y:56965" http.request.uri="/v1/repositories/scheduler/images" http.request.useragent="docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64" http.response.contenttype="text/plain; charset=utf-8" http.response.duration="93.282µs" http.response.written=0 instance.id=6503ae81-1ab5-46d1-a178-1e3fb83b9caf service=registry version=v2.0.0
y.y.y.y - - [29/Apr/2015:15:40:06 +0000] "GET /v1/repositories/scheduler/images HTTP/1.1" 404 19 "" "docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64"

Seems to be related to the latest version of Docker

Can you run the daemon in debug mode "docker -d -D" and send the logs. It appears that contacting the v2 registry has already failed (likely same timeout) and the final ping message is the last endpoint it will attempt (http to v1).

Here's the daemon logs:
DEBU[1835] Calling POST /images/create
INFO[1835] POST /v1.18/images/create?fromImage=registry.corp.snips.net%3A5000%2Fscheduler%3A0.5
INFO[1835] +job pull(registry.corp.snips.net:5000/scheduler, 0.5)
INFO[1835] +job resolve_repository(registry.corp.snips.net:5000/scheduler)
INFO[1835] -job resolve_repository(registry.corp.snips.net:5000/scheduler) = OK (0)
DEBU[1835] pulling image from host "registry.corp.snips.net:5000" with remote name "scheduler"
DEBU[1835] pinging registry endpoint https://registry.corp.snips.net:5000/v0/
DEBU[1835] attempting v2 ping for registry endpoint https://registry.corp.snips.net:5000/v2/
DEBU[1835] https://registry.corp.snips.net:5000/v2/ -- HEADERS: map[User-Agent:[docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64]]
DEBU[1840] attempting v1 ping for registry endpoint https://registry.corp.snips.net:5000/v1/
DEBU[1840] https://registry.corp.snips.net:5000/v1/_ping -- HEADERS: map[User-Agent:[docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64]]
DEBU[1840] Error unmarshalling the _ping RegistryInfo: invalid character 'p' after top-level value
DEBU[1840] RegistryInfo.Version: ""
DEBU[1840] Registry standalone header: ''
DEBU[1840] RegistryInfo.Standalone: true
DEBU[1840] attempting v1 ping for registry endpoint https://registry.corp.snips.net:5000/v1/
DEBU[1840] https://registry.corp.snips.net:5000/v1/_ping -- HEADERS: map[User-Agent:[docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64]]
DEBU[1840] Error unmarshalling the _ping RegistryInfo: invalid character 'p' after top-level value
DEBU[1840] RegistryInfo.Version: ""
DEBU[1840] Registry standalone header: ''
DEBU[1840] RegistryInfo.Standalone: true
DEBU[1840] Endpoint https://registry.corp.snips.net:5000/v1/ is eligible for private registry. Enabling decorator.
DEBU[1840] pulling v1 repository with local name "registry.corp.snips.net:5000/scheduler"
DEBU[1840] [registry] Calling GET https://registry.corp.snips.net:5000/v1/repositories/scheduler/images
DEBU[1840] https://registry.corp.snips.net:5000/v1/repositories/scheduler/images -- HEADERS: map[Authorization:[Basic Og==] User-Agent:[docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64]]
Error: image scheduler:0.5 not found

And what did you see in the logs from run where you pulled these logs from?  It appears the interaction with the registry was successful, its just the image does not exist on the registry.

I wasn't running the registry in debug mode for the first error, but I'll post here if I get the ping timeout error.
What I don't understand is that some docker pulls show a 404 or a ping timeout, but work 95% of the time.
Also, why is it falling back on the /v1 endpoint instead of /v2?

Do you have a proxy in front of the v2 registry? From the daemon logs it appears the v2 ping did not succeed while the v1 ping did.  This would imply that the "/v2" endpoint is either not returning a 200/401 or the "Docker-Distribution-API-Version" header is missing (see #460).

No proxy in front of the registry. I just happen to see a ping timeout event again. Here's the daemon logs:
INFO[1378] POST /v1.18/images/create?fromImage=registry.corp.snips.net%3A5000%2Fscheduler%3A0.5
INFO[1378] +job pull(registry.corp.snips.net:5000/scheduler, 0.5)
INFO[1378] +job resolve_repository(registry.corp.snips.net:5000/scheduler)
INFO[1378] -job resolve_repository(registry.corp.snips.net:5000/scheduler) = OK (0)
Get https://registry.corp.snips.net:5000/v2/: dial tcp: i/o timeout
INFO[1383] -job pull(registry.corp.snips.net:5000/scheduler, 0.5) = ERR (1)
ERRO[1383] Handler for POST /images/create returned error: Get https://registry.corp.snips.net:5000/v2/: dial tcp: i/o timeout
ERRO[1383] HTTP Error: statusCode=500 Get https://registry.corp.snips.net:5000/v2/: dial tcp: i/o timeout

and the registry logs:
2015/04/29 20:07:22 http: TLS handshake error from 172.17.42.1:45434: tls: first record does not look like a TLS handshake
time="2015-04-29T20:10:01.475923831Z" level=debug msg="authorizing request" environment=production http.request.host="registry.corp.snips.net:5000" http.request.id=955c4731-02e1-4961-90b4-80fede18b49d http.request.method=GET http.request.remoteaddr="10.181.8.3:43994" http.request.uri="/v2/" http.request.useragent="docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64" instance.id=6503ae81-1ab5-46d1-a178-1e3fb83b9caf service=registry version=v2.0.0
time="2015-04-29T20:10:01.475986379Z" level=info msg="response completed" environment=production http.request.host="registry.corp.snips.net:5000" http.request.id=955c4731-02e1-4961-90b4-80fede18b49d http.request.method=GET http.request.remoteaddr="10.181.8.3:43994" http.request.uri="/v2/" http.request.useragent="docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.910903ms http.response.written=0 instance.id=6503ae81-1ab5-46d1-a178-1e3fb83b9caf service=registry version=v2.0.0
10.181.8.3 - - [29/Apr/2015:20:10:01 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.13.0-32-generic os/linux arch/amd64"
2015/04/29 20:13:49 http: TLS handshake error from 172.17.42.1:46179: tls: first record does not look like a TLS handshake

whereas a successful run on the daemon looks like
INFO[1678] POST /v1.18/images/create?fromImage=registry.corp.snips.net%3A5000%2Fscheduler%3A0.5
INFO[1678] +job pull(registry.corp.snips.net:5000/scheduler, 0.5)
INFO[1678] +job resolve_repository(registry.corp.snips.net:5000/scheduler)
INFO[1678] -job resolve_repository(registry.corp.snips.net:5000/scheduler) = OK (0)
INFO[1679] +job trust_key_check(/scheduler)
INFO[1679] -job trust_key_check(/scheduler) = OK (0)
INFO[1679] +job log(pull, registry.corp.snips.net:5000/scheduler:0.5, )
INFO[1679] -job log(pull, registry.corp.snips.net:5000/scheduler:0.5, ) = OK (0)
INFO[1679] -job pull(registry.corp.snips.net:5000/scheduler, 0.5) = OK (0)
INFO[1680] POST /v1.18/containers/create
INFO[1680] +job create()
Your kernel does not support swap limit capabilities. Limitation discarded.
INFO[1680] +job log(create, d0e393ef8ae7d4c17c016793ac35b29154cee7d100c25a07fcaaeda1b3a4d1d5, registry.corp.snips.net:5000/scheduler:0.5)
INFO[1680] -job log(create, d0e393ef8ae7d4c17c016793ac35b29154cee7d100c25a07fcaaeda1b3a4d1d5, registry.corp.snips.net:5000/scheduler:0.5) = OK (0)
INFO[1680] -job create() = OK (0)

Got same issue
When i try to pull 4 images from registry2.0 at same time in one docker host.
docker says:
6941bfcbbfca: Pulling fs layer
41459f052977: Pulling fs layer
fd44297e2ddb: Pulling fs layer
1af8cd1f5388: Pulling fs layer
9cbef98312db: Pulling fs layer
e209fd7dcb60: Pulling fs layer
b13f079178bf: Pulling fs layer
59d1aafaec5a: Pulling fs layer
54c4a17d72b2: Pulling fs layer
927bafcb5b7b: Pulling fs layer
a1adf1d3096e: Pulling fs layer
d33c13eb716a: Pulling fs layer
10cdc145c583: Layer already being pulled by another client. Waiting.
latest: Pulling from docker.localtest.cn:5000/registry-small-test1
6941bfcbbfca: Download complete
41459f052977: Layer already being pulled by another client. Waiting.
......
after a while
......
FATA[0000] Error: image registry-small-test1:latest not found
FATA[0000] Error: image registry-small-test2:latest not found
FATA[0000] Error: image registry-small-test3:latest not found
FATA[0000] Error: image registry-small-test4:latest not found

and i can see the log from registry:
INFO[1425] Base.ReadStream("/docker/registry/v2/blobs/sha256/1a/1ac41a9515494fae407ef4cdc33b955a8610639100022d683fa55e67f7348515/data", 0)  trace.duration=30.891µs trace.file=/usr/local/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).ReadStream trace.id=254e4ce2-d74c-4eed-a2c2-8539d5896174 trace.line=79
INFO[1425] Base.ReadStream("/docker/registry/v2/blobs/sha256/1a/1ac41a9515494fae407ef4cdc33b955a8610639100022d683fa55e67f7348515/data", 0)  trace.duration=33.486µs trace.file=/usr/local/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func=github.com/docker/distribution/registry/storage/driver/base.(*Base).ReadStream trace.id=1833c1ec-c61d-4d85-a321-86889bdc5c0a trace.line=79
INFO[1425] response completed                            environment=production http.request.host=docker.localtest.cn:5000 http.request.id=fe5d470d-e31b-4c3d-9bdd-d4e166fe7561 http.request.method=GET http.request.remoteaddr=10.142.104.250:59716 http.request.uri=/v2/registry-small-test4/blobs/sha256:1ac41a9515494fae407ef4cdc33b955a8610639100022d683fa55e67f7348515 http.request.useragent=docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.12.35-100.el6.x86_64 os/linux arch/amd64 http.response.contenttype=application/x-gzip http.response.duration=140.21819ms http.response.status=200 http.response.written=2111 instance.id=3eab1c58-8be0-4a53-9eab-2a6233a0f368 service=registry version=v2.0.0-22-gaa09c6c.m
10.142.104.250 - - [03/May/2015:11:19:19 +0800] "GET /v2/registry-small-test4/blobs/sha256:1ac41a9515494fae407ef4cdc33b955a8610639100022d683fa55e67f7348515 HTTP/1.1" 200 2111 "" "docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.12.35-100.el6.x86_64 os/linux arch/amd64"
INFO[1425] response completed                            environment=production http.request.host=docker.localtest.cn:5000 http.request.id=99d215ba-009e-4090-92c8-ed8bd826aba5 http.request.method=GET http.request.remoteaddr=10.142.104.250:59718 http.request.uri=/v1/repositories/registry-small-test4/images http.request.useragent=docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.12.35-100.el6.x86_64 os/linux arch/amd64 http.response.contenttype=text/plain; charset=utf-8 http.response.duration=67.226µs http.response.status=404 http.response.written=19 instance.id=3eab1c58-8be0-4a53-9eab-2a6233a0f368 service=registry version=v2.0.0-22-gaa09c6c.m
10.142.104.250 - - [03/May/2015:11:19:19 +0800] "GET /v1/repositories/registry-small-test4/images HTTP/1.1" 404 19 "" "docker/1.6.0 go/go1.4.2 git-commit/4749651 kernel/3.12.35-100.el6.x86_64 os/linux arch/amd64"

I have upgrade docker to 1.6 ,why it GET from v1 api?
but If i pull only one image, didn't get these error before.
docker version
Client version: 1.6.0
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 4749651
OS/Arch (client): linux/amd64
Server version: 1.6.0
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 4749651
OS/Arch (server): linux/amd64

docker info
Containers: 0
Images: 41
Storage Driver: devicemapper
 Pool Name: docker-253:1-1179939-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.337 GB
 Data Space Total: 107.4 GB
 Data Space Available: 105 GB
 Metadata Space Used: 3.215 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Udev Sync Supported: false
 Data loop file: /usr/local/docker/devicemapper/devicemapper/data
 Metadata loop file: /usr/local/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 3.12.35-100.el6.x86_64
Operating System: <unknown>
CPUs: 24
Total Memory: 63.01 GiB
Name: app07.add.bjcc.qihoo.net
ID: GH5C:XVHB:JIQ6:GAOM:6N34:N4GK:D2FZ:3WJY:JGB6:NIRW:GXDN:NNXJ

+1
My registry docker version is
[root@docker1 current]# docker info
Containers: 1
Images: 74
Storage Driver: devicemapper
 Pool Name: docker-202:1-150166-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.743 GB
 Data Space Total: 107.4 GB
 Data Space Available: 4.186 GB
 Metadata Space Used: 3.604 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.144 GB
 Udev Sync Supported: true
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 3.14.44-32.39.amzn1.x86_64
Operating System: Amazon Linux AMI 2015.03
CPUs: 2
Total Memory: 7.341 GiB
Name: docker1
ID: ZDE6:4PI2:O33L:VZ2W:6NU2:DINQ:ZL4K:LUKV:IEJ5:GGFW:NL2E:O5AL

docker version is
[root@docker1 current]# docker version
Client version: 1.6.2
Client API version: 1.18
Go version (client): go1.4.2
Git commit (client): 7c8fca2/1.6.2
OS/Arch (client): linux/amd64
Server version: 1.6.2
Server API version: 1.18
Go version (server): go1.4.2
Git commit (server): 7c8fca2/1.6.2
OS/Arch (server): linux/amd64

Also got a proxy in front of registry, but no matter there is proxy or not, it just didn't work for pulling images at the latest image, the same as @tinytub .
> curl -v -s https://xxxxx/v2
* Hostname was NOT found in DNS cache
*   Trying 10.97.217.22...
* Connected to dcsrd-docker-registry.trendmicro.com (10.97.217.22) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
* Server certificate: *.trendmicro.com
* Server certificate: Trend Micro S2 CA
* Server certificate: AffirmTrust Commercial
> GET /v2 HTTP/1.1
> User-Agent: curl/7.37.1
> Host: dcsrd-docker-registry.trendmicro.com
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: text/html
< Date: Thu, 09 Jul 2015 10:31:59 GMT
< Docker-Distribution-Api-Version: : registry/2.0
* Server nginx/1.7.12 is not blacklisted
< Server: nginx/1.7.12
< WWW-Authenticate: Basic realm="Restricted"
< Content-Length: 195
< Connection: keep-alive
<

<title>401 Authorization Required</title>

401 Authorization Required

nginx/1.7.12

* Connection #0 to host dcsrd-docker-registry.trendmicro.com left intact

Is there anyone knows how to resolve this? Thank you very much.

Got same issue with @tinytub. The client v.1.6.2 will fail in the last image pull from private registry.
Eventually resolve the issue by upgrading docker client to v.1.7.0. (I didn't change any setting from docker registry side)
Hope it helps someone who also encounters this issue.

Based on a cursory look, it seems this is likely caused by a misconfiguration of tls:

Something is starting out with a band handshake when connecting to the registry. This is likely not a docker/distribution issue, so I recommend filing over in docker/docker.

I think I'm seeing this bug too using Docker 1.7.0 on boot2docker requesting images from a registry:2.0 container.
If I look at the logs of the registry, this is what I see:
172.19.2.93 - - [14/Jul/2015:21:55:41 +0000] "GET /v2/kcontrol/blobs/sha256:db797c3f390882d0ffa4384179a343eb5b340023272bc8d886e90e702cb43563 HTTP/1.1" 200 90 "" "docker/1.7.0 go/go1.4.2 git-commit/0baf609 kernel/4.0.5-boot2docker os/linux arch/amd64"
time="2015-07-14T21:55:41.016677367Z" level=info msg="filesystem.ReadStream(\"/docker/registry/v2/blobs/sha256/10/107c99f248e2939f2f73b97f75667d2a0518b660df26350fc1b6d2476c5976f0/data\", 0)" trace.duration=43.602µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).ReadStream" trace.id=7ba711d2-8c35-413c-9e63-04948f82b15f trace.line=79
time="2015-07-14T21:55:41.191180162Z" level=info msg="response completed" environment=development http.request.host="csig.ddns.net:5000" http.request.id=2fe28f3e-46b5-4800-b840-d80de3d7bf92 http.request.method=GET http.request.remoteaddr="172.19.2.93:46494" http.request.uri="/v2/kcontrol/blobs/sha256:107c99f248e2939f2f73b97f75667d2a0518b660df26350fc1b6d2476c5976f0" http.request.useragent="docker/1.7.0 go/go1.4.2 git-commit/0baf609 kernel/4.0.5-boot2docker os/linux arch/amd64" http.response.contenttype="application/x-gzip" http.response.duration=185.849234ms http.response.status=200 http.response.written=6018204 instance.id=872df3d1-616e-4242-be3d-890e3eb88214 service=registry version=v2.0.1

172.19.2.93 - - [14/Jul/2015:21:55:41 +0000] "GET /v2/kcontrol/blobs/sha256:107c99f248e2939f2f73b97f75667d2a0518b660df26350fc1b6d2476c5976f0 HTTP/1.1" 200 6018204 "" "docker/1.7.0 go/go1.4.2 git-commit/0baf609 kernel/4.0.5-boot2docker os/linux arch/amd64"
time="2015-07-14T21:55:43.029150689Z" level=info msg="response completed" environment=development http.request.host="csig.ddns.net:5000" http.request.id=3fcab66b-ae5b-415e-96c2-005dff17e10b http.request.method=GET http.request.remoteaddr="172.19.2.93:46499" http.request.uri="/v1/repositories/kontrol/images" http.request.useragent="docker/1.7.0 go/go1.4.2 git-commit/0baf609 kernel/4.0.5-boot2docker os/linux arch/amd64" http.response.contenttype="text/plain; charset=utf-8" http.response.duration="103.962µs" http.response.status=404 http.response.written=19 instance.id=872df3d1-616e-4242-be3d-890e3eb88214 service=registry version=v2.0.1

172.19.2.93 - - [14/Jul/2015:21:55:43 +0000] "GET /v1/repositories/kcontrol/images HTTP/1.1" 404 19 "" "docker/1.7.0 go/go1.4.2 git-commit/0baf609 kernel/4.0.5-boot2docker os/linux arch/amd64

And this is the message from the Docker client pulling:
latest: Pulling from x.y.z:5000/kcontrol
562fccfb6d84: Pull complete
cecc1f2b656b: Pull complete
82e59b0bcc13: Pull complete
cb77c9a66b88: Download complete
626d5a81290d: Download complete
83e4dde6b9cf: Already exists
b670fb0c7ecd: Already exists
29460ac93442: Already exists
d2a0ecffe6fa: Already exists
0446758af79d: Already exists
2c93318dc1f9: Already exists
4e82b4cb1ec8: Already exists
Pulling repository x.y.z:5000/kcontrol
Error: image kcontrol:latest not found

Based on the registry logs, it is operating properly. This seems like an issue with docker itself. Please file an issue over in the other project.

I'm using docker and I write this code
docker run -p 8888:8888 -it --rm b.gcr.io/tensorflow-udacity/assignments but it fails
Is there a method that allow me to download one by one?

@AbdelazimGalal this is a closed issue, and your problem is certainly unrelated.
Since you are using Google Container Registry, and your problem is pulling from there, you should really start there.

authors: AbdelazimGalal corradio dmcgowan dmp42 msfuko sayash stevvooe tinytub

repo: docker/distribution


text:
Clean up Makefile

Clean up Makefile and keep the same targets between rules and PHONY.
Signed-off-by: yuexiao-wang wang.yuexiao@zte.com.cn

Current coverage is 51.27% (diff: 100%)

Merging #1913 into master will decrease coverage by 10.19%

@@             master      #1913   diff @@
==========================================
  Files           123        123
  Lines         10919      10919
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits           6712       5599   -1113
- Misses         3330       4583   +1253
+ Partials        877        737    -140

Powered by Codecov. Last update 010e063...11bc169

@dmcgowan
Sorry, the PR is from an unknown repository and I can't rebase. Therefore, I  will close this PR and reopen a new one for it as follows: #2032

authors: codecov-io yuexiao-wang


text:
Minor corrections to restore command arguments

Describe the proposed changes

Project version

Related issue

Related issue or PR in another project

Please take a look

Corrected command arguments for restore.
Listed correct link for restore documentation.
Changed usage of /tmp to /var/tmp for storage of backups as /tmp gets cleaned out on reboot.

Thanks for updating this! LGTM 🚀


text:
Troubleshoot.md has incorrect command, missing "/usr/bin/log" before "show"

Proposed changes
Improve historical log command - missing log command in example.

Deploy preview for modest-edison-79d82e ready!
Built with commit 8bfa9d5c6b1f3109ade8470d5ccd670d249e723b
https://deploy-preview-7068--modest-edison-79d82e.netlify.com

authors: peter-docker rfay


text:
v1.9.0-RC2 - FreeBSD jenkins build failure

Hi there,
Any chances that someone take a look at the FreeBSD jenkins job that fails since 14 days ? =)
Not an expert in this field but seems that it is only a workspace issue :
https://jenkins.dockerproject.org/job/Docker Master (freebsd)/1555/console
Caused by: hudson.plugins.git.GitException: Command "git -c core.askpass=true fetch --tags --progress https://github.com/docker/docker.git +refs/heads/*:refs/remotes/origin/*" returned status code 1:
stdout:
stderr: remote: Counting objects: 8, done.
remote: Total 8 (delta 5), reused 5 (delta 5), pack-reused 2
From https://github.com/docker/docker
   75128c4..9328fc7  master     -> origin/master
error: Cannot lock ref 'refs/remotes/origin/release/1.8': 'refs/remotes/origin/release' exists; cannot create 'refs/remotes/origin/release/1.8'
 ! [new branch]      release/1.8 -> origin/release/1.8  (unable to update local ref)
error: Cannot lock ref 'refs/remotes/origin/release/v1.9': 'refs/remotes/origin/release' exists; cannot create 'refs/remotes/origin/release/v1.9'
 ! [new branch]      release/v1.9 -> origin/release/v1.9  (unable to update local ref)
error: some local refs could not be updated; try running
 'git remote prune https://github.com/docker/docker.git' to remove any old, conflicting branches

Hi!
Please read this important information about creating issues.
If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.
If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.
This is an automated, informational response.
Thank you.
For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

BUG REPORT INFORMATION
Use the commands below to provide key information from your environment:
docker version:
docker info:
uname -a:
Provide additional environment details (AWS, VirtualBox, physical, etc.):
List the steps to reproduce the issue:
1.
2.
3.
Describe the results you received:
Describe the results you expected:
Provide additional info you think is important:
----------END REPORT ---------
#ENEEDMOREINFO

Thanks for reporting! I'll try to get someone to have a look

ping @kvasdopil we may need your help here :heart:

@thaJeztah hi, i'll take a look asap, thx for ping

Thanks @kvasdopil !

The error doesn't seem to be related to freebsd build, looks like some kind of repository misconfiguration, maybe @jfrazelle could take a look.
However freebsd build is also broken, i'll try to fix it.

FreeBSD build faild on git fetch in jenkins:

git -c core.askpass=true fetch --tags --progress https://github.com/docker/docker.git +refs/heads/:refs/remotes/origin/
ERROR: Error fetching remote repo 'origin'

It's looks like simple or it not important?
Anyway thank you for your great job!

Freebsd machine is stopped for 4 months and also there is no active work on it. I'm closing this issue for now. Feel free to post new.

authors: GordonTheTurtle LK4D4 kalw kvasdopil sakal thaJeztah

repo: docker/docker


text:
--env-add flag doesn't update the variable as the help states

Output of docker version:
Client:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:00:36 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.0
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   8eab29e
 Built:        Thu Jul 28 22:00:36 2016
 OS/Arch:      linux/amd64

Output of docker info:
Containers: 18
 Running: 7
 Paused: 0
 Stopped: 11
Images: 10
Server Version: 1.12.0
Storage Driver: devicemapper
 Pool Name: docker-202:1-149180-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 5.9 GB
 Data Space Total: 107.4 GB
 Data Space Available: 963.2 MB
 Metadata Space Used: 8.262 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 963.2 MB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.77 (2012-10-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge overlay null
Swarm: active
 NodeID: **
 Is Manager: true
 ClusterID: **
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot interval: 10000
  Heartbeat tick: 1
  Election tick: 3
 Dispatcher:
  Heartbeat period: 5 seconds
 CA configuration:
  Expiry duration: 3 months
 Node Address: **
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-92-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.859 GiB
Name: ip-10-100-20-69
ID: ***
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Steps to reproduce the issue:

Create a service: i.e: docker service create -e A=c --name test redis:latest

docker service inspect test --pretty gives you the following:
ID:		a666bsuxvm6yem3nub2ikplyi
Name:		test
Mode:		Replicated
 Replicas:	1
Update status:
 State:		updating
 Started:	1 seconds ago
 Message:	update in progress
Placement:
UpdateConfig:
 Parallelism:	1
 On failure:	pause
ContainerSpec:
 Image:		redis:latest
 Env:		A=c
Resources:

You can see the environment variable has been passed.

Now, update the environment variable A following the description of --env-add, which states it's used to update the variable as well as add a new one. I understand it work like an 'upsert':

docker service update --env-add A=b test

See the output of docker service inspect test --pretty

ID:		a666bsuxvm6yem3nub2ikplyi
Name:		test
Mode:		Replicated
 Replicas:	1
Update status:
 State:		completed
 Started:	3 seconds ago
 Completed:	1 seconds ago
 Message:	update completed
Placement:
UpdateConfig:
 Parallelism:	1
 On failure:	pause
ContainerSpec:
 Image:		redis:latest
 Env:		A=c A=b
Resources:

Describe the results you received:
As it's evident, the environment variable is duplicated:
 Env:		A=c A=b

I expect the variable A to be updated instead.

ping @dnephin PTAL

@hvpareja @thaJeztah @dnephin I created a pull request #25427 to address this issue. Please let me know if there are any issues.

authors: hvpareja thaJeztah yongtang


text:
Docker Engine Swarm Mode + Load Balancing

Description
A service running in one node with a published port is not accessible when accessed with the same port on another node. I have 4 nodes in the cluster with 3 managers and 1 worker
Steps to reproduce the issue:

Deploy a service (web app) with a port (8080) published to the swarm.
The scheduler schedules it in node 1. The service (web app) is accessible from a browser at node1:8080
When I tried to access the service using node2:8080, the service (app) does not come up.
as per the docs,  Docker Swarm automatically makes a published port available on ALL nodes, whether or not the service is running locally on that node. Is this not true? if yes any idea why its not working for me?

Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:
 OS/Arch:      linux/amd64

Output of docker info:
Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 209
Server Version: 1.12.1
Storage Driver: devicemapper
 Pool Name: docker-253:0-204218855-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 5.067 GB
 Data Space Total: 107.4 GB
 Data Space Available: 31.06 GB
 Metadata Space Used: 8.471 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.139 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: active
 NodeID: 3y23oi8zayd4r5ehs8sq2438q
 Is Manager: true
 ClusterID: eytqq32a3nwwkal9iv6i936jd
 Managers: 3
 Nodes: 4
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.1.2.56
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.28.2.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 47.01 GiB
Name: cmcmdmz101
ID: N6XG:EQOG:FPUM:GYCY:HXHQ:DOVU:AHFM:TFBB:4GM6:WKC3:QCN2:6RLP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: cmcdocker
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

I meet the same problem ,is there someone could explain why the service create via swarm could not access from the node without running containers ?

Suspected reason is that ingress network not sync to that node when no containers got running.

@saithala Could you please retry on 1.12.2? Thanks

@aluzzardi no luck on 1.12.2 too.

@saithala can you verify if the required ports for overlay networking are opened, and accessible between nodes? https://docs.docker.com/engine/swarm/swarm-tutorial/#/open-ports-between-the-hosts
Also may be worth checking if a dependency / configuration is missing on your nodes using the check-config script https://github.com/docker/docker/blob/master/contrib/check-config.sh

I'm starting to run into this. Except in our case, this used to work up until this morning. Suddenly, ports on other nodes are no longer accessible. Running docker 1.12.2
docker info Containers: 15  Running: 5  Paused: 0  Stopped: 10 Images: 28 Server Version: 1.12.2 Storage Driver: aufs  Root Dir: /var/lib/docker/aufs  Backing Filesystem: extfs  Dirs: 192  Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins:  Volume: local  Network: host bridge overlay null Swarm: active  NodeID: 6t1dwdfrcosk816bf5oxxkzei  Is Manager: true  ClusterID: 2989ooasbl08rjlotjxx2yc6x  Managers: 3  Nodes: 3  Orchestration:   Task History Retention Limit: 5  Raft:   Snapshot Interval: 10000   Heartbeat Tick: 1   Election Tick: 3  Dispatcher:   Heartbeat Period: 5 seconds  CA Configuration:   Expiry Duration: 3 months  Node Address: 10.10.14.39 Runtimes: runc Default Runtime: runc Security Options: apparmor seccomp Kernel Version: 4.4.0-36-generic Operating System: Ubuntu 16.04.1 LTS OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 31.42 GiB Name: ip-10-10-14-39 ID: H4KX:W7CG:C5P7:ABZY:QM3I:QHRE:KX3R:VPUE:AOMI:I65K:ID27:HEPN Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ WARNING: No swap limit support Insecure Registries:  127.0.0.0/8

@nsamala can you check daemon logs of each node, to see if something stands out?

Sorry, I should've updated this! I found that we were using Kong and I had unnecessarily published port 7946. This is what caused our issue as far as I can tell.
Only discovered this after starting a new swarm and turning services on one by one. Haven't had the problem since.

@nsamala no problem, that's good news :smile:

I'm having this exact same issue. I posted on StackOverflow but I'll copy it here:
I am running a three node Swarm Mode cluster in AWS; one master and two workers. This is swarm mode not to be confused with docker swarm from pre 1.12. Using the master node I have created a service with the following
docker service create --replicas 1 --name myapp -p 3000 myapp

When I run docker service ps myapp I get the following output
ID                         NAME     IMAGE         NODE             DESIRED STATE  CURRENT STATE            ERROR
02awst8p9pezgpkfzqgz8z79t  myapp.1  myapp:latest  swarm-node-01    Running        Running 19 minutes ago

The running task is deployed to swarm-node-01.
I checked the auto-selected port which was published publicly
$ docker service inspect myapp | jq .[].Endpoint.Ports[].PublishedPort
30000

According to the documentation:

External components, such as cloud load balancers, can access the service on the PublishedPort of any node in the cluster whether or not the node is currently running the task for the service. All nodes in the swarm route ingress connections to a running task instance.

But when I try to curl the nodes who do not have the task running I'm getting connection refused.
$ curl $(docker-machine ip swarm-node-01):30000/stats
{"uptime":"2016-11-09T14:48:35Z","requestCount":7,"statuses":{"200":7},"pid":1,"open_db_conns":0}

$ curl $(docker-machine ip swarm-node-02):30000/stats
curl: (7) Failed to connect to [the IP] port 30000: Connection refused

note: I scrubbed the IP of node-02

My Troubleshooting:

The nodes are both properly connected to the swarm
Scaling the service up to 5 (which inherently deploys the task to every node) makes curl work on every node, because the task is deployed to every node.

@blockloop did you check my previous comment https://github.com/docker/docker/issues/27237#issuecomment-254006465

I just found this error on the nodes syslog
Nov  9 15:37:54 ubuntu docker[23092]: time="2016-11-09T15:37:54.494743354Z" level=error msg="Failed to join memberlist [10.0.0.12] on retry: 1 error(s) occurred:\n\n* Failed to join 10.0.0.12: dial tcp 10.0.0.12:7946: getsockopt: connection refused"

So, yes, looks like the required ports for overlay networking are not opened between your nodes

The security group in AWS is set to allow all traffic from within the same SG.

Running this on master shows that that port isn't listening
ubuntu@swarm-master-01:~$ sudo lsof -i :7946
ubuntu@swarm-master-01:~$ cat < /dev/tcp/10.0.0.12/7946
-bash: connect: Connection refused
-bash: /dev/tcp/10.0.0.12/7946: Connection refused
ubuntu@swarm-master-01:~$ cat < /dev/tcp/0.0.0.0/7946
-bash: connect: Connection refused
-bash: /dev/tcp/0.0.0.0/7946: Connection refused

Found this in the syslog of the master but I don't remember trying to explicitly remove the ingress network. Perhaps this is an internal process
Nov  8 21:44:03 ubuntu docker[24901]: time="2016-11-08T21:44:03.724145514Z" level=error msg="remove task failed" error="network ingress not found" module=taskmanager task.id=1w7yas48o7zq64umfersauxo8

I believe the culprit is the master who does not appear to be listening on port 7946. netstat shows that 7946 is listening on the nodes, but not the master. When I check the syslogs for the nodes I see the following error
level=error msg="Failed to join memberlist [10.0.0.12] on retry: 1 error(s) occurred:\n\n* Failed to join 10.0.0.12: dial tcp 10.0.0.12:7946: getsockopt: connection refused"

When you initialized the swarm, did you provide a --advertise-addr (and possibly --listen-addr)?

I provided both as 10.0.0.12:2377

@mavenugo any thoughts?

I was able to get around the issue for now, but I'd like to know how to figure out why the overlay network port wasn't listening on master-01. Here is what I did to circumvent the issue:

created another node with docker-machine called swarm-master-02
joined swarm-master-02 to the cluster as a master
demoted master-01 which set master-02 as the leader
restarted the docker daemon on each node (might not have been necessary)

Now all of the machines are working as expected except for master-01. One task is running on node-01 and curl works against all nodes by forwarding the traffic to the proper container on the proper node. However, swarm-master-01 refuses to listen on the overlay network and curl does not work against this node. I was only able to fix swarm-master-01 by completely removing it from the cluster and joining it again as a master. Now 7946 is listening on that machine.

All nodes are running 1.12.3?

Yes

authors: LamCiuLoeng aluzzardi blockloop cmingxu nsamala saithala thaJeztah


text:
Error: No such service: ucp-agent

Description
I am getting the following error while trying to uninstall the docker ucp.

$ docker run --rm -it
-v /var/run/docker.sock:/var/run/docker.sock
--name ucp
docker/ucp uninstall-ucp --interactive
INFO[0000] Your engine version 1.13.1-cs1, build 8709b81 (3.10.0-514.el7.x86_64) is compatible
INFO[0000] We're about to uninstall from this swarm cluster. UCP ID: 6BID:R7QS:O5OZ:3MX5:3LDN:2UM7:AA4H:UX5G:MJEK:UOHL:V2AV:5LCB
WARN[0000] WARNING: After uninstalling, you will not be able to join new nodes to the
WARN[0000] cluster until UCP is reinstalled or all nodes are removed from the current swarm.
WARN[0000] WARNING: UCP must be reinstalled prior to expiration of your certificates,
WARN[0000] or the cluster will experience problems.
Do you want proceed with the uninstall? (y/n): y
FATA[0000] Error: No such service: ucp-agent
Output of docker version:
$ docker version
Client:
 Version:      1.13.1-cs1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   8709b81
 Built:        Thu Feb  9 02:32:02 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1-cs1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   8709b81
 Built:        Thu Feb  9 02:32:02 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:
$ docker info
Containers: 8
 Running: 6
 Paused: 0
 Stopped: 2
Images: 12
Server Version: 1.13.1-cs1
Storage Driver: overlay
 Backing Filesystem: xfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: likikti3440spjnij1stnpppj
 Is Manager: true
 ClusterID: jmga31ro7rf5vz2dm4k2glje1
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 172.31.44.44
 Manager Addresses:
  172.31.44.44:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-514.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.3 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 991.2 MiB
Name: ip-172-31-44-44.eu-west-1.compute.internal
ID: PCWN:OS2N:CFMX:CWZY:ATUN:2CRX:UPRY:VGMW:JR2W:BX4B:WBAZ:NCM2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

I am running ec2 servers on  AWS.

@msavdert prior to trying to uninstall, if you do docker service ls do you see the ucp-agent service running?

@duglin Thanks for the quick response. I found a workaround like this. I removed all ucp containers with docker rm -f . After i had a clean setup as below, it improved.
docker run --rm -it --name ucp
-v /var/run/docker.sock:/var/run/docker.sock
docker/ucp install
--host-address
--interactive

@msavdert hmm ok I guess.  Can we close this issue then?

Yes. Thank you.

ok will do - thanks

authors: duglin msavdert


text:
SwarmKit in docker does not validate hosts parameters in a service create request

Description
I foundt that SwarmKit in docker does not validate hosts parameters in a service create request.
Steps to reproduce the issue:

use api to create a service, in the service spec, we add parameter of Hosts.
in parameter, we add an invalid value, like aaaa, valid value should has a format of name:ip
send the request to create this service

request like the following pic:

Describe the results you received:
Service created OK
Describe the results you expected:
parameter invalid
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
root@ubuntu:~# docker version
Client:
 Version:      17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 10:57:47 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:        Thu Feb 23 10:57:47 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:
root@ubuntu:~# docker info
Containers: 3
 Running: 3
 Paused: 0
 Stopped: 0
Images: 116
Server Version: 17.03.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 148
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: vfb7vr7wp1dirmop8fbvmmgh7
 Is Manager: true
 ClusterID: mzzbi2s44h4vesargizzywryz
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 192.168.59.103
 Manager Addresses:
  192.168.59.103:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
 apparmor
Kernel Version: 3.19.0-25-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.954 GiB
Name: ubuntu
ID: FXOY:JCOY:HKDI:VO5U:FYDM:UEXV:YIFN:AISM:NR6U:VMW5:V4MZ:RQWF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 52
 Goroutines: 154
 System Time: 2017-03-27T14:28:38.061668428+08:00
 EventsListeners: 3
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

reopening, because https://github.com/docker/swarmkit/issues/2058 was not vendored yet

authors: allencloud thaJeztah


text:
reset bufReader to avoid extreme growth of buffers

This PR is an attempt to help solve problems like #9139 and #8502.
This PR changes bufReader to try to avoid the growth of bytes.Buffer to extreme sizes (hundreds of megabytes or even gigabytes). This makes long running containers which need to log to the console more reliable over time.
Please note that this PR only fixes this problem for now. Proper logging support is still needed.
Fixes #9139
Fixes #8539
Fixes #8502

ping @LK4D4

I tested - this fixing problem. I'm not reviewed code though.

@LK4D4 Thanks for trying it out!

This PR fixes 3 issues and it needs to be reviewed so that I can make any necessary changes after review and feedback.
ping @crosbymichael @dmcgowan @duglin @erikh @icecrime @jfrazelle @jlhawn @LK4D4 @tiborvass @vbatts

looks good, the branching logic for reset should be cleaned up for readability and reduce the if blocks as Josh suggested

@LK4D4 I've made some of the changes, but I'd like to keep those numbers as 64 ints everywhere to avoid having to sprinke int64(var) in some places.
@stevvooe There's a description now. Please take a look and let me know if it looks good to you.

I've updated the PR to remove the truncate.

@unclejack should this be moved to the 1.6 milestone? iiuc, there won't be a 1.5.1 release?
Also wondering if something is holding up this PR (apart from time constraint), as it seems to be resolving quite some issues :+1:

I've rebased this PR on master and I'm using a binary built from this branch as my Docker daemon.
@thaJeztah It needs more testing and reviews at this point. This was the simplest change which could have been made to fix this problem without hurting performance after trying many other options. There's not much left to change regarding the design itself.

@unclejack Thanks for the explanation. So, this is effectively a buffer reader that does some read ahead?

:+1:

@stevvooe Yes, it does read ahead so that it doesn't block the other readers by avoiding a reader to take too long to read everything.
We need to figure out what else needs to be done for this and get it merged. Many users are affected by the problem this PR fixes.

The PR has been rebased on master and comments have been added.
@tiborvass PTAL

This PR needs to be in 1.6.0.

LGTM

LGTM

authors: LK4D4 dmcgowan jfrazelle lqez stevvooe thaJeztah unclejack