text:
Fix networkRange race condition and global state corruption

While the networkRange function calculates lastIP, it uses
net.IPv4zero or net.IPv6zero as a starting value from which it
builds the final last IP. The issue is that these are global
variables defined in the standard library, and modifying them affects
all future users of those variables across the program.
This leads to a not-so-rare race condition when creating multiple
libvirt networks. These networks are being created concurrently and they
both race modifying that global variable. Their lastIP DHCP
configuration gets mixed up as result. The manner in which the
configuration gets mixed up is unpredictable, but it could lead to a
corrupt configuration that cannot be applied.
The solution is to copy the zero IP addresses to a new variable rather
than use them directly. Instead for simplicity I just instantiate an
empty slice with the same length as the net IP, which would be filled
with zeroes anyway.
This commit also solves another unrelated bug in getNetworkIPConfig
where the ^ operator was used as if it's a power operator, while in
reality it's a bitwise-xor that leads to slightly incorrect results.
This makes the validation only slightly wrong and is overly strict (e.g.
it will not allow you to create /28 IPv4 network even though such
network has far more available addresses than the condition blocks)
A similar race condition can be simply reproduced and visualized with
this short go code:
package main

import (
	"fmt"
	"net"
	"sync"
)

func getNetMaskWithMax16Bits(m net.IPMask) net.IPMask {
	ones, bits := m.Size()
	if bits-ones > 16 {
		if bits == 128 {
			return net.CIDRMask(128-16, 128)
		}
		return net.CIDRMask(32-16, 32)
	}
	return m
}

func networkRange(network *net.IPNet) (net.IP, net.IP) {
	netIP := network.IP.To4()
	lastIP := net.IPv4zero.To4()
	if netIP == nil {
		netIP = network.IP.To16()
		lastIP = net.IPv6zero.To16()
	}
	firstIP := netIP.Mask(network.Mask)
	intMask := getNetMaskWithMax16Bits(network.Mask)
	for i := 0; i < len(lastIP); i++ {
		lastIP[i] = netIP[i] | ^intMask[i]
	}
	return firstIP, lastIP
}

func update(wg *sync.WaitGroup, cidr string, id int) {
	address := cidr
	_, ipNet, _ := net.ParseCIDR(address)
	start, end := networkRange(ipNet)
	start[len(start)-1]++
	start[len(start)-1]++
	end[len(end)-1]--
	fmt.Printf("Start %d: %s\n", id, start.String())
	fmt.Printf("End %d: %s\n", id, end.String())
	wg.Done()
}

func main() {
	var wg sync.WaitGroup
	wg.Add(2)
	go update(&wg, "192.168.145.0/24", 0)
	go update(&wg, "192.168.127.0/24", 1)
	wg.Wait()
}

Then run:
watch -n0.1 -d go run main.go

To see it happen for short moments.
Alternatively, create this main.tf file:
terraform {
  required_providers {
    libvirt = {
      source = "dmacvicar/libvirt"
      version = "0.6.14"
    }
  }
}

provider "libvirt" {
  uri = "qemu:///system"
}

resource "libvirt_network" "net" {
  name = "omer-net"
  mode = "nat"
  addresses = ["192.168.127.0/24"]
  autostart = true
}

resource "libvirt_network" "secondary_net" {
  name = "omer-second-net"
  mode = "nat"
  addresses = ["192.168.145.0/24"]
  autostart = true
}

And run:
while true; do
    if ! (terraform apply -auto-approve && terraform destroy -auto-approve); then
        break
    fi
done

Until the bug reproduces with the following error:
│ Error: error defining libvirt network: internal error: range
192.168.127.2 - 192.168.145.254 is not entirely within network
    192.168.127.1/24 -   <network>
    │       <name>omer-net</name>
    │       <forward mode="nat"></forward>
    │       <bridge stp="on"></bridge>
    │       <dns enable="no"></dns>
    │       <ip address="192.168.127.1" family="ipv4" prefix="24">
    │           <dhcp>
    │               <range start="192.168.127.2"
    end="192.168.145.254"></range>
    │           </dhcp>
    │       </ip>
    │       <options
    xmlns="http://libvirt.org/schemas/network/dnsmasq/1.0"></options>
    │   </network>
    │
    │   with libvirt_network.net,
    │   on main.tf line 14, in resource "libvirt_network" "net":
    │   14: resource "libvirt_network" "net" {

Please make sure you read the contributor documentation before opening a Pull Request.

Maintainers, can we merge this?  This bug has been biting me semi-regularly for months.  Thanks for tracking it down @omertuc !

@dmacvicar

Thanks for the patch.

authors: dmacvicar ericroy omertuc osherdp

repo: dmacvicar/terraform-provider-libvirt


text:
MixedModels, no method matching mul<U+03B1><U+03B2>!

Got an error:
no method matching mul<U+03B1><U+03B2>!(::Array{Float64,2}, ::BlockedSparse{Float64,Int32}, ::Adjoint{Float64,Array{Float64,2}}, ::Float64, ::Float64)
Closest candidates are:
mul<U+03B1><U+03B2>!(::Union{DenseArray{T<:Union{Complex{Float32}......
with formula PRICE ~ (Y|ASSESSMENT_NBHD)+(LANDAREA|ROOMS)+(YR_RMDL|QUADRANT)
Any help is much appreciated!
Dataset:
dataset_mult_err.csv.zip

I'll look at the problem with the MixedModels code but first there are two things to note:

you have an extreme outlier on PRICE at row number 33109 in the file where PRICE is 1
as I mentioned previously, if you have a term like (Y|ASSESSMENT_NBHD) in the formula you should also have Y in the formula.

First, there is a method missing and I will add it.
I do want to point out that your model specifications are very peculiar and this is why these issues are cropping up.  Consider that QUADRANT has 4 levels.  A term like (YR_RMDL|QUADRANT), which is equivalent to (1 + YR_RMDL|QUADRANT) says to fit two variances and a covariance to information from 4 quadrants.  Furthermore, you do not have a fixed-effect for YR_RMDL so the random-effects for slope are going to be centered about 0.
Also, ROOMS as a grouping factor for a random effect is very peculiar.
If you know the terminology, ASSESSMENT_NBHD is nearly nested within QUADRANT but apparently not completely nested.  Some neighbourhoods overlap with two quadrants and one even overlaps with 3.  This causes a sparse structure in the [3,1] block but a dense structure in the [3,2] (ROOMS by QUADRANT)
julia> describeblocks(m)
1,1: UniformBlockDiagonal{Float64} (110, 110) UniformBlockDiagonal{Float64}
2,1: Array{Float64,2} (58, 110) Array{Float64,2}
2,2: UniformBlockDiagonal{Float64} (58, 58) Array{Float64,2}
3,1: SparseArrays.SparseMatrixCSC{Float64,Int32} (8, 110) BlockedSparse{Float64,Int32}
3,2: Array{Float64,2} (8, 58) Array{Float64,2}
3,3: UniformBlockDiagonal{Float64} (8, 8) Array{Float64,2}
4,1: Array{Float64,2} (4, 110) Array{Float64,2}
4,2: Array{Float64,2} (4, 58) Array{Float64,2}
4,3: Array{Float64,2} (4, 8) Array{Float64,2}
4,4: Array{Float64,2} (4, 4) Array{Float64,2}
5,1: Array{Float64,2} (1, 110) Array{Float64,2}
5,2: Array{Float64,2} (1, 58) Array{Float64,2}
5,3: Array{Float64,2} (1, 8) Array{Float64,2}
5,4: Array{Float64,2} (1, 4) Array{Float64,2}
5,5: Array{Float64,2} (1, 1) Array{Float64,2}

If you plot the data, say PRICE versus Y, you will see that there are extreme outliers in PRICE.  You almost certainly want to look at log(PRICE)

Thank you for pointing that out. I used this dataset for only one purpose - to reproduce the error that I had on a real data (that I couldn't share). So I didn't pay much attention to formula notation in this case. There is a chance that my original data has the same or similair problems though, so I'll look closely into that.
Really appreciate you feedbacks, thanks a lot!
Will you add both methods for mul and rankupdate? Could you probably give an approximate deadline when are you planning on doing that?

source: gharchive/issue

authors: dmbates marynavickt

license: mit

license_type: permissive

license_source: bigquery


text:
Updated Duplicati version

Bumped Duplicati build from 2016-09-29 to 2016-10-19

@rahuljawale thank you.

Thanks buddy! That was quick! :+1:

source: gharchive/pull-request

authors: dmitryint rahuljawale

repo: dmitryint/docker-duplicati


text:
BERT example crashes when using MXNet naive engine

It works well when MXNET_ENGINE_TYPE is unset.
Environment:
(gluonnlp-py3) [lvtao1@mlt-gpu207 bert]$ pip list
Package    Version        Location
---------- -------------- --------------------------------------
certifi    2019.3.9
chardet    3.0.4
gluonnlp   0.6.0          /home/lvtao1/Workspace/gluon-nlp-3/src
graphviz   0.8.4
idna       2.8
mxnet      1.5.0b20190525
numpy      1.14.6
pip        19.0.3
requests   2.21.0
setuptools 41.0.0
six        1.12.0
urllib3    1.24.2
wheel      0.33.1

Error log:
(gluonnlp-py3) [lvtao1@mlt-gpu207 bert]$ python finetune_classifier.py --task_name MRPC --batch_size 32 --optimizer bertadam --epochs 1 --lr 2e-5 --max_len 128 --only_inference --model_parameter ./output_dir/model_bert_MRPC_2.params --pad --dev_batch_size 1
[10:52:25] src/engine/engine.cc:55: MXNet start using engine: NaiveEngine
INFO:root:Namespace(accumulate=None, batch_size=32, bert_dataset='book_corpus_wiki_en_uncased', bert_model='bert_12_768_12', dev_batch_size=1, epochs=1, epsilon=1e-06, gpu=None, log_interval=10, lr=2e-05, max_len=128, model_parameters='./output_dir/model_bert_MRPC_2.params', only_inference=True, optimizer='bertadam', output_dir='./output_dir', pad=True, pretrained_bert_parameters=None, seed=2, task_name='MRPC', warmup_ratio=0.1)
INFO:root:loading model params from ./output_dir/model_bert_MRPC_2.params
<model architecture>
INFO:root:processing dataset...
INFO:root:Now we are doing evaluation on dev with cpu(0).
INFO:root:[Batch 10/408] loss=0.4793, metrics:f1:0.6000,accuracy:0.9000
INFO:root:[Batch 20/408] loss=0.3724, metrics:f1:0.5500,accuracy:0.8500
INFO:root:[Batch 30/408] loss=0.3786, metrics:f1:0.5667,accuracy:0.8333
INFO:root:[Batch 40/408] loss=0.5535, metrics:f1:0.5750,accuracy:0.8500
INFO:root:[Batch 50/408] loss=0.0884, metrics:f1:0.5600,accuracy:0.8600
INFO:root:[Batch 60/408] loss=0.2810, metrics:f1:0.6167,accuracy:0.8667
INFO:root:[Batch 70/408] loss=0.0490, metrics:f1:0.6286,accuracy:0.8857
INFO:root:[Batch 80/408] loss=0.3837, metrics:f1:0.6625,accuracy:0.8875
INFO:root:[Batch 90/408] loss=0.4702, metrics:f1:0.6556,accuracy:0.8778
INFO:root:[Batch 100/408] loss=0.6682, metrics:f1:0.6600,accuracy:0.8700
INFO:root:[Batch 110/408] loss=0.0228, metrics:f1:0.6636,accuracy:0.8818
INFO:root:[Batch 120/408] loss=0.4268, metrics:f1:0.6750,accuracy:0.8833
INFO:root:[Batch 130/408] loss=0.0172, metrics:f1:0.6846,accuracy:0.8923
INFO:root:[Batch 140/408] loss=0.5646, metrics:f1:0.6714,accuracy:0.8929
INFO:root:[Batch 150/408] loss=0.8041, metrics:f1:0.6600,accuracy:0.8800
INFO:root:[Batch 160/408] loss=0.3771, metrics:f1:0.6625,accuracy:0.8812
INFO:root:[Batch 170/408] loss=0.0108, metrics:f1:0.6706,accuracy:0.8882
Traceback (most recent call last):
  File "finetune_classifier.py", line 539, in <module>
    train(task.metrics)
  File "finetune_classifier.py", line 471, in train
    metric_nm, metric_val = evaluate(dev_data, metric, segment)
  File "finetune_classifier.py", line 511, in evaluate
    for batch_id, seqs in enumerate(loader_dev):
  File "/home/lvtao1/miniconda3/envs/gluonnlp-py3/lib/python3.6/site-packages/mxnet/gluon/data/dataloader.py", line 450, in __next__
    batch = pickle.loads(ret.get()) if self._dataset is None else ret.get()
  File "/home/lvtao1/miniconda3/envs/gluonnlp-py3/lib/python3.6/site-packages/mxnet/gluon/data/dataloader.py", line 57, in rebuild_ndarray
    fd = fd.detach()
  File "/home/lvtao1/miniconda3/envs/gluonnlp-py3/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
  File "/home/lvtao1/miniconda3/envs/gluonnlp-py3/lib/python3.6/multiprocessing/reduction.py", line 182, in recv_handle
    return recvfds(s, 1)[0]
  File "/home/lvtao1/miniconda3/envs/gluonnlp-py3/lib/python3.6/multiprocessing/reduction.py", line 161, in recvfds
    len(ancdata))
RuntimeError: received 0 items of ancdata

Segmentation fault: 11

Stack trace:
  [bt] (0) /home/lvtao1/miniconda3/envs/gluonnlp-py3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2b3d7b0) [0x7f2016bcf7b0]
  [bt] (1) /usr/lib64/libc.so.6(+0x35270) [0x7f2067a02270]
  [bt] (2) /home/lvtao1/miniconda3/envs/gluonnlp-py3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x233ba47) [0x7f20163cda47]
  [bt] (3) /home/lvtao1/miniconda3/envs/gluonnlp-py3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x233beea) [0x7f20163cdeea]
  [bt] (4) /home/lvtao1/miniconda3/envs/gluonnlp-py3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x233c095) [0x7f20163ce095]
  [bt] (5) /home/lvtao1/miniconda3/envs/gluonnlp-py3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x233a175) [0x7f20163cc175]
  [bt] (6) /home/lvtao1/miniconda3/envs/gluonnlp-py3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x233a99b) [0x7f20163cc99b]
  [bt] (7) /home/lvtao1/miniconda3/envs/gluonnlp-py3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2b431f0) [0x7f2016bd51f0]
  [bt] (8) /home/lvtao1/miniconda3/envs/gluonnlp-py3/bin/../lib/libstdc++.so.6(+0x9b64a) [0x7f205d3ef64a]

@zhreshold does data loader conflict with naive engine?

I think it might indicate some potential bugs in naive engine, especially after fork.

authors: TaoLv eric-haibin-lin zhreshold

repo: dmlc/gluon-nlp

license_source: github-api


text:
[request] model trained on wikipedia articles

The pre-trained imagenet model is super useful.  A model pre-trained on wikipedia (or another large text corpus) would be super useful too!

Sample  usage: https://github.com/dmlc/mxnet/blob/master/example/notebooks/predict-with-pretrained-model.ipynb

authors: antinucleon zachmayer

repo: dmlc/mxnet-model-gallery


text:
Why

A: I got a poor ACC and it still betwwen 0 ---0.005,
B: when i am running, i got a warning " ...use mx.mod.module ..instead mx.model.FeedFoward...", i don't know how to use "mx.mod.module"
C: I found use im2rec with recursive ture or false all lead the same label,and when i use shuffle false,it also can't work, i got a randomize file.
result info:
2017-03-17 09:21:13,137 Node[0] Epoch[18] Batch [17300]	Speed: 258.90 samples/sec	Train-top_k_accuracy_10=0.001250
2017-03-17 09:21:13,137 Node[0] Epoch[18] Batch [17300]	Speed: 258.90 samples/sec	Train-top_k_accuracy_20=0.003750
2017-03-17 09:21:16,237 Node[0] Epoch[18] Batch [17350]	Speed: 258.04 samples/sec	Train-accuracy=0.000000
2017-03-17 09:21:16,237 Node[0] Epoch[18] Batch [17350]	Speed: 258.04 samples/sec	Train-top_k_accuracy_5=0.001250
2017-03-17 09:21:16,238 Node[0] Epoch[18] Batch [17350]	Speed: 258.04 samples/sec	Train-top_k_accuracy_10=0.001250
2017-03-17 09:21:16,238 Node[0] Epoch[18] Batch [17350]	Speed: 258.04 samples/sec	Train-top_k_accuracy_20=0.002500
2017-03-17 09:21:19,317 Node[0] Epoch[18] Batch [17400]	Speed: 259.79 samples/sec	Train-accuracy=0.000000
2017-03-17 09:21:19,317 Node[0] Epoch[18] Batch [17400]	Speed: 259.79 samples/sec	Train-top_k_accuracy_5=0.000000
2017-03-17 09:21:19,317 Node[0] Epoch[18] Batch [17400]	Speed: 259.79 samples/sec	Train-top_k_accuracy_10=0.000000
2017-03-17 09:21:19,317 Node[0] Epoch[18] Batch [17400]	Speed: 259.79 samples/sec	Train-top_k_accuracy_20=0.001250
2017-03-17 09:21:22,382 Node[0] Epoch[18] Batch [17450]	Speed: 261.07 samples/sec	Train-accuracy=0.000000
2017-03-17 09:21:22,382 Node[0] Epoch[18] Batch [17450]	Speed: 261.07 samples/sec	Train-top_k_accuracy_5=0.000000
2017-03-17 09:21:22,382 Node[0] Epoch[18] Batch [17450]	Speed: 261.07 samples/sec	Train-top_k_accuracy_10=0.000000
this is my list file :
198956	2502.000000	2077/00000535.jpg
26282	393.000000	10350/028.jpg
203015	2571.000000	2139/168.jpg
246952	3195.000000	2700/066.jpg
674824	9463.000000	8342/027.jpg
181413	2256.000000	1856/004.jpg
512786	7096.000000	6211/00000848.jpg
145517	1730.000000	1382/00000525.jpg
738795	10317.000000	9110/008.jpg
162113	1966.000000	1595/046.jpg
mycode
from common import find_mxnet
import mxnet as mx
import argparse
import os, sys
from common import train_model
def googlenet():
def ConvFactory(data, num_filter, kernel, stride=(1,1), pad=(0, 0), name=None, suffix=''):
conv = mx.symbol.Convolution(data=data, num_filter=num_filter, kernel=kernel, 				stride=stride,pad=pad, name='conv_%s%s' %(name, suffix))
		 act = mx.symbol.Activation(data=conv, act_type='relu', name='relu_%s%s' %(name, suffix))
		 return act

def InceptionFactory(data, num_1x1, num_3x3red, num_3x3, num_d5x5red, num_d5x5, pool, proj, name):
# 1x1
 c1x1 = ConvFactory(data=data, num_filter=num_1x1, kernel=(1, 1), name=('%s_1x1' % name))

# 3x3 reduce + 3x3
 c3x3r = ConvFactory(data=data, num_filter=num_3x3red, kernel=(1, 1), name=('%s_3x3' % name), 			suffix='_reduce')
 c3x3 = ConvFactory(data=c3x3r, num_filter=num_3x3, kernel=(3, 3), pad=(1, 1), name=('%s_3x3' % name))
# double 3x3 reduce + double 3x3
 cd5x5r = ConvFactory(data=data, num_filter=num_d5x5red, kernel=(1, 1), name=('%s_5x5' % name), 		suffix='_reduce')
 cd5x5 = ConvFactory(data=cd5x5r, num_filter=num_d5x5, kernel=(5, 5), pad=(2, 2), name=('%s_5x5' % 			name))
# pool + proj
 pooling = mx.symbol.Pooling(data=data, kernel=(3, 3), stride=(1, 1), pad=(1, 1 ), pool_type=pool, 			name=('%s_pool_%s_pool' % (pool, name)))
 cproj = ConvFactory(data=pooling, num_filter=proj, kernel=(1, 1), name=('%s_proj' %  name))
# concat
 concat = mx.symbol.Concat(*[c1x1, c3x3, cd5x5, cproj], name='ch_concat_%s_chconcat' % name)
 return concat

#######################
def InceptionFactory2(data, num_3x3red, num_3x3, num_d5x5red, num_d5x5, pool, proj, name):
# 3x3 reduce + 3x3
 c3x3r = ConvFactory(data=data, num_filter=num_3x3red, kernel=(1, 1), name=('%s_3x3' % name), 			suffix='_reduce')

 c3x3 = ConvFactory(data=c3x3r, num_filter=num_3x3, kernel=(3, 3), stride=(2, 2), pad=(1, 1), name=('%s_3x3' % name))
# double 3x3 reduce + double 3x3
	 cd5x5r = ConvFactory(data=data, num_filter=num_d5x5red, kernel=(1, 1), name=('%s_5x5' % name), suffix='_reduce')
 cd5x5 = ConvFactory(data=cd5x5r, num_filter=num_d5x5, kernel=(5, 5),stride=(2, 2),  pad=(2, 2), name=('%s_5x5' % name))
# pool + proj
 pooling = mx.symbol.Pooling(data=data, kernel=(3, 3), stride=(2, 2), pad=(1, 1 ), pool_type=pool, name=('%s_pool_%s_pool' % (pool, name)))

# concat
 concat = mx.symbol.Concat(*[c3x3, cd5x5, pooling], name='ch_concat_%s_chconcat' % name)
 return concat

data  = mx.sym.Variable("data")
conv1 = ConvFactory(data, 64, kernel=(7, 7), stride=(2,2), pad=(3, 3), name="conv1")

pool1 = mx.sym.Pooling(conv1, kernel=(3, 3), stride=(2, 2), pool_type="max")
conv2 = ConvFactory(pool1, 64, kernel=(1, 1), stride=(1,1), name="conv2")
conv3 = ConvFactory(conv2, 192, kernel=(3, 3), stride=(1, 1), pad=(1,1), name="conv3")
pool3 = mx.sym.Pooling(conv3, kernel=(3, 3), stride=(2, 2), pool_type="max")

in3a = InceptionFactory(pool3, 64, 96, 128, 16, 32, "max", 32, name="in3a")
in3b = InceptionFactory(in3a, 64, 96, 128, 32, 64, "max", 64, name="in3b")

#######################
in3c = InceptionFactory2(in3b, 128, 256, 32, 64, "max", 64, name="in3c")
########################
#pool4 = mx.sym.Pooling(in3c, kernel=(3, 3), stride=(2, 2),pad=(1, 1), pool_type="max")
in4a = InceptionFactory(in3c, 256, 96, 192, 32, 64, "max", 64, name="in4a")
in4b = InceptionFactory(in4a, 224, 112, 224, 32, 64, "max", 64, name="in4b")
in4c = InceptionFactory(in4b, 192, 128, 256, 32, 64, "max", 64, name="in4c")
in4d = InceptionFactory(in4c, 160, 144, 288, 32, 64, "max", 64, name="in4d")
in4e = InceptionFactory2(in4d, 160, 256, 64, 128, "max", 128, name="in4e")

pool5 = mx.sym.Pooling(in4e, kernel=(3, 3), stride=(2, 2), pool_type="max")
in5a = InceptionFactory(in4e,384, 192, 384, 48, 128, "max", 128, name="in5a")
in5b = InceptionFactory(in5a, 384, 192, 384, 48, 128, "max", 128, name="in5b")
pool6 = mx.sym.Pooling(in5b, kernel=(7, 7), stride=(1,1), pool_type="avg")
flatten = mx.sym.Flatten(data=pool6)
fc1 = mx.sym.FullyConnected(data=flatten, num_hidden=1000)

softmax = mx.symbol.SoftmaxOutput(data=fc1, name='softmax')

return softmax

def get_iterator(args, kv):
data_shape = (3, 224, 224)
train = mx.io.ImageRecordIter(
path_imgrec = "/media/pixel/ssd512G/tran.rec",
mean_img ="/media/pixel/ssd512G/mean2.bin",
data_name ='data',
shuffle = True,
data_shape  = data_shape,
batch_size  = args.batch_size,
rand_crop   = True,
rand_mirror = True,
num_parts   = kv.num_workers,
part_index  = kv.rank)
val = mx.io.ImageRecordIter(
    path_imgrec = "/media/pixel/ssd512G/tran0.rec",
    mean_img ="/media/pixel/ssd512G/mean21.bin",
data_name='data',
shuffle = False,
    rand_crop   = False,
    rand_mirror = False,
    data_shape  = data_shape,
    batch_size  = args.batch_size,
    num_parts   = kv.num_workers,
    part_index  = kv.rank)

return (train, val)

def parse_args():
parser = argparse.ArgumentParser(description='train an image classifer on mnist')
parser.add_argument('--network', type=str, default='googlenet',
help = 'the cnn to use')
parser.add_argument('--data-dir', type=str, default='/media/pixel/ssd512G/',
help='the input data directory')
parser.add_argument('--gpus', type=str,default='0,1',
help='the gpus will be used, e.g "0,1,2,3"')
parser.add_argument('--num-examples', type=int, default=5359,
help='the number of training examples')
parser.add_argument('--batch-size', type=int, default=16,
help='the batch size')
parser.add_argument('--lr', type=float, default=.01,
help='the initial learning rate')
parser.add_argument('--model-prefix', type=str,
help='the prefix of the model to load/save')
parser.add_argument('--save-model-prefix', type=str,
help='the prefix of the model to save')
parser.add_argument('--num-epochs', type=int, default=2000,
help='the number of training epochs')
parser.add_argument('--load-epoch', type=int,
help="load the model on an epoch using the model-prefix")
parser.add_argument('--kv-store', type=str, default='local',
help='the kvstore type')
parser.add_argument('--lr-factor', type=float, default=0.5,
help='times the lr with a factor for every lr-factor-epoch epoch')
parser.add_argument('--lr-factor-epoch', type=float, default=1,
help='the number of epoch to factor the lr, could be .5')
return parser.parse_args()
if name == 'main':
args = parse_args()
net =googlenet()
# train
train_model.fit(args, net, get_iterator)

Environment info
Operating System: Ubuntu
Package used (Python/R/Scala/Julia): Python
MXNet version: 0.9.3

I have used lr==0.01 lr=0.05 lr=0.001 batchsize =32 or 64 , i think  my fault is not about my para.

now i  think i got the right result with im2rec.py in 0.8.0,  the wrong all beacasue of my awful data

Closed

@Kdecem Please redefine the problem according to the description, which may help more people like you, thanks :)

repo: dmlc/mxnet


text:
Padding GPU allocations

Padding GPU allocations - a first step to allow scattering arrays across GPUs for NCCL support.

Ok. Let's use some constant like 32 or 64.
My only concern is this will prevent cuda-memcheck from catching some bugs.

Can one of the admins verify this patch?

authors: piiswrong ptrendx


text:
Add dmlc-core to the list of installed header directories.

There are dependencies on dmlc-core in TVM public API headers
(e.g. some headers include dmlc/logging.h) so it needs to be installed
as part of TVM for TVM headers to be actually usable.

Thanks @ndl

authors: ndl tqchen

repo: dmlc/tvm


text:
Rank:pairwise  instance weight not supported?

Hi,
I am trying to use lambdamart for pairwise rank.
Refer to here:https://github.com/dmlc/xgboost/blob/master/doc/input_format.md#instance-weight-file.  I can give the instance weight in xxx.weight file
After I set the weight for some instance, I found the model, however,  still look the same compared to the model trained by all instance weighted the same.
I check out the code in rank_obj.cc and I do not find the weight is used to construct or set the pair.
So...Could you tell me the if the instance weight is supported for LambdaMart?
Thanks!
Attach the code I've checked:
// get lambda weight for the pairs
        this->GetLambdaWeight(lst, &pairs);
        // rescale each gradient and hessian so that the lst have constant weighted
        float scale = 1.0f / param_.num_pairsample;
        if (param_.fix_list_weight != 0.0f) {
          scale *= param_.fix_list_weight / (gptr[k + 1] - gptr[k]);
        }
        for (size_t i = 0; i < pairs.size(); ++i) {
          const ListEntry &pos = lst[pairs[i].pos_index];
          const ListEntry &neg = lst[pairs[i].neg_index];
          const bst_float w = pairs[i].weight * scale;
          const float eps = 1e-16f;
          bst_float p = common::Sigmoid(pos.pred - neg.pred);
          bst_float g = p - 1.0f;
          bst_float h = std::max(p * (1.0f - p), eps);
          // accumulate gradient and hessian in both pid, and nid
          gpair[pos.rindex].grad += g * w;
          gpair[pos.rindex].hess += 2.0f * w * h;
          gpair[neg.rindex].grad -= g * w;
          gpair[neg.rindex].hess += 2.0f * w * h;
}
...
class PairwiseRankObj: public LambdaRankObj{
 protected:
  void GetLambdaWeight(const std::vector<ListEntry> &sorted_list,
                       std::vector<LambdaPair> *io_pairs) override {}
};

Thanks @shengyu-guo for reporting this issue, https://github.com/dmlc/xgboost/pull/3379 PR should solve it, which is now merged to master.

repo: dmlc/xgboost


text:
Crash when xgb.cv is run for some time with GPU support

Hi,
I'm running with GPU support on, and the system crashes sometimes with the following message. (Logs attached)
python': malloc(): smallbin double linked list corrupted
Configuration:
'gpu_id': 0,
'max_bin': 16,
'tree_method': 'gpu_hist',
Environment info
Operating System:
Ubuntu 16.04.1
Compiler:
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.5)
nvcc: NVIDIA (R) Cuda compiler driver
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61
Package used (python/R/jvm/C++):
python
xgboost version used:
xgboost-0.6-py3.6.
If installing from source, please provide

The commit hash (git rev-parse HEAD)
9a81c74a7bd8db06c2f2ccb28b88ffeecf4c0309
Logs will be helpful (If logs are large, please upload as attachment).

If you are using jvm package, please

add [jvm-packages] in the title to make it quickly be identified
the gcc version and distribution

If you are using python package, please provide

The python version and distribution
Anaconda, Python 3.6
The command to install xgboost if you are not installing from source
(from source)

Steps to reproduce
Extremely difficult, happens randomly after a long run.
What have you tried?
Any advice? Not sure what to try even.
error.txt

According to the stack trace the issue originated from gnu parallel sort during the AUC metric calculation. Does the error still occur if you use the 'error' evaluation metric instead of AUC?

Well, I think I'm one of those affected by the not so rare AMD Ryzen bug where when stressed, programs segfault randomly. I bricked my mobo trying to get a new version of the BIOS. Finally got the RMA and tested the code after disabling opcache and SMT. So far so good.. Have been running the same code close to an hour and no segfaults yet :)

authors: RAMitchell thusithaC


text:
Make sure MaxRSS and Memory are settable by Task

In the same way we need to schedule cores task by task we need to do the same with the Memory requested and the MaxRSS used for killing jobs on the WN.

This is the commit for the multicore stuff
https://github.com/ericvaandering/WMCore/commit/204a1f7074b8b0bf2774f26e886f712f778c79d1

repo: dmwm/WMCore


text:
[py2py3] Migration at level scr/python/A/B/C - slice 17

Fixes #10141
Status
Ready
Related PRs

Previous migration PR: #10321  (slice16, issue #10140)
Next migration PR: #10331  (slice18, issue #10142  )

Description
Run futurize and some manual changes on the first batch of src/python/A/B/C.
"native string" approach

src/python/WMCore/FwkJobReport/Report.py: imported str from builtins, but used only in a narrow case.

addError: refactored. See #8403 for previous work
refactored heavily

src/python/WMCore/WMRuntime/Scripts/SetupCMSSWPset.p: there is only one line where this may be problematic: inputTypeAttrib.fileNames.append(str(fileLFN)), but I do not think that there is any incentive to try to change it at this stage

Is it backward compatible (if not, which system it affects?)
It should be. (Any possible cause for errors will we reported here)
External dependencies / deployment changes
Requires python-future in both py2 and py3 environments.

One small treat: instead of ignoring errors, we could replace the errors with the unicode character � so that it is clear in the logs (to these error details end up in logs?) when and where there is some malformed byte.

@mapellidario Dario, IMO we should simply drop those malformed characteres. It will usually be part of a long (or very long) error information that happened during job runtime and I don't see much value in keeping the question mark char in place.
What do you think?

Yeah sure! I mean, the last call is always up to you! I made a new commit replacing 'replace' with 'ignore' and squashed the review commits! Now this is all for you to review (again)!

@amaltaro you are the one that should make the last call! I replaced replace with ignore!
I pushed two new commits that contain all the changes after your first review, these are ready to be review as well! Thanks again for the feedback!

Thanks Dario. Last 2 commits look good to me as well. Can you please squash them accordingly? Thanks

Thanks Dario.

authors: amaltaro mapellidario


text:
MSOutput: do not fail workflow if there are containers unknown to Rucio

Fixes #10368
Status
not-tested
Description
This PR provides two changes:

bug-fix to producing/consuming workflows where not all the output datasets have been produced and published to DBS/Rucio. In the current state, MSOutput keeps trying to produce them over and over, forever!
create an alert (email + slack notification) via AlertManager Prometheus system, for each workflow with their list of missing output datasets.

Is it backward compatible (if not, which system it affects?)
yes
Related PRs
none
External dependencies / deployment changes
none

FYI @haozturk

test this please

test this please

Testing this on my VM will be quite time consuming. So I'm going to merge it and use testbed to run the final testing.
Review and comments are welcome at any time though.


text:
Assign test_authz_key in unit test

This recipe may fix some of the failed unit tests.

Beginning pylint and unit tests.
See https://cmssdt.cern.ch/jenkins/job/DMWM-PR-test/438/ for details

Thanks Valentin,
I will merge this when the test finished.

And if Valentin fixes 10 tests in one go, I owe him a beer. Or vodka.

No pylint warnings for pull request 6180:

Unit test changes for pull request 6180:

WMCore_t.WorkQueue_t.Policy_t.Start_t.MonteCarlo_t.MonteCarloTestCase:testLHEProductionWorkload changed from failure to success
WMCore_t.WorkQueue_t.Policy_t.Start_t.MonteCarlo_t.MonteCarloTestCase:testShiftedStartSplitting changed from failure to success

The PR fixed the unit test test_accept_reject_xfoo and couple of others.
I'll count on free beer.
I'll walk through test area and fix other unit tests which inherit this bug.
On  0, Eric Vaandering notifications@github.com wrote:

And if Valentin fixes 10 tests in one go, I owe him a beer. Or vodka.

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6180#issuecomment-135860235

Here PR for other two tests with similar issue: https://github.com/dmwm/WMCore/pull/6181

Jenkins say's it's still broken. Happy to be wrong, though.

Which version of cherrypy jenkins use? Where does it comes from?
The cherrypy version I have does not have testmain method of cherrypy.helper.
On  0, Eric Vaandering notifications@github.com wrote:

Jenkins say's it's still broken. Happy to be wrong, though.

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6180#issuecomment-135870460

It comes from deploying the latest WMAgent RPM.
On Aug 28, 2015, at 15:12, Valentin Kuznetsov <notifications@github.commailto:notifications@github.com> wrote:
Which version of cherrypy jenkins use? Where does it comes from?
The cherrypy version I have does not have testmain method of cherrypy.helper.
On 0, Eric Vaandering <notifications@github.commailto:notifications@github.com> wrote:

Jenkins say's it's still broken. Happy to be wrong, though.

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6180#issuecomment-135870460

—
Reply to this email directly or view it on GitHubhttps://github.com/dmwm/WMCore/pull/6180#issuecomment-135877116.
Eric W. Vaandering
Computing Physics Developer
CMS Computing Development Lead
Scientific Computing Division
Fermi National Accelerator Laboratory
630 840 6678 office
www.fnal.govhttp://www.fnal.gov/

Hi,
digging further with code I found that the unittests in WMCore relies on
cherrypy-3.1.2 (see helper.testmain()) while the RPMs we are testing use
cherrypy-3.2.2 which modified its helper class (there is no anymore testmain).
I modified Api_t.py to work with cherrypy-3.2.2 and it passed all tests in my
local environment. Can we take PR: https://github.com/dmwm/WMCore/pull/6183
and test jenkins with it. I intentionally added couple of print statements
to check what test is doing. Therefore please do not merge #6183 yet, instead I
need to see output from jenkins.
Valentin.
On  0, Eric Vaandering notifications@github.com wrote:

Jenkins say's it's still broken. Happy to be wrong, though.

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6180#issuecomment-135870460

Yes, the cherry py tests are a PITA. I looked at converting them to 3.2 and gave up. Thanks for picking this up.

@vkuznet, Valentin, I can just close this one without merging it, right?

yes
On  0, ticoann notifications@github.com wrote:

@vkuznet, Valentin, I can just close this one without merging it, right?

Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/6180#issuecomment-137197846

authors: DMWMBot ericvaandering ticoann vkuznet