text:
Monotonic Constraints can not be used inside an Scikit-Learn Pipeline without additional processing

Dear colleagues,
I have the following dataframe:

feat1
feat2
feat3
target

0.1
0
1
0.01

0.4
0
2
0.02

1.1
1
3
0.05

1.2
1
4
0.1

I have the following scikit learn pipeline:
numeric_transformer = Pipeline(
        steps=[
            ("imputer", SimpleImputer(strategy="constant",
             fill_value=-1, add_indicator=True)),
            # originally: ("imputer", SimpleImputer(strategy="mean")),
            ("scaler", StandardScaler()),
            ("variance_selector", VarianceThreshold(threshold=0.03))
        ]
    )

logging.info("Defining categorical transformer: one-hot encoding")
categorical_transformer = Pipeline(
    steps=[
        ("imputer", SimpleImputer(strategy="constant", fill_value="missing")),
        ("onehot", OneHotEncoder(handle_unknown="ignore"))]
)

preprocessor = ColumnTransformer(
    transformers=[
        ("numeric_transf", numeric_transformer, numerical_features),
        ("categorical_transf", categorical_transformer, one_hot_features)],
    remainder='drop')

And the following configuration for XGBoost:

mono_pos_values = [1] * len(args.keys_pos_contraints)
        dic_mono_constraints = dict(
             zip(args.keys_pos_contraints, mono_pos_values))

param_grid_clf = {
                'n_estimators': args.clf_n_estimators,
                'max_depth': args.clf_max_depth,
                'max_bin': args.clf_max_bin,
                'learning_rate': args.clf_learning_rate,
                'gamma': args.clf_gamma,
                'colsample_bytree': args.clf_colsample_bytree,
                'colsample_bylevel': args.clf_colsample_bylevel,
                'reg_alpha': args.clf_reg_alpha,
                'reg_lambda': args.clf_reg_lambda,
                'subsample': args.clf_subsample,
                'random_state': args.random_seed,
                'tree_method': 'hist',  # cpu: 'hist' | gpu: 'gpu_hist'
                'objective': 'binary:logistic',
                'eval_metric': 'logloss',
                'monotone_constraints': dic_mono_constraints
                #non-indicated features will be considered as 0.
                # For clf, I expect to apply this to the predict_proba function
            }

When creating my pipeline as:
        model = xgb.XGBClassifier(**param_grid_clf)
        clf = Pipeline(
            [
                # now preprocessor is an independent pipeline
                ('preprocessor', preprocessor),
                ('estimator', model)
            ]
        )

I get the following error:
ValueError: Constrained features are not a subset of training data feature names
The only way we have found to fix this is to split our scikit learn pipeline in 2 and use the column name after the preprocessor.fit_transform() which is ['numeric_transf__tgt_period']
However this seems suboptional as we end up ditching the purpose of having an scikit learn pipeline.
Would it be possible for someone to look into this?

Okay but then this loses completely the point of using the scikit learn API. Is there any other work around?

Hi @trivialfis ,
Thanks for the support.
Let me try to explain myself better as I think I have not done a good job before:

The pipeline I have has a set of input features that I predefine.
Some of this features might be dropped by the variance selector. In future it might be another method.
The monotonic constraint is applied to a feature that its always there (tgt_period). Tgt period never gets dropped. However after it goes through the column transformer of the preprocessor step this feature becomes "name_of_numeric_transformer_feature". In my case numeric_trans_tgt_period.
If I use the original feature name (tgt_month) in the motonic constraint parameter it does not work. If I use the output name from the numeric transformer "numeric_transf__tgt_period" it also does not work. If I split my pipeline in 2 steps (one for the column transformer and another for the model) it works. The input to the model then is a pandas data frame with columns coming from the Column transformer: preprocessor.get_features_names_out().
If I understand the documentation about monotonic constraints correctly.  If I refer to the monotonic constraint with a dictionary where I set up the position of the column. This can make the model to put the constraint in a different feature as the column order for my feature changes (because some features are dropped by the variance selector).

Is my statement in 5 correct or did I misunderstood?

If I refer to the monotonic constraint with a dictionary where I set up the position of the column

I think you meant tuple? If so,

This can make the model to put the constraint in a different feature as the column order for my feature changes (because some features are dropped by the variance selector).

Yes.

If I use the output name from the numeric transformer "numeric_transf__tgt_period" it also does not work.

Could you please check why it doesn't work? Any chance there's an unexpected change of name? Here's a simple example I extracted from the tests:
import numpy as np
import pandas as pd
import xgboost as xgb

def make_mc():
    number_of_dpoints = 1000
    x1_positively_correlated_with_y = np.random.random(size=number_of_dpoints)
    x2_negatively_correlated_with_y = np.random.random(size=number_of_dpoints)

    x = np.column_stack(
        (x1_positively_correlated_with_y, x2_negatively_correlated_with_y)
    )
    zs = np.random.normal(loc=0.0, scale=0.01, size=number_of_dpoints)
    y = (
        5 * x1_positively_correlated_with_y
        + np.sin(10 * np.pi * x1_positively_correlated_with_y)
        - 5 * x2_negatively_correlated_with_y
        - np.cos(10 * np.pi * x2_negatively_correlated_with_y)
        + zs
    )
    return x, y

params = {
    "tree_method": "hist",
    "grow_policy": "lossguide",
    "monotone_constraints": {"feature_0": 1, "feature_1": -1},
}

reg = xgb.XGBRegressor(**params)
x, y = make_mc()

feature_names = ["feature_0", "feature_1"]
xdf = pd.DataFrame(x, columns=feature_names)
reg.fit(xdf, y)

As a side note, some sklearn estimators/transformers return numpy array instead of pandas dataframe, in which case, the feature name would be loss.

Hi @trivialfis,
That example works, and it is actually precisely what I am doing:
clf_preprocessor = clf['preprocessor']
clf_preprocessor.fit(data_train)

data_train_processed = pd.DataFrame(clf_preprocessor.transform(data_train),
                                            columns=list(
                                            clf_preprocessor.get_feature_names_out()))

clf_model = clf['estimator'].fit(data_train_processed, target_train)

However this seems a bit inconvenient, and as mentioned before, it ditch the purpose of wrapping xgboost into a pipeline. In principle if a set up scikit learn to return pandas output this should be fixed, but by doing that I will have to set sparse=False in one hot encoder which will cause the matrix size to explode.
You mentioned before about writing a wrapper, would you be so kind to give more additional indications on this?

In order for XGBoost to select/constrain the features based on their names, one needs to first ensure XGBoost knows what are the names of the features right? With an array or CSR input, this is not possible since it doesn't contain names for the columns. Within the pipeline, XGBoost doesn't know which column is numeric_transf__tgt_period.
The categorical tree in xgboost can help remove the use of the one-hot encoder, see https://xgboost.readthedocs.io/en/stable/tutorials/categorical.html for details. In this case, you still need to make sure the input to XGBoost is stored in pandas. Similar to the column name issue, array and CSR don't have the type of the columns (categorical or numerical).

Feel free to reopen if there's further info.

source: gharchive/issue

authors: edgBR trivialfis

repo: dmlc/xgboost

license_type: permissive

license_source: github-api


text:
Paused jobs from replay using WMCore agent 1.5.2 with error SetupCMSSWPset due issue in edm_pset_tweak.py

Impact of the bug
Tier0 agent using 1.5.2 fails workflows
Describe the bug
While testing WMCore agent 1.5.2, with the CMSSW CMSSW_11_3_4  and 12_0_0, the replay shows paused jobs with an error within SetupCMSSWPset.py when trying to execute  edm_pset_tweak.py
How to reproduce it
Replay using T0 release 3.0.0 based on 1.5.2 agent.
Expected behavior
No errors from edm_pset_tweak.py
Additional context and error message
Please take into account that Tier0 was able to deploy the replay using the 1.5.2 agent including the following PR #10801

More details on the error after reproducing it interactively:
$ edm_pset_tweak.py --input_pkl PSet.pkl --output_pkl PSet_output.pkl --json PSetTweak.json --allow_failed_tweaks
We got here
Traceback (most recent call last):
  File "edm_pset_tweak.py", line 160, in <module>
    main()
  File "edm_pset_tweak.py", line 147, in main
    err_val = apply_tweak(process, tweak[0], tweak[1],args.skip_if_set, args.create_untracked_psets )
  File "edm_pset_tweak.py", line 82, in apply_tweak
    setattr(param,key_split[-1],value)
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_4/python/FWCore/ParameterSet/Mixins.py", line 276, in __setattr__
    self.__addParameter(name, value)
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_4/python/FWCore/ParameterSet/Mixins.py", line 244, in __addParameter
    self.__raiseBadSetAttr(name)
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_4/python/FWCore/ParameterSet/Mixins.py", line 299, in __raiseBadSetAttr
    raise TypeError(name+" does not already exist, so it can only be set to a CMS python configuration type")
TypeError: fileName does not already exist, so it can only be set to a CMS python configuration type

$ echo $?
1

So, the problem is --allow_failed_tweaks isn't ignoring tweaks that fail, which we do want to happen sometimes like here (in this case, we want to ignore setting up process.sqLite.xxx variables in the PSet.

This was fixed with the external commit below from David L.:
https://github.com/cms-sw/cmssw-wm-tools/commit/aa1626fb2d2fdbde6b3259e4b44828220883a809
and was verified to work by @germanfgv

repo: dmwm/WMCore


text:
subscribe any new block we inject from WMAgent to DataOps ?

Followup to #5945. Talked to Nicolo and this could work technically.
Basically, when the WMAgent creates a new block in PhEDEx, ie. when file injections create a new block, immediately make a block level subscription for that block to the site we inject it at under DataOps.
This would help with space accounting for DataOps. Not sure if it would break anything else.

Please no, it would create an insane amount of mails/subscriptions. It looks much more easy - for me at least - to allow phedex group for data injection, since when we are injecting a block against a specific site, we are also saying this data is available there...

You can make a subscription without mails...

Data groups at injection time is a absolute no-go because injection is a file level operation, which ownership is recorded at block level. You need separate calls for them.

Hm, anyways I wonder if there would be any impact handling big datasets with 2k different subscriptions, that looks ... not clean to me.
But don't we have 2 separate calls anyways? We need to inject either a block and its files into TMBD.

You inject FILES (and this might or might not create a new block). You subscribe BLOCKS. Very different things.
Nicole said the extra subscriptions should be manageable. They would be temporary anyways because most of the data DataOps produces is eventually moved somewhere else for aggregation purposes, right ? That would remove the individual block level subscriptions from the system.

How the block names get defined them? When the agent injects file, does it get back the block name to which the file was injected? Thinking how we then have the same block name between DBS and PhEDEx...
Lots of things have changed in operation and I'm not sure anymore whether there is a final move subscription to aggregate data. @vlimant for sure knows that.

Not sure myself about your first question. The agent somehow gets the block name, not sure if that's a return from the file injection, it creates it upfront or it specifically checks for it. These three things come to mind. @ticoann , any comments on this ?

There is no "final move" for various practical reasons irrelevant to the question of whether should dataops own a block dataops inject

Without a final move this needs to be rethought. We cannot just leave blocks owned by DataOps scattered across the sites. At the very least all these subscriptions would need to be changed to be AnalysisOps. DataOps has limited space, we need to cleanup after ourselves.

#5945 """There are no more moves ; only transfers and deletes"""

My statement stands, DataOps doesn't have enough space available to them to not cleanup (where cleanup can be delete or subscribing to a different group) data it produces. The DataOps space is designed to keep processing input samples and to temporarily store processing output. Not permanently store processing output.

agreed, hence the deletes.

duplicate issue #5945

May we please re-open this and discuss this more ?

authors: amaltaro hufnagel ticoann vlimant


text:
Duplicate lumisection in single workflow

For this workflow which is the only one writing in its output
pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417
event completion real 1999031 expected 2000000
event completion real 1997739 expected 2000000
pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417 has duplicates
{
"/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/RunIISummer15wmLHEGS-MCRUN2_71_V1-v1/GEN-SIM": false,
"/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/RunIISummer15wmLHEGS-MCRUN2_71_V1-v1/LHE": true
}
{
"/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/RunIISummer15wmLHEGS-MCRUN2_71_V1-v1/GEN-SIM": {},
"/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/RunIISummer15wmLHEGS-MCRUN2_71_V1-v1/LHE": {
"0:7592": [
"/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root",
"/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/D09C66C9-AFA2-E611-9A52-0CC47AA98A0E.root"
],
"0:7591": [
"/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root",
"/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/D09C66C9-AFA2-E611-9A52-0CC47AA98A0E.root"
]
}
it's the second instance in not too long so there must be something wrong
@prozober

The only thing I found in the agent (vocms0116) were logs like these [1]. @amaltaro do you think these errors could cause duplicates?
[1]
install/wmagent/JobAccountant/ComponentLog:2016-11-03 20:34:08,337:140528953976576:ERROR:AccountantWorker:Bad FwkJobReport Path: /data/srv/wmagent/v1.0.21.patch2/install/wmagent/JobCreator/JobCache/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417/Production/JobCollection_11079_0/job_119483/Report.0.pkl
install/wmagent/ErrorHandler/ComponentLog:2016-11-03 20:36:36,076:140194997511936:ERROR:ErrorHandlerPoller:Failed to find FWJR for job 119483 in location /data/srv/wmagent/v1.0.21.patch2/install/wmagent/JobCreator/JobCache/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417/Production/JobCollection_11079_0/job_119483/Report.0.pkl

Please keep this workflow active if possible. I'll try to have a look at the job configurations to see if they were really meant to produce the same lumi sections (bear in mind this week is not very productive in terms of real work).
Just in case, these 2 different files really produced the same lumi section 7591 (as cryptically reported above):
https://cmsweb.cern.ch/dbs/prod/global/DBSReader/filelumis?logical_file_name=/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root
https://cmsweb.cern.ch/dbs/prod/global/DBSReader/filelumis?logical_file_name=/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/D09C66C9-AFA2-E611-9A52-0CC47AA98A0E.root

7592 is also duplicated

We won't be able to keep the workflow in that status forever, as we have to deliver it.
Any chance you can grab the information you need from the db to understand this before we deliver this back to analysis ?

Hi Jean-Roch, you can process I copied problem job reports here
/afs/cern.ch/user/c/cmst1/test/duplicatelumi
Problem seems to be there is difference in fwjr and wmbs record. (vocms0116)
Looking at the wmbs,  following is unmerged file containing (7591, 7592) lumis.
id=378511
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/BEA7849C-FDA1-E611-B937-0CC47AD98B90.root
This file contains the above unmerged file.  id=565303 (merged one) created by job 280836
/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/D09C66C9-AFA2-E611-9A52-0CC47AA98A0E.root
But first one does not.  (in webs_file_parent table) id=395554 created by job 266543
/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root'
But both of merged files  contains duplicate lumis although second job doesn't have the input file which contain the lumis.
select * from wmbs_file_runlumi_map where fileid=395554 and lumi in (7591, 7592);
select * from wmbs_file_runlumi_map where fileid=565303 and lumi in (7591, 7592);
Looking at the fwjr for job 266543, following input file seems to have lumis  (7591, 7592)
id=284841
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root
But wmbs has different record, (lump 7499, 7500)
select run, lumi from wmbs_file_runlumi_map wm inner join wmbs_file_details wd
on wm.fileid=wd.id where wd.lfn='/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root';
1	7499
1	7500
Looking at the source code.  This is where input file information is added
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/FwkJobReport/XMLParser.py#L82
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/FwkJobReport/Report.py#L493
I am not completely sure how CMSSW gets the run lumi information.
But it looks like there there is difference between CMSSW framework report run lumi information and wmbs run lumi information.  There might some edge case masking problem since lumi numbers are only off by 2.

@prozober please do.
@ticoann that could be worth passing on the edm people over HN @Dr15Jones

Do you have logs and configuration files for the jobs which produced output with the same lumis?

Hi Chris, Here is the config file.
https://cmsweb.cern.ch/couchdb/reqmgr_config_cache/0e64d45e2cc39207afb73e7fd4bb62f4/configFile
Right now I only have a condor logs, @amaltaro, Alan, does logArchive job contains CMSSW log?
Anyway, logArchive contains cmssw log, here is pfns - (just for tracking purpose). (It might be already deleted)
srm://srm.rcac.purdue.edu:8443/srm/v2/server?SFN=/mnt/hadoop//store/unmerged/logs/prod/2016/11/4/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417/Production/ProductionMergeLHEoutput/0000/0/a7ca0da2-a235-11e6-82a7-02163e017c3c-0-0-logArchive.tar.gz
srm://cmssrm.hep.wisc.edu:8443/srm/v2/server?SFN=/hdfs/store/unmerged/logs/prod/2016/11/4/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417/Production/ProductionMergeLHEoutput/0000/0/4aadd12e-a2af-11e6-82a7-02163e017c3c-2-0-logArchive.tar.gz
If they are deleted, we can dig into logArchive tar ball containing above logs.
Let me know if you want me to send the log files via email.
Thanks,

Yes, and it should also contain the original FJR (and the PSet)

On Nov 10, 2016, at 9:42 PM, ticoann notifications@github.com wrote:
Let me know if you want me to send the log files via email.

If you can make the log files, FJR and the PSet available (can just be in AFS or on the web) that would be very helpful.
Chris

Christopher Jones
Fermi National Accelerator Laboratory
cdj@fnal.gov

Of course the files are already deleted. :-(.
I think these are log files containing above logArchive files.
I am trying to copy the log I am not sure how big these are whether I have enough space. Alan do you usually copy in your home dir? (I just need to use xrdcp command, right? do you know the command I can check the size of file?)
Thanks,
root://castorcms.cern.ch//castor/cern.ch/cms/store/logs/prod/2016/11/WMAgent/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417-ProductionLHEoutputMergeLogCollect-g31n41-3-logs.tar

root://castorcms.cern.ch//castor/cern.ch/cms/store/logs/prod/2016/11/WMAgent/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417/pdmvserv_TOP-RunIISummer15wmLHEGS-00054_00206_v0__161102_162051_8417-ProductionLHEoutputMergeLogCollect-cms-h003-11-logs.tar

Seangchan, these are the Merge logcollects and hopefully there is only one tarball inside each of them. I copied their logArchiver over here:
http://amaltaro.web.cern.ch/amaltaro/forWMCore/a7ca0da2-a235-11e6-82a7-02163e017c3c-0-0-logArchive.tar.gz
http://amaltaro.web.cern.ch/amaltaro/forWMCore/a7ca0da2-a235-11e6-82a7-02163e017c3c-0-0-logArchive.tar.gz
where you can find information for the 2 LFNs you previously mentioned:
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/BEA7849C-FDA1-E611-B937-0CC47AD98B90.root

According to the Merge FJR, indeed two files had lumi sections 7591 and 7592.
I asked Valentin/Yuyi to search for the logArchive where those 2 files were created.

And these are the logs that produced the unmerged files:
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-749-0-logArchive.tar.gz
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-795-1-logArchive.tar.gz
It can be seen in these FJRs, but indeed one job produced a file with lumis 7499 and 7500 and the other job produced a file with lumis 7591 and 7592.
Seangchan, WMCore does not provide run/lumi mask for merge jobs, does it? We just provide the unmerged files to be merged and the framework takes care of it AND creates the FJR, that is then used by the agent for publishing this data and so on. Am I right?
IF so, then what needs to be figured out is why CMSSW wrote the wrong lumi sections into the FJR.

Thanks Alan for the logAchive files.

Seangchan, WMCore does not provide run/lumi mask for merge jobs, does it? We just provide the unmerged files to be merged and the framework takes care of it AND creates the FJR, that is then used by the agent for publishing this data and so on. Am I right?

Yes you are right. We don't provide run/lumi mask. @Dr15Jones, how CMSSW gets the run lumi information from unmarked file. (Is the file contains that information?)
Here are the log files Alan posted.
merge job
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/a7ca0da2-a235-11e6-82a7-02163e017c3c-0-0-logArchive.tar.gz
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/4aadd12e-a2af-11e6-82a7-02163e017c3c-2-0-logArchive.tar.gz
production job
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-749-0-logArchive.tar.gz
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-795-1-logArchive.tar.gz

Hi everyone,

On Nov 11, 2016, at 5:27 PM, ticoann notifications@github.com wrote:
Thanks Alan for the logAchive files.
Seangchan, WMCore does not provide run/lumi mask for merge jobs, does it? We just provide the unmerged files to be merged and the framework takes care of it AND creates the FJR, that is then used by the agent for publishing this data and so on. Am I right?
Without a mask the framework would merge all Lumis into the output.

Yes you are right. We don't provide run/lumi mask. @Dr15Jones, how CMSSW gets the run lumi information from unmarked file. (Is the file contains that information?)
The LuminosityBlocks are known internally to the file. There really is no way the framework can misread that information.

Looking at the logs from the jobs, the framework does in deed say the files in question have the same luminosity blocks. The following are excerpts from the merge job cmsRun logs:

--------------------------merge a-------------
04-Nov-2016 11:58:16 CDT  Successfully opened file root://cmsxrootd.hep.wisc.edu//store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/BEA7849C-FDA1-E611-B937-0CC47AD98B90.root
%MSG-w MemoryCheck:  source 04-Nov-2016 11:58:17 CDT PostBeginLumi
MemoryCheck: module source:source VSIZE 550.98 2.00391 RSS 158.156 2.98438
%MSG
Begin processing the 22934th record. Run 1, Event 1225786, LumiSection 7591 at 04-Nov-2016 11:58:17.120 CDT
...
Begin processing the 23256th record. Run 1, Event 1226108, LumiSection 7592 at 04-Nov-2016 11:58:17.302 CDT
04-Nov-2016 11:58:17 CDT  Closed file root://cmsxrootd.hep.wisc.edu//store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/BEA7849C-FDA1-E611-B937-0CC47AD98B90.root
04-Nov-2016 11:58:17 CDT  Initiating request to open file root://cmsxrootd.hep.wisc.edu//store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F6C514D5-5CA1-E611-857C-003048F5B69A.root
04-Nov-2016 11:58:17 CDT  Successfully opened file root://cmsxrootd.hep.wisc.edu//store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F6C514D5-5CA1-E611-857C-003048F5B69A.root
%MSG-w MemoryCheck:  source 04-Nov-2016 11:58:17 CDT PostBeginLumi
MemoryCheck: module source:source VSIZE 552.984 2.00391 RSS 159.082 0.464844
%MSG
Begin processing the 23257th record. Run 1, Event 1227078, LumiSection 7599 at 04-Nov-2016 11:58:17.543 CDT
...
----------------------------merge b-----------
03-Nov-2016 22:25:19 EDT  Successfully opened file /mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root
Begin processing the 111759th record. Run 1, Event 1225786, LumiSection 7591 at 03-Nov-2016 22:25:19.258 EDT
Begin processing the 112081st record. Run 1, Event 1226108, LumiSection 7592 at 03-Nov-2016 22:25:19.352 EDT
03-Nov-2016 22:25:19 EDT  Closed file /mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root
03-Nov-2016 22:25:19 EDT  Initiating request to open file /mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/4203A3B8-5CA1-E611-A8FE-24BE05CE4D91.root
03-Nov-2016 22:25:19 EDT  Successfully opened file /mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/4203A3B8-5CA1-E611-A8FE-24BE05CE4D91.root
%MSG-w MemoryCheck:  source 03-Nov-2016 22:25:19 EDT PostBeginLumi
MemoryCheck: module source:source VSIZE 547.305 0 RSS 298.508 0.40625
%MSG
Begin processing the 112082nd record. Run 1, Event 1211251, LumiSection 7501 at 03-Nov-2016 22:25:19.431 EDT
Is it possible to get access to the files in question?
How do you know that the files here match the logs for the production?
Chris

Is it possible to get access to the files in question?

I think it is all deleted again. I tried to get it from worker node but wasn't there. And we don't keep the unmerged file some where else. (@amaltaro, Alan, that is correct, right?)

How do you know that the files here match the logs for the production?

We keep track in agent side as well.
To summarize what how the accounting process goes in agent.

Production(Processing as well) job is created with specific job splitting algorithm (mask is passed for each job) - When job is finished, output file info is comes in FWJR (WMAgent FWJR which contains run lumi information for output file from CMSWS FWJR)
We record this in wmbs db.

When merge job is created and submitted the same thing happen.

What we found is basically unmatched record in following file.
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root
The FWJR for the production job for above output file contains run 1,  lumi 7499, 7500.
But FWJR for the merge job including above file indicates it has run 1, lumi 7501, 7502.
Although it is the same file. I am guessing second information (FWJR from merge job) is incorrect.
But I am not sure how CMSSW gets that information and how the difference could happen.

If you look at the fwjr from  it says lumi section, 7499 and 7500
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-749-0-logArchive.tar.gz
<LFN>/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root</LFN>
<PFN>LHEoutput.root</PFN>
<Catalog></Catalog>
<ModuleLabel>LHEoutput</ModuleLabel>
<GUID>A685DDD8-5CA1-E611-9020-A0000420FE80</GUID>
<Branches>
  <Branch>LHEEventProduct_externalLHEProducer__SIM.</Branch>
</Branches>
<OutputModuleClass>PoolOutputModule</OutputModuleClass>
<TotalEvents>323</TotalEvents>

<DataType>MC</DataType>

<BranchHash>baad18a51a4cbab54f85db48954b2717</BranchHash>

<Runs>
<Run ID="1">
   <LumiSection ID="7499"/>
   <LumiSection ID="7500"/>
</Run>

but from merge Job fwjr for the same file lumisections are 7591, 7592
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/a7ca0da2-a235-11e6-82a7-02163e017c3c-0-0-logArchive.tar.gz
<InputFile>
<State  Value="closed"/>
<LFN>/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root</LFN>
<PFN>/mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root</PFN>
<Catalog></Catalog>
<ModuleLabel>source</ModuleLabel>
<GUID>A685DDD8-5CA1-E611-9020-A0000420FE80</GUID>
<Branches>
  <Branch>LHEEventProduct_externalLHEProducer__SIM.</Branch>
</Branches>
<InputType>primaryFiles</InputType>
<InputSourceClass>PoolSource</InputSourceClass>
<EventsRead>323</EventsRead>
<Runs>
<Run ID="1">
   <LumiSection ID="7591"/>
   <LumiSection ID="7592"/>
</Run>

On Nov 11, 2016, at 9:54 PM, ticoann notifications@github.com wrote:
The FWJR for the production job for above output file contains run 1, lumi 7499, 7500.
But FWJR for the merge job including above file indicates it has run 1, lumi 7501, 7502.
I believe you mistyped and meant 7591 and 7592 (since that is what is shown in the cmsRun log file).

Although it is the same file.
That is actually an assumption. We do not have any proof it is actually the same file since we can't look at it.

I am guessing second information (FWJR from merge job) is incorrect.
No, that information is correct. The framework reads to LuminosityBlock information from the file. From the actually cmsRun log file output we can see that the framework read the LuminosityBlock as 7591 and 7592 which does not match with what the log file for production said the job which supposedly created that file used as LuminosityBlock numbers.

The most likely scenario to me is either
1) the production log file you pointed me to was not the one that created that file OR
2) the file isn't the file the system thinks it is
Either is pretty scary.
I do not see any sign this is a framework problem as the log files produced by cmsRun match exactly with the FrameworkJobReport for those same logs.
	Chris

No, we don't keep the unmerged files. Once they are merged, there is a cleanup job that comes in and thrown then into the trash bin.

I do not see any sign this is a framework problem as the log files produced by cmsRun match exactly with the FrameworkJobReport for those same logs.

I agree, for the production jobs. However, each of those two merge jobs read these files and in their fjr they report different lumi sections.

I believe you mistyped and meant 7591 and 7592 (since that is what is shown in the cmsRun log file).
Sorry my mistake.  updated. :smile:

The most likely scenario to me is either

the production log file you pointed me to was not the one that created that file OR
the file isn't the file the system thinks it is

I am still confused how this could explain the difference between fwjr from unmarged file and merged file result for exact the same file. (Maybe 2) does.
This is weird since framework reads Lumi information directly from file, both should be the same. (Unless there is uuid collision and same name of the file is create after which is unlikely since we have the file with those lumis).

On Nov 11, 2016, at 10:26 PM, Alan Malta Rodrigues notifications@github.com wrote:
No, we don't keep the unmerged files. Once they are merged, there is a cleanup job that comes in and thrown then into the trash bin.
I do not see any sign this is a framework problem as the log files produced by cmsRun match exactly with the FrameworkJobReport for those same logs.
I agree, for the production jobs. However, each of those two merge jobs read these files and in their fjr they report different lumi sections.

In all cases the log file output of cmsRun matches the framework job report made by that same cmsRun job. So that backs the claim that the LuminosityBlock recorded in the file being read is what is in the log file.
Therefore the most likely conclusion is the file written by the production job is not actually the same file being read by the merge job.
Do you have the final merge files? In principal we could look at the values for the events in those 'same' LuminosityBlocks and see if they are actually the same events.
Chris

On Nov 11, 2016, at 10:27 PM, ticoann notifications@github.com wrote:
This is weird since framework reads Lumi information directly from file, both should be the same. (Unless there is uuid collision and same name of the file is create after which is unlikely since we have the file with those lumis).

Looking at the configuration file given to cmsRun, the framework is told to use a very generic name for the output file
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
something in the workflow management system then renames the file. Therefore we actually have no idea if the file read by the merge job is actually the one produced by the production job.
Chris

The production file is not the same file as used in the merge file. The GUIDs do not match.
From the FJR of the production job we have

/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/GEN-SIM/MCRUN2_71_V1-v1/130000/RAWSIMoutput.root
RAWSIMoutput.root

RAWSIMoutput
006427D9-5CA1-E611-9020-A0000420FE80
but from the FJR of the merge job we have

/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root
/mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root

source
A685DDD8-5CA1-E611-9020-A0000420FE80

LHEEventProduct_externalLHEProducer__SIM.

primaryFiles
PoolSource
323

The very first part of the GUIDs do not match
006427D9-5CA1-E611-9020-A0000420FE80
A685DDD8-5CA1-E611-9020-A0000420FE80
Chris

Hi Chris, I think you are looking at the wrong file. GUUI matches with Production job. (A685DDD8-5CA1-E611-9020-A0000420FE80)
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-749-0-logArchive.tar.gz
<LFN>/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root</LFN>
<PFN>LHEoutput.root</PFN>
<Catalog></Catalog>
<ModuleLabel>LHEoutput</ModuleLabel>
<GUID>A685DDD8-5CA1-E611-9020-A0000420FE80</GUID>
<Branches>
  <Branch>LHEEventProduct_externalLHEProducer__SIM.</Branch>
</Branches>
<OutputModuleClass>PoolOutputModule</OutputModuleClass>
<TotalEvents>323</TotalEvents>

<DataType>MC</DataType>

<BranchHash>baad18a51a4cbab54f85db48954b2717</BranchHash>

<Runs>
<Run ID="1">
   <LumiSection ID="7499"/>
   <LumiSection ID="7500"/>
</Run>

Looking at the configuration file given to cmsRun, the framework is told to use a very generic name for the output file
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
something in the workflow management system then renames the file. Therefore we actually have no idea if the file read by the merge job is actually the one produced by the production job.

Is that mean actually file is created in generic name and renamed later? Actual file name is created by LHEoutput to GUID in fwjr.
<ModuleLabel>LHEoutput</ModuleLabel>
<GUID>A685DDD8-5CA1-E611-9020-A0000420FE80</GUID>

Isn't the causing the race condition if two jobs putting the files in the same directory? Is that that agent code which rename the file?

On Nov 11, 2016, at 11:02 PM, ticoann notifications@github.com wrote:
Hi Chris, I think you are looking at the wrong file. GUUI matches with Production job. (A685DDD8-5CA1-E611-9020-A0000420FE80)
http://amaltaro.web.cern.ch/amaltaro/forWMCore/Issue_7371/1e19574a-a152-11e6-a6e4-02163e017c3c-749-0-logArchive.tar.gz

The confusion was there are two files written by the production job, both with very similar GUIDs
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
LHEoutput.root

LHEoutput
A685DDD8-5CA1-E611-9020-A0000420FE80
and
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/GEN-SIM/MCRUN2_71_V1-v1/130000/RAWSIMoutput.root
RAWSIMoutput.root

RAWSIMoutput
006427D9-5CA1-E611-9020-A0000420FE80

After talking to Chris, Eric,
I am guessing this is what happens (but I don't have concrete evidence)
For production job - 266543

CMSSW framework copy  the unmerged file with following name (from PSet config which is controlled by PSetTweeks code) - cmssw frame work job report is already created at this point.

/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root

WMCore code? renames the output by replacing LHEoutput to GUID from fwjr. - I still couldn't find the code doing this).

In this case there could a race condition there
Other job can writ the the same file  before 2 happens.
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
Basically first job which wrote to that path is overwritten by second job before it gets renamed.
So first unmerged output file is lost and second unmerged output file is duplicated to different files.
In our case,
/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root
and
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/BEA7849C-FDA1-E611-B937-0CC47AD98B90.root
To fix this, following patch might work.
https://github.com/dmwm/WMCore/commit/84cfa939ee56733e8e6fa2b026ae69889b8b740e
But I can't still find where the step 2 happen which would affect the patch. @amaltaro, do you know where the code resides doing that?

Just to clarify, the framework does not copy the unmerged file. The framework just writes the file in the local directory of the worker node. The copy is done by the workflow system.

The file
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
never exists. It's a bogus entry in the some intermediate reports that is then supplemented by the stageout code which creates the real file on the mass storage. Are you saying we actually wrote an LHEoutput.root file to mass storage ???

Chris told me that (/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root)
is what cmssw framework writes. Not the unmerged file which exists.
(/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root)
@Dr15Jones, Chris, did I misunderstood what you told me?

The framework writes `./LHEoutput.root'.

but where? which parameter is it using for current directory?

I'm pretty sure ./LHEoutput.root is written in the pilot scratch area. When it's moved to the storage, it's moved with a UID.

The GUID makes it into the report here
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/FwkJobReport/FileInfo.py#L72
This is also where we override the report to account for direct stageout to merged.
One comment, this assume that the report already contains
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
so either @Dr15Jones is wrong about the ./LHEoutput.root or we have yet another piece of wrapper code.

And yes, we write LHEoutput.root into the current working directory, but that doesn't neccessarily mean that the FJR also reports ./LHEoutput.root.

Hmm then, I am not sure how this happens. pilot scratch area is unique to a job, right?
Then where following parameter is used
https://github.com/dmwm/WMCore/blob/master/src/python/PSetTweaks/WMTweak.py#L509

I believe the FJR reports the 'logical' file name as passed to it from the configuration.

Yep, the LFN reported by the FJR has NOTHING to do with where the job puts the file on the local filesystem. It's a LFN that only has meaning for the stageout as a remote directory.

@hufnagel, so PFN below is the current working (job running directory).
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/LHEoutput.root
LHEoutput.root
and stage out moves the file to directly to below pin, (meaning file name is not renamed after file is copied but before file is copied, correct?)
/mnt/hadoop/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/A685DDD8-5CA1-E611-9020-A0000420FE80.root

No. Current working directory is unknown, no idea what it is. We don't need to know.
/store/unmerged/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/
has no meaning whatsoever as far as the worker node file system is concerned. It's a string, nothing more.
That string + GUID IS the stageout location though.
Renamed is the wrong word here. These are all just string manipulations, the first time this corresponds to an actual path and file on a disk somewhere is for the stageout target.

Ie. LFN from the CMSSW FJR does NOT correspond to an actual file somewhere, it's completely bogus. It's just a way for us to forward the UnmergedLFNBase from the workflow configuration to the FJR.

Not sure why this was implemented like this. Before my time. We could also have skipped the LFN completely and in
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/FwkJobReport/FileInfo.py#L72
constructed the LFN from scratch by doing UnmergedLFNBase+GUID.

Two different pilots never run in the same work space (directory). This is correct statement, right?
So now I don't know how this happened, then. 😞

Correct.

I haven't had time to follow all this, but the LHEOutput.root thing smacks of cmsRun forking off a script to produce the LHE file or something like this. There might actually be code in cmssw that deals with this. IIRC these files are produced in a strange way and @vlimant worked on this.

Ok, so let me summarize:

We have the two processing job logArchive and according to the logs and FJR we wrote out different lumis.

We have the log and FJR of the merge job and according to log and FJR it

a) read in the two files from the jobs in 1) and
b) wrote out different lumis than we had in 1)
Is that correct and all confirmed ?

Hm, so it could be a bug in that code which causes the lumis for one of the jobs in 1) to be reported wrong ? Is that what your are saying ?

The LHEOutput.root file is written by the standard PoolOutputModule. The file format you are thinking about ends with .lhe.

I believe there were two separate merge jobs. Both jobs report having merged the same 2 lumis.

I believe there were two separate merge jobs. Both jobs report having merged the same 2 lumis.

Ok, but one of these merge jobs had an input that was an output of one of the files in 1) and there is a match in file name but a mismatch in lumis ?
If so, sometime between the job in 1) writing it's output and the merge job in 2) the content of the file changed. Which is a bit scary... Apart from the name, does other meta-data match ? Like number of events written by the job in 1) and number of events read by the merge job in 2) ?

Ok, but one of these merge jobs had an input that was an output of one of the files in 1) and there is a match in file name but a mismatch in lumis ?

Yes that is correct. FJR for unmerged file contains lumi [7499, 7500]. but in the merge job FJR for the same file lumi is reported as [7591, 7599].

If so, sometime between the job in 1) writing it's output and the merge job in 2) reading it's input the content of the file changed. Which is a bit scary... Apart from the name, does other meta-data match ? Like number of events written by the job in 1) and number of events read by the merge job in 2) ?

Yes,  event matches. But this is MC (and by definition of splitting algorithm), basically all the jobs has the same events.
The reason I suspect of race condition was the unmerged file from other job which contains [7499, 7500] is located in the same directory and I was confused about cmssw creating the generic name of the file directly to that directory and WMAgent renaming it which doesn't seems to be possible after Dirk pointed out.
Either

something overwrites  the initial unmerged file before merge job ran.
Or FWJ report has wrong information about the input file(unmerged) in the merge job.
I would think it is more plausible 1. case but not sure how could that happen.

@Dr15Jones, Chris told me, that we can actually check what lumis the file contains by examining actual file, right?
Following is the merged file which contains mismatching (between unmerged and merged) lumi case. Can we check whether this actually contains [7591, 7599] or [7499, 7500]?
/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root

On Nov 14, 2016, at 10:24 PM, ticoann notifications@github.com wrote:
Either

something overwrites the initial unmerged file before merge job ran.
Or FWJ report has wrong information about the input file(unmerged) in the merge job.
I would think it is more plausible 1. case but not sure how could that happen.
The log file for the merge job and the FWJ both say the file being read have the unexpected lumi blocks.

@Dr15Jones, Chris told me, that we can actually check what lumis the file contains by examining actual file, right?
Following is the merged file which contains mismatching (between unmerged and merged) lumi case. Can we check whether this actually contains [7591, 7599] or [7499, 7500]?
/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root

One can get the list of Run/Lumi/Events in the file by doing
edmFileUtil -e /store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root
However, my attempt to do so gives this error
Error in TNetXNGFile::Open: [ERROR] Server responded with an error: [3011] Unable to open file /eos/cms/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root; No such file or directory
ERR Could not open file root://eoscms.cern.ch//eos/cms/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root

Hi everyone,

On Nov 15, 2016, at 10:08 AM, Chris Jones ChrisDJones15@gmail.com wrote:

@Dr15Jones, Chris told me, that we can actually check what lumis the file contains by examining actual file, right?
Following is the merged file which contains mismatching (between unmerged and merged) lumi case. Can we check whether this actually contains [7591, 7599] or [7499, 7500]?
/store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root

One can get the list of Run/Lumi/Events in the file by doing
edmFileUtil -e /store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root

edmFileUtil -e root://cmsxrootd-site.fnal.gov//store/mc/RunIISummer15wmLHEGS/ST_FCNC-TLL_Tleptonic_kappa_zut-MadGraph5-pythia8/LHE/MCRUN2_71_V1-v1/130000/F08187C8-35A2-E611-ADBF-A0000420FE80.root | grep '(Lumi)' >& lumis.log
I was eventually able to get this to work from CERN. The relevant bit of the output is below:
          1           7496              0            689 (Lumi)
          1           7497              0            690 (Lumi)
          1           7498              0            691 (Lumi)
          1           7591              0            692 (Lumi)
          1           7592              0            693 (Lumi)
          1           7501              0            694 (Lumi)
          1           7502              0            695 (Lumi)

The second column is the LuminosityBlock number and the fourth column is the index in the file to find that lumi. The workflow management system gives the files to merge in what it think is increasing LuminosityBlock number. You can see that '7591' and '7592' are not in the proper order because the workflow system thought the input file being read had luminosity block 7499 and 7500, which it obviously does not.

Ok, this is a bit scary. Question, were both the "fake" lumi 7499/7500 and 7591/7592 files located at the same site ? Ie. is it possible the local storage system gave us the wrong file ? That's about the only thing I can think off. According to all the records we can access from our end, it really looks like we wrote file A from job 1 and then read file A from job B, except it wasn't the same file, even though it had the same name.
Everyone agree with this interpretation ?
If this was indeed at the same site, any chance to ask the site about logs for the two files (creation, access, deletion, general meta-data ?)

Everyone agree with this interpretation ?

Or initial unmerged file is over written after it is created and before merged job runs some how some where. 😄

Everyone agree with this interpretation ?
Or initial unmerged file is over written after it is created and before merged job runs some how some where. 😄

Not a much better option :-).  Either the filesystem gave us the wrong file or someone (not us from what we can tell) overwrote the file with another files content...
Either way, the only way I can see to even have a hope of understanding this is from the mass storage logs (if there are any).

Seems to me like we should find out all we can about the jobs which made the original files

what sites were they produced on?
if the same site, what nodes were they produced on?
-once produced, where were the temporary files moved to?
what site did the merge?
how did the merge job retrieve the files (local file system, xrootd, etc.)?

I have a couple of questions as well.

Can a pilot run multiple jobs when pilot life time is longer then i.e 2 jobs running there?
If 1. is correct statement,
Are jobs running cuncurrently in the pilot?
Are the work space shared between the jobs?
Does workspace is shared?

On Nov 16, 2016, at 09:47, ticoann <notifications@github.commailto:notifications@github.com> wrote:
I have a couple of questions as well.

Can a pilot run multiple jobs when pilot life time is longer then i.e 2 jobs running there?

Yes
If 1. is correct statement,
2. Are jobs running cuncurrently in the pilot?
3. Are the work space shared between the jobs?
3. Does workspace is shared?
2 - yes, multiple jobs/pilot
3 and 3 - no, should not be
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/dmwm/WMCore/issues/7371#issuecomment-260980625, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABmErsb4yWbDYLd5FDcysSOq-saleENhks5q-yV4gaJpZM4Kqd7G.

Does workspace is shared?
3 and 3 - no, should not be

I guess then what is the exact mechanism used to separate them?

Workspace is separate for the payloads. Pilot creates different working directories per payload.

But how does the pilot name those directories?

For example /home/glidein_pilot/glide_iHWb9z/execute/dir_25339 (Tier0 job). Last level is payload specific,  there are three of them in this pilot for the three currently running jobs.

And the CMSSW working directory is
/home/glidein_pilot/glide_iHWb9z/execute/dir_25339/job/WMTaskSpace/cmsRun1
every step has it's own working directory.

Assuming after dir_ is the job id, what if the job ID wrapped back around during processing?

It's not the job id, it's some internal pilot counter. GlideInWMS has not idea what is in the payload. And it doesn't wrap around.

Anyways, this is checkable. Job condor logs or maybe even the logArchive will contain the working directory for the job.

authors: Dr15Jones amaltaro ericvaandering hufnagel prozober ticoann vlimant


text:
A simple function with python-like indentation. #9521

Fixes #9521
Status
ready
Description
A simple function with python-like indentation (for more features, see the issue resolved by the current PR).
Substitute pprint with twPrint
Is it backward compatible (if not, which system it affects?)
YES
Related PRs
No
External dependencies / deployment changes
No

test this please

test this please

Todor, don't forget to ask for code review by using the github Reviewers option.
Anyhow, I did a test with this script and the output seems much more readable now, thanks!
I'd have one suggestion though, can you get rid of those blank lines before and after each section? Example of current output:
======> Request information from reqmgr2 db:

    InputDataset:
    RequestNumEvents: 20000
    Runtime: 4.29111111111
    TotalEstimatedJobs: 100
    TotalInputEvents: 20000
    TotalInputFiles: 0
    TotalInputLumis: 200
    lumis: 200

======> ReqMgr2 output dataset info:

    /DYJetsToLL_Pt-50To100_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/DMWM_Test-StepChain_MC_HG2002_Val_Alanv2-v20/GEN-SIM
    /DYJetsToLL_Pt-50To100_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/DMWM_Test-StepChain_MC_HG2002_Val_Alanv2-v20/LHE
    /DYJetsToLL_Pt-50To100_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/DMWM_Test-StepChain_MC_HG2002_Val_Alanv2-v20/GEN-SIM-RAW
    /DYJetsToLL_Pt-50To100_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/DMWM_Test-StepChain_MC_HG2002_Val_Alanv2-v20/AODSIM

======> Couch output dataset info:

    /DYJetsToLL_Pt-50To100_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/DMWM_Test-StepChain_MC_HG2002_Val_Alanv2-v20/AODSIM:
        dsetSize: 714397797

the way I see without the blank lines is:
======> Request information from reqmgr2 db:
    InputDataset:
    RequestNumEvents: 20000
    Runtime: 4.29111111111
    TotalEstimatedJobs: 100
    TotalInputEvents: 20000
    TotalInputFiles: 0
    TotalInputLumis: 200
    lumis: 200

======> ReqMgr2 output dataset info:
    /DYJetsToLL_Pt-50To100_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/DMWM_Test-StepChain_MC_HG2002_Val_Alanv2-v20/GEN-SIM
    /DYJetsToLL_Pt-50To100_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/DMWM_Test-StepChain_MC_HG2002_Val_Alanv2-v20/LHE
    /DYJetsToLL_Pt-50To100_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/DMWM_Test-StepChain_MC_HG2002_Val_Alanv2-v20/GEN-SIM-RAW
    /DYJetsToLL_Pt-50To100_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/DMWM_Test-StepChain_MC_HG2002_Val_Alanv2-v20/AODSIM

======> Couch output dataset info:
    /DYJetsToLL_Pt-50To100_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/DMWM_Test-StepChain_MC_HG2002_Val_Alanv2-v20/AODSIM:
        dsetSize: 714397797

If it takes you more than 15min, then forget it and we go ahead with the current changes.

Done. I actually reorganized how the new lines are placed, so we do not accumulate those on every recursive call any more. And it took me 'JUST' 15 min. :)

Thanks, Todor. However, I still see double blank lines in a few places :-D
----------------------------------------------------
==> amaltaro_TaskChain_ProdMinBias_HG2002_Val_200128_200646_765
----------------------------------------------------
Comments:
[u'MC from scratch; Given 100 EpLumi for Task1, set EpJob and EpLumi to 41 (instead of 100), so 1LpJob;', u'Task2 and Task3 EventAwareLumiBased with 800 and 900 EpJob; diff ProcStr']

----------------------------------------------------

|***************************************************|
|                         | CouchDB | PhEDEx | DBS  |
|---------------------------------------------------|
| Same dataset name       | ok      | ok     | ok   |
| Same dataset size       | NOPE    | NOPE   | NOPE |
| Same number of blocks   | --      | ok     | ok   |
| Same number of files    | NOPE    | NOPE   | NOPE |
| Same number of events   | NOPE    | --     | NOPE |
| Same number of lumis    | NOPE    | --     | NOPE |
| Same PhEDEx Node Name   | --      | ok     | ok   |
|***************************************************|

======> Request information from reqmgr2 db:
    InputDataset:
    RequestNumEvents: 9000
    Runtime: 3.45888888889
    TotalEstimatedJobs: 220
    TotalInputEvents: 9000
    TotalInputFiles: 0
    TotalInputLumis: 220
    lumis: 220

======> ReqMgr2 output dataset info:
    /RelValProdMinBias/DMWM_Test-ProdMinBias_TaskChain_ProdMinBias_HG2002_Val_Alanv2-v11/GEN-SIM
    /RelValProdMinBias/DMWM_Test-DIGIPROD1_TaskChain_ProdMinBias_HG2002_Val_Alanv2-v11/GEN-SIM-RAW
    /RelValProdMinBias/DMWM_Test-RECOPROD1_TaskChain_ProdMinBias_HG2002_Val_Alanv2-v11/AODSIM
    /RelValProdMinBias/DMWM_Test-RECOPROD1_TaskChain_ProdMinBias_HG2002_Val_Alanv2-v11/GEN-SIM-RECO

======> Couch output dataset info:
...

source: gharchive/pull-request

authors: amaltaro todor-ivanov


text:
🛑 MX Auth is down

In fa98647, MX Auth (https://auth.siembro.mx) was down:

HTTP code: 502
Response time: 4554 ms

Resolved: MX Auth is back up in b85ecee.

license: MIT


text:
Feat : CD 구축

💡 왜 PR을 올렸나요?

신규 피처

💁 무엇이 어떻게 바뀌나요?

팀 repository로 vercel을 이용할 경우 금액이 발생합니다. 그래서 개인 repository로 clone > vercel로 자동 배포하는 식으로 구현하였습니다.

추후에 AWS 배포 혹은 vercel 유료 이용하게 될 경우 CD를 수정하면 될 것 같습니다.

📆 작업 예정인 것이 있나요 ?

💬 리뷰어분들께

앗 요 부분 저도 배포 확인하려고 fork한 개인 레포 vercel로 연결 해놨었는데 작업이 중복 된 것 같네요🫣

앗 정말요 !..

authors: guesung seondal


text:
DotNetNuke.Services.Mail::sendMail doesn't use bodyEncoding for alternative view in multipart message

Description of bug
DotNetNuke.Services.Mail::sendMail doesn't use bodyEncoding for alternative view in multipart message. Emails are sent as ASCII but whenever a special character such as \n or \r is included the content type becomes UTF-8.
Steps to reproduce
List the steps to reproduce the behavior:
Create a Simple Email module

Go to a page > Add Module > Module Creator
For the View.ascx.cs, put in the sendmail method with ASCII encoding in the button event.
 Mail.SendMail("from@local.me", "", "test@local.me", "", 0, "", 0, Encoding.ASCII, "body\n", "", "", "", "", "");
Save the module. Save the field to send email and observe the email received. Check the email's Content-Type.
Change the the "body\n" to "body". Save the module. Save the field to send email and observe the email received. Check the email's Content-Type.

Current result
Email with body - "body\n" Content-Type received is text/plain; charset=utf-8
Email with body - "body" Content-Type received is text/plain; charset=us-ascii
Expected result
Email's Content-Type charset should always consistent with encoding. If encoding is set to Encoding.ASCII, then charset should always ascii in
regardless the body content
Screenshots
https://www.loom.com/share/b9ebd96afcbf4070be17c063e7326752 - video recording
Error log
Nope
Additional context
https://github.com/dnnsoftware/Dnn.Platform/blob/584b0f54bd6b6d237594dd9f96053f14db0b1a53/DNN Platform/Library/Services/Mail/Mail.cs#L109 - the problem is
the encoding is  set to null. We shall set the encoding with the bodyEncoding
Affected version

[x] 9.3.2
[x] 9.3.1
[x] 9.2.2
[x] 9.2.1
[x] 9.2
[x] 9.1.1
[x] 9.1
[x] 9.0

Affected browser

Browser independent issue

[x] Chrome
[x] Firefox
[x] Safari
[x] Internet Explorer
[x] Edge

Due to branching changes, this change needs to be re-evaluated to see if it is needed for the 10.x release.  If so, the changes from PR #2900 will need to be replayed against the new 10.x branch.

authors: mitchelsellers tingung

license: mit

license_source: bigquery


text:
Update bug report issue template

Summary
Replace "nightly build" phrase with "non-release build".
Resolves #3172

Pushed new commit to use "alpha" instead of "non-release".


text:
My contribution

warnings_soultions.xlsx

Summary

Fixes  #2648
Please note that the warning solutions are listed in the excel file attached in detail.
(https://github.com/dnnsoftware/Dnn.Platform/files/3800837/warnings_soultions.xlsx)
Only one warning is left by now :
"C:\Dnn.Platform\packages\Yarn.MSBuild.1.16.0\build\Yarn.MSBuild.targets" cannot be imported again. It was already imported at "C:\Dnn.Platform\Build\BuildScripts\Module.Build (2,4)". This is most likely a build authoring error. This subsequent import will be ignored.	Dnn.PersonaBar.Extensions	11	C:\Dnn.Platform\Dnn.AdminExperience\Extensions\Content\Dnn.PersonaBar.Extensions	C:\Dnn.Platform\Dnn.AdminExperience\Extensions\Content\Dnn.PersonaBar.Extensions\Dnn.PersonaBar.Extensions.csproj	743

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: eng-aomar sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Also, if this is no breaking change and you can target the release/9.4.x branch, it could be merged earlier, if not, this will only get into Dnn 10. We are planning on changing the branching strategy to make this easier on contributors but unfortunately cannot make that happen before Dnn10.

The manifest versions need to stay blank because they are dynamically set by GitVersion, can you undo that change please.

Do you mean removing the version number 10.00.00 from version tag, replacing it with version="" instead of version="10.00.00" .

It may works but I am not 100% sure, to be save I would remove the whole version="" thing on those files

I had some time to look a bit more into this pull request and there are too many changes that do not make sense... A lot of methods where marked deprecated without discussing deprecating them and without a deprecation message, also this resolves most warning by just disabling them, we need to instead switch usage to the new way of doing things and not just silence those warnings. I am sorry be we cannot accept this pull request the way it is

I had some time to look a bit more into this pull request and there are too many changes that do not make sense... A lot of methods where marked deprecated without discussing deprecating them and without a deprecation message, also this resolves most warning by just disabling them, we need to instead switch usage to the new way of doing things and not just silence those warnings. I am sorry be we cannot accept this pull request the way it is

Thank you for your time reviewing this pull request.
I am managing to start from scratch with a different approach for solving the warnings, I will categorize the warnings according to their types, and make smaller pull requests, so we can reduce the number of warnings at least.

Awesome, thanks

authors: dnfclas eng-aomar valadas


text:
Fixes Memory Leak for MVC Modules

Fixes: #3344
Summary
Fixes memory leak in MVC modules by utilizing the Microsoft recommendations for Dependency Injection in ASP.NET MVC. The Docs were very useful and while they use the Unity Container, this was adapted for IServiceProvider

ASP.NET MVC Dependency Injection

Notable Changes

Removed UseMvc() method which registers all MVC controllers to the container. It will now use the default controller activation technique in ASP.NET
Deleted Custom IControllerFactory as we are now using default ASP.NET controller factory
Added new IDependencyResolver for DNN

Notes
In my initial tests, this appears to have fixed the memory leak, but I would like some others to take the install package and test it out prior to us accepting this change

Nice work @ahoefling thank you very much for your effort!

authors: ahoefling thabaum


text:
Fix React build process (WIP)

Webpack now integrated with lerna workspaces

Sorry. This doesn't look like it's going anywhere. It won't build properly with this react project in our main monorepo approach. We'll have to take a different route.

Why is this request closed?

authors: donker kurtwilbies


text:
Implement a Token Provider interface

#3819.
The token replacement code is very exposed through public and protected members. My implementation leaves all of those in place to avoid breaking existing functionality. Therefore, all the core tokenization code still lives in the BaseCustomTokenReplace, BaseTokenReplace and TokenReplace classes.
If the provider is different than the CoreTokenProvider, it calls that provider instead. The CoreTokenProvider also has a simple implementation that calls the TokenReplace class. This is in case someone decides to use the provider instead of calling TokenReplace directly.
The changes I did also consolidate all parameters related to tokenization into a single TokenContext class that can be easily passed around.
Tested on DNN 9.6.1 with some simple scenarios and everything seems to work as before.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: bogdan-litescu sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

did you test with modules implementing custom TR, like "DNN Form & List"?

did you test with modules implementing custom TR, like "DNN Form & List"?

I haven't tested Form and List yet. I tested the HTML module, which also uses a derived TR HtmlTokenReplace and the skin tokens, such as the language flags.

tokenReplace was originally developed as part of Form&List module, which still uses it more extensively than HTML, Skin or Newsletter module.

@bogdan-litescu this contribution looks great!  There are some failing tests, though.  I'm assuming you'll need to adjust how some things are created in the tests, maybe by creating mock implementations.  Feel free to ask questions if you have trouble tracking the issues down.  Thanks!

@bogdan-litescu this contribution looks great! There are some failing tests, though. I'm assuming you'll need to adjust how some things are created in the tests, maybe by creating mock implementations. Feel free to ask questions if you have trouble tracking the issues down. Thanks!

@bdukes Thanks for pointing this out. I registered the CoreTokenProvider in the ComponentFactory and that seems to do the trick.

authors: bdukes bogdan-litescu dnfadmin sleupold


text:
最新版本的docker 镜像国内DNS解析异常

版本latest
国内DNS解析异常，现象为部分网站图片无法加载。
退回v1.3.5
后回复正常。

可否在无法加载图片的网页上，按 F12 进入开发者模式，在网络选项中查看图片资源 URL 所在域名，或在 AdGuardHome 日志中查找解析出错的上游返回值，提交一下这些具体信息帮助定位错误。
从 1.3.5 升级到 latest 版本，其中变动仅在于编译新稳定版 dnsproxy 与 AdGuardHome，这个问题大概率是 ADG 团队在上游更新了某些东西导致的。
PS: AdGuardHome 的 v0.107.31 版本就因为严重 bug 重发了 hotfix，有可能是它还存在其他问题。

您可以尝试最新版的镜像，它升级 AdGuardHome 至 v0.107.33 版本，该版本存在诸多错误修复，可能已解决 issue 问题。

昨晚尝试升级后问题依旧。之前忘说了，用新版本后微软teams，office的解析有也问题，teams提示离线，office一直在连接服务器，更换DNS后就恢复正常。

我看了一下docker日志，报错是ADG启动失败：unknown configuration schema version 23
查询到https://github.com/AdguardTeam/AdGuardHome/issues/2458
解决办法是删除映射目录下的adguard文件夹，然后重启即可。
应该是不同版本的ADG不兼容。

喔😲居然是这个导致的，解决了就好


text:
Ways to get the output of a certain layer?

Sorry to bother you, I have a question about getting the output of convolution or hidden layers.

I am not very familiar with lasagne/nolearn functions, it seems to me that for a given dataset which I already had a model trained on it, I can get the output of a particular layer by calling function similar to this: predict_proba = output_layer.get_output(X_batch, deterministic=True)
but how to let the network know which layer I want?
I try to count the index of the layer I want to use and do:

layer_i = self.get_all_layers()[3]
for Xb, yb in self.batch_iterator_test(X):
    out.append(layer_i.get_output(Xb,deterministic=True))

However, it give me error float argument required, not TensorVariable
Ideally, I want to write a function similar to predict_proba(self, X) which take X and layer name as input and return the output for that layer.
    def predict_proba(self, X):
        probas = []
        for Xb, yb in self.batch_iterator_test(X):
            probas.append(self.predict_iter_(Xb))
        return np.vstack(probas)

Another similar question is how to get the parameters of a convolution layer, the dataset I am working on is 1 dimensional data, however, using one layer of convolution significantly improved the performance, I would like to see how original features are interacted in the convolution layer.

Thanks in advance:)

Regarding your first question: there's an open pull request that adds a transform method that'll get you the output of a layer other than the output layer: #89.
Regarding getting the parameters of a conv layer:  Once you get hold of a layer, (use nn.layers_['<layername>']) you can call its get_params method, which will return a list of Theano shared variables, which have a get_value method.

authors: YunshuLiu dnouri

repo: dnouri/nolearn


text:
E: Unable to locate package xserver-xorg-input-vmmouse

Please paste the output of the following command here: sudo edit-chroot -all

name: sid
encrypted: no
Entering /mnt/stateful_partition/crouton/chroots/sid...
crouton: version 1-20161129162558~master:5f2f11f9
release: sid
architecture: armhf
targets: lxde,keyboard,xorg
host: version 9150.0.0 (Official Build) dev-channel daisy
kernel: Linux localhost 3.8.11 #1 SMP Wed Jan 4 03:01:11 PST 2017 armv7l GNU/Linux
freon: yes
Unmounting /mnt/stateful_partition/crouton/chroots/sid...
Please describe your issue:
The chroot setup can't finish, because it's trying to do something with xserver-xorg-input-vmmouse, a package it can't find.
If known, describe the steps to reproduce the issue:
Just try to create a chroot:
sudo sh ~/Downloads/crouton -r sid -u -t lxde,keyboard,xorg

For others getting the same problem, until that is really fixed in crouton, you can force things by doing:
sudo enter-chroot -u 0 -n sid
this will get you a root shell, allow you to edit /setup.sh (or some such -- a shell script at the root) to remove the line mentioning the *vmmouse package (don't forget to remove the continuation backslash at the previous line too!) ; then get out and:
sudo enter-chroot
will ask to finish the setup, and answering 'yes' will actually work.

@SnarkBoojum,
Thanx for the workaround.
It looks like it's coming from xorg#L189, maybe it is not valid with the 'sid' / 'arm' combo ???
Perhaps some more conditionals will have to be added once the devs get a chance to look at it.
Hope this helps,
-DennisL

This also occurs with the stretch target, but the same work around applies (modify prepare.sh, etc).

As a reference https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=831420 xserver-xorg-input-vmmouse package is obsolete. This will also affect newer ubuntu releases (need to check which). Checked Kali-rolling it is also missing the package.
Should be labelled as a crouton bug.

@SnarkBoojum @msturgill Can one of you please post the error messages you get with xserver-xorg-input-vmmouse . I can't replicate it on amd64 and I don't have an arm based chromebook. It will be trying to remove xserver-xorg-input-vmmouse, but maybe for some reason when doing that it errors out on arm.

Well, the error message is that the package doesn't exist anymore ; if you look here, you'll see it is gone from testing, unstable and experimental : it won't be in the next stable. My guess is that you can't replicate because you try to install the current stable, which is the last one to have it.

@SnarkBoojum On amd64 it works ok. It does see that the package doesn't exists and ignores uninstalling just fine and continues the install script. I need the exact error messages that it spits out on arm, so we can fix it in crouton. It should not error out on a non existing package trying to uninstall.

E: Unable to locate package xserver-xorg-input-vmmouse
Failed to complete chroot setup.
The armhf package for xserver-xorg-input-vmmouse doesn't exist.

authors: DennisLfromGA SnarkBoojum cjfh divx118 msturgill

repo: dnschneid/crouton


text:
"Shell" is not working!

Please paste the output of the following command here: sudo edit-chroot -all

hello everyone so when i type the shell command is says crosh> shell
ERROR: unknown command: shell can anyone help me this is really annoying!

You need to change your chromebook to Developer Mode for the command to work. The procedure is device dependant.
A simple google search like "chromebook model developer mode will give you results. Follow the procedure and the shell command will work as intented after that.

@PMaxDS its in developer mode....

@LiamPlayz,
Can you login as user 'chronos' in VT2 (via ctrl+alt+F2) ?
-DennisL

@DennisLfromGA nope doesnt work

the thing wont even open D:

@LiamPlayz,
Even though you think you are still in Developer mode it sounds like you may not be any longer.
Do you get something similar to the following and have to press ctrl+D to boot into the login screen -

If not, then take a look at these links for more info. -

Developer Information for Chrome OS Devices - click on the Model name of your device
Poking around your Chrome OS Device

Hope this helps,
-DennisL

@DennisLfromGA i re downloaded it looks like some glitch happends with my laptop working fine now thanks very much

👍

authors: DennisLfromGA LiamPlayz PMaxDS


text:
Cras compiler errors

I will post tommoroa

Please describe your issue:
Hello, I get an cras error about cras.client.5 write something with 2 and 1 and 3, I searched  the error and havent found anything that worked, please help me.
If known, describe the steps to reproduce the issue:

Please do not open multiple issues, it does not help solve the problem.

Refer to:

#4450
#4618

Refer to:

#4450
#4618

authors: BROKENIST2 BetaPictoris DennisLfromGA


text:
working with ChromeOS, X11 Xiwi fails

Discussed in https://github.com/dnschneid/crouton/discussions/4750

Originally posted by TomTravis May 26, 2022

I now a ChromeOS Dev have it running now
reading the x11 code and review the crosh shell and crostini
SUSE and maybe others will be added to the install list RPM file being converted to DEB now
back to write a vscode issue and see what happens. the webserver maybe best to run on a amd64 linux the new x11 on arm64 run the xorg just fine no xorg.config file freon is running
xiwi will not find the display new display.
Xorg finds the display just fine. Xiwi has the x86 dummy device hard coded and fails
3017.444]
X.Org X Server 1.21.1.99
X Protocol Version 11, Revision 0
[  3017.445] Current Operating System: Linux localhost 5.10.106-15290-g6e9d70c8a9b4 #1 SMP PREEMPT Wed May 18 19:34:18 PDT 2022 aarch64
Kernel command line: cros_secure console= loglevel=7 init=/sbin/init cros_secure drm.trace=0x106 root=/dev/dm-0 rootwait ro dm_verity.error_behavior=3 dm_verity.max_bios=-1 dm_verity.dev_wait=1 dm="1 vroot none ro 1,0 5324800 verity
Using a default monitor configuration.
[  3017.458] (==) Automatically adding devices
[  3017.458] (==) Automatically enabling devices
[  3017.458] (==) Automatically adding GPU devices
[  3017.458] (==) Automatically binding GPU devices
[  3017.461] (==) Max clients allowed: 256, resource mask: 0x1fffff
3044.513] (II) modeset(0): ShadowFB: preferred NO, enabled NO
[  3044.513] (II) modeset(0): Output eDP-1 has no monitor section
[  3044.513] (II) modeset(0): Output DP-1 has no monitor section
[  3044.514] (II) modeset(0): EDID for output eDP-1
[  3044.514] (II) modeset(0): Manufacturer: CMN  Model: 14e5  Serial#: 0
[  3044.514] (II) modeset(0): Year: 2018  Week: 4
[  3044.514] (II) modeset(0): EDID Version: 1.4
[  3044.514] (II) modeset(0): Digital Display Input
[  3044.514] (II) modeset(0): 8 bits per channel
[  3044.514] (II) modeset(0): Digital interface is DisplayPort
[  3044.514] (II) modeset(0): Max Image Size [cm]: horiz.: 31  vert.: 17

are you running the latest official crouton release on an official build of ChromeOS?

well so here it is, xserver is offical release from Xorg yes is was complied on  both amd64 and arm64 Debian run amd64 both Xiwi and  Xorg stable. the crouton -r -t use the latest X86 free driver  and complies it for the install. i then use the X11 update to Xorg libs
the Debootstrap script you wrote  compiles to the latest Xorg Libs and Xorg run the older server for  Xiwi automatic
jammy fails both Xorg and Xiwi arm64  bullseye and bookworm run the same Xorg works Xiwi fails

hello

Sorry, I'm having a hard time following what you're saying. Did you compile the latest Xorg yourself? If so, you might be missing some of the patches that crouton applies to the dummy device.

hello good to have your thoughts back excellent
i spent some time with Git today to update the branches and update my repo
I really do not think the X86free drive is used anymore it is still supports on VM boxes with no display, The Xserver code is in full Dev still, it knows chomebook is not  a VM device with dummy driver, you admit this is a hack LOL

authors: TomTravis dnschneid


text:
⚠ Site ANTS has degraded performance

In f40915e, Site ANTS (https://ants.gouv.fr) experienced degraded performance:

HTTP code: 200
Response time: 17206 ms

Resolved: Site ANTS performance has improved in 7a8c818.

repo: dnum-mi/stats-sites-api


text:
Lag in markdown when new version of plugin is installed

Since Intellij Ultimate 2021.1+ and plugin 1.7+ there is lag when you enter characters into markdown (5-15seconds). I tried to investigate why this happened but so far I see that in UI thread is open a network socket. When I use 1.5 version everything is ok, but 1.7 and 1.8 cause lag.
threadDump-20210702-150056.txt

thanx for reporting this issue.
Where do you enter the markdown code? Within the diagrams.net editor?

No. The interesting part is that I use bundled Markdown. We use your plugin to create diagrams, export to svg and use in *.md. But when the new plugin is installed I can enter 1 or 2 characters in markdown and have to wait 5 to 15 seconds. The issue is inside ui thread. At least I think.

I have a similar issue, but once the draw.io plugin is enabled, the official Jetbrains markdown plugin doesn't even load anything at all and it only shows a blan white window. This at least happens on CLion, most likely on other Jetbrain's products too.

I've analyzed this; this is due to the SVG parsing feature I introduced for #48. If a SVG file contains an external references or a document type with a URL that resolves slowly, the UI will block. Thank you  @dylanaut for providing a SVG example.
Update plugin 0.1.9 is available from GitHub releases and will be available from the JetBrains marketplace soon after approval by the marketplace team.

As I was able to reproduce the problem and the fix locally, I close this issue. If the problem persists, please comment here and I'll re-open the issue.

I needed to rework this issue, as valid draw.io SVGs can contain a DTD. These files were ignored by the plugin 0.1.9.
0.1.10 will be available soon and fix this. Builds are already running; might take 1-2 days to appear in JetBrains Marketplace.

Release is available from GitHub releases

@ahus1 thanks for fixing the issue. However, I can see the 0.1.10 on Jetbrains now and updated it, but it still shows as white blank window. I've invalidated the jetbrains cache and tried again, but same..

@hiradyazdan - let's separate the white-preview problem to issue #87 and continue the discussion over there as it seems unrelated to this issue to me.

authors: ahus1 hiradyazdan marianondrovic-alexnett rdmueller


text:
response = self.client.create_project(self, TypeError: DoccanoClient.create_project() got multiple values for argument 'name'

response = self.client.create_project(self,

TypeError: DoccanoClient.create_project() got multiple values for argument 'name'

Please help to resolve this issue
doccano version - 1.8.0
client-version - 1.2.6

The lack of information. Please paste your code correctly.

self.client.create_project(self,
You shouldn't be passing self to this function

authors: Hironsan david-engelmann zic13

repo: doccano/doccano-client


text:
how to start docker in alpine without 'service docker start'  & 'rc-update add docker boot'

Dear all,
I have an issue that after I pull docker from internet,I find out I can't start the docker because it turns out error when I type 'service docker start'  or 'rc-update add docker boot'.As there is no related command within the alpine system.So,how can I start the docker service without those two commands offered?

The docs for each image provides information for running the image. For DIND there is a video that gives the commands https://asciinema.org/a/24707
$ docker run -it --privileged --name some-docker -d docker:dind
7f4547f661892f6c8cc3cde262f56128b59ee4c2600671b68dc2db5a6178aa8e
$ docker exec -it some-docker ash
/ # docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
/ #

Thanks a lot.But how could I start the docker service when I pull docker:latest instead of docker:dind?
After I pull docker:latest,I find out when I type 'docker version' in the container,the system prints out only the version of the docker client without showing that of the docker server,unlike docker:dind showing both the information.

You can do dockerd & and then ctrl+c to stop it from outputting to terminal. Requires --privileged

repo: docker-library/docker


text:
[33] FATAL:  password authentication failed for user "postgres" after image update

I'm using almost a year my postgres container. I've started with version 11 back then, successfully upgraded to version 12 in the past. Currently using the latest version.
I check if there's new image for the tag 12. There was some updates in the last month, but I had no time to pull the image, because I've plenty other containers using the database, so I've to stop them first.
So, I thought I pull the latest image.

Stopped all of the containers using the database. docker-compose down
Stopped postgres. docker-compose down
Pulled the image. docker-compose pull
Started postgres container.  docker-compose up

When I started the container I got
[33] FATAL:  password authentication failed for user "postgres"
[33] DETAIL:  Password does not match for user "postgres".
Connection matched pg_hba.conf line 95: "host all all all md5"
[34] FATAL:  password authentication failed for user "postgres"
[34] DETAIL:  Password does not match for user "postgres".
Connection matched pg_hba.conf line 95: "host all all all md5"
.
.
.

And so on. I've panicked, because there was no postgres update, only the docker image was updated. I searched on the net, but could not find anything. However I can connect to the postgres database with docker exec -it postgres psql -U postgres, and I can connect with pgadmin as well, but still the container spamming authentication failure, which is not healthy I guess. I use the same docker-compose.yml and .env like before pulling the newest image.
My docker-compose.yml for reference.
version: "3.7"

services:
  db:
    image: postgres:12
    container_name: postgres
    restart: always
    ports:
      - 5432:5432
    environment:
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
    volumes:
      - postgres-data:/var/lib/postgresql/data
    networks:
      - postgres-network

volumes:
  postgres-data:
    driver: local-persist
    driver_opts:
      mountpoint: "${DOCKER_VOLUMES}/postgres"

networks:
  postgres-network:
    external: true

TL;RD:
I can connect with the password using pgadmin, but the container spamming authentication failure after starting. Can you help me?

Never mind, I just dumped everything and created a new container, imported the data.

Junpet, Can you explain how you did that ? coz I am having same issue here and unable to move further with my project.

@Rajesh25B Since I could connect with Pgadmin, I just dumped everything through the UI first. Then stopped the container and set new volume path (empty folder), so after starting the container again I got a server without any data. After that I connected through Pgadmin again, then imported the previously dumped data.

repo: docker-library/postgres


text:
Remove docker compose v1

It's officially EOL.
See https://github.com/docker/compose/commit/30fcb72c

I think there are cases where new one doesn't work for people, until we bump golang to a newer one. (something something proxy/networking whatever, everything is broken, must revert to python one).
So I would be happy to do this, as long as we are on latest v2 compose, which is built with the latest golang.

Given https://github.com/docker/roadmap/issues/257#issuecomment-1427024325 and https://github.com/docker-snap/docker-snap/pull/118 we can merge this.

authors: xnox zhsj

repo: docker-snap/docker-snap


text:
ci: add GitHub-based login to web UI

TODO:

[x] Add some kind of policy to say which GitHub users should be allowed to use the CI

Not sure if access policy works with private repositories. Will check once I get approval from test org.

repo: docker/datakit


text:
NULL tags in catalog API

I'm not sure this is a bug or simply an undocumented aspect of the catalog API, but when listing the tags of a repository that does not exist, the API returns a 200 OK with the JSON payload {"name":"<repo-name>","tags":null}. In this case, it would be more practical if tags were an empty list, as this would facilitate usage such as piping the response body through e.g. jq '.tags[]'. It would also be more in line with what one would expect after reading the API documentation.
To reproduce: curl -s <some-2.1.1-registry>/v2/nosuchrepohere/tags/list. This is using an S3 backend version 2.1.1 registry.
No big deal of course, but maybe something for someone wanting a quick win some rainy afternoon. :-)

Hello, we have had a similar situation in Portus but this time when calling this method while the image was being uploaded (see SUSE/Portus#552). I'd also expect an empty list to be returned honestly.
Anyways, if this is expected behavior, maybe we could improve the documentation of this method :)

This is a bug. Empty tags should probably result in a 404 from the specification.
Sorry about this. I really don't like the use of null in json and didn't mean impose that horror on anyone else. :sob:
@bnordbo Had you ever pushed to that repository then deleted it? There is actually a unit test for this and I can't reproduce this with the filesystem driver. I am suspecting some s3 oddness here. I've submitted #1187 to check for the test condition and we'll take if from there.

@bnordbo @mssola After testing out the PR, I am pretty certain this is due to an inconsistency between storage driver implementations. It looks like certain drivers don't return PathNotFound when the listing a non-existent path.
I am going to examine the impact of adding something to test suite and simple fix.

Thank you for looking into this @stevvooe! In my example, I had never pushed anything to the repository, i.e. it did not exist.

@stevvooe thanks for taking care of this :)

Closed as fixed by #1187.  This will be in the imminent release.

authors: RichardScothern bnordbo mssola stevvooe

repo: docker/distribution


text:
404 at: /engine/reference/builder/

URL: /engine/reference/builder/
Currently when searching for onbuild in the on-site search engine (https://docs.docker.com/search/?q=onbuild) an invalid or broken link is returned as the first argument: https://www.google.com/url?q=https://docs.docker.com/engine/reference/builder/&sa=U&ved=0ahUKEwjC3r2ot9DRAhVJ3iwKHTLJDZ0QFggEMAA&client=internal-uds-cse&usg=AFQjCNHWrFeuzOjTpux3y4YlhSoEHAl2rg

Fixed in #1286

authors: casperc mstanleyjones


text:
404 at /desktop/mac/release-notes/3.x/undefined

I found a broken link : https://docs.docker.com/desktop/mac/release-notes/3.x/undefined
I arrived on this page through: https://docs.docker.com/desktop/mac/release-notes/3.x/

Closing this ticket in favor of https://github.com/docker/docker.github.io/issues/13553, but if you're able to provide more details that could help us find the problem, feel free to comment on that ticket.

authors: craig-osterhout patrickmindia


text:
Cluster docs are incorrect for providing a subscription

File: cluster/cluster-file.md
From reading the docs under https://docs.docker.com/cluster/cluster-file/#cluster, I would expect the syntax to be:
cluster:
  subscription: sub-<id>
  license: /path/to/docker_subscription.lic

but that doesn't work...it is:
cluster:
  subscription:
    id: sub-<id>
    license: /path/to/docker_subscription.lic

There appears to be a missing section of all of the values under subscription.  I am not sure if there is more than id and license.

Thx @mbentley

authors: mbentley rupakg


text:
_includes/head.html: fix escaping in generated descriptions and titles

Commits 7e5352f1ae2dbe7e9d02f8193096d4513de3a2ba (https://github.com/docker/docker.github.io/pull/11488) and e72030d2c6fb4ffa31af7a6f9e5905d70db38de9 (https://github.com/docker/docker.github.io/pull/11489) added automatic generation of page titles and descriptions from the page's content if no front-matter metadata was present.
Some pages may include characters that should be escaped before using as a HTML attribute or JSON field.
This patch adds escaping to those texts to prevent the HTML or JSON from being invalid.
Before this:
HTML meta:
<meta name="description" content="docker build: The `docker build` command builds Docker images from a Dockerfile and a " context".="" a="" build's="" context="" is="" the="" set="" of="" files="" located="" in="" specified="" `path`="" or="" `url`...."="" />

JSON meta:
<script type="application/ld+json">{"@context":"http://schema.org","@type":"WebPage","headline":"docker build","description":"docker build: The `docker build` command builds Docker images from a Dockerfile and a "context". A build's context is the set of files located in the specified `PATH` or `URL`....","url":"https://docs.docker.com/engine/reference/commandline/build/"}</script>

After this:
HTML meta:
<meta name="description" content="docker build: The `docker build` command builds Docker images from a Dockerfile and a "context". A build's context is the set of files located in the specified `PATH` or `URL`...." />

JSON meta:
<script type="application/ld+json">{"@context":"http://schema.org","@type":"WebPage","headline":"docker build","description":"docker build: The `docker build` command builds Docker images from a Dockerfile and a \"context\". A build's context is the set of files located in the specified `PATH` or `URL`....","url":"https://docs.docker.com/engine/reference/commandline/build/"}</script>

@usha-mandya PTAL


text:
registry/azure.md: fix broken syntax/links

It looks like this doc, which was in docker/distribution repository before, is converted to HTML tables from Markdown tables. And in the table we used to have Markdown usage for

links
backtick-escaped sections

Since the table is HTML now, those Markdown usages won't work inside HTML.
I am just noticing this and adding them back. I wish the transition was more smooth.
Signed-off-by: Ahmet Alp Balkan ahmetalpbalkan@gmail.com

Deploy preview ready!
Built with commit 87e77ef6537f6b252370bbad3791d254d37d0626
https://deploy-preview-819--docker-docs.netlify.com

Thanks for contributing. I don't see any reason for this table to be HTML and not Markdown. Do you want to convert it?

@joaofnfernandes I don't know how we ended up here... We can keep it as is for now. Pushed just one more little fix about link rendering at the footer.

Thanks for catching that! As soon as the tests pass I'll get that merged.