text:
[ERROR]Task Node(0-rawreads/build) failed with exit-code=1 using SLURM

Job script:
#!/bin/bash

#SBATCH -p LM
#SBATCH -t 336:00:00
#SBATCH -N 1
#SBATCH --mem=200GB

# run it!
fc_run fc_run.cfg

Config file
[General]
input_fofn = fasta.fofn
input_type = raw
pa_DBdust_option=true
pa_fasta_filter_option=streamed-median

pa_DBsplit_option = -a -x500 -s200
ovlp_DBsplit_option =-x500 -s200

pa_HPCTANmask_option=
pa_REPmask_code=0,300;0,300;0,300

genome_size=1100000000
seed_coverage=30
length_cutoff=-1
pa_HPCdaligner_option=-v -B128 -M24
pa_daligner_option=-e0.8 -l2000 -k18 -h480  -w8 -s100
falcon_sense_option=--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400
falcon_sense_greedy=False

ovlp_daligner_option=-e.96 -s1000 -h60
ovlp_HPCdaligner_option=-v -M24 -l500

overlap_filtering_setting=--max-diff 100 --max-cov 100 --min-cov 2
fc_ovlp_to_graph_option=
length_cutoff_pr=1000

[job.defaults]
job_type=slurm
pwatcher_type=blocking
JOB_QUEUE = default
MB = 32768
NPROC = 6
njobs = 32
submit = srun --wait=0 -p myqueue -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} --mem-per-cpu=${MB}M --cpus-per-task=${NPROC} ${JOB_SCRIPT}

Error
falcon-kit 1.2.6
pypeflow 2.1.1
[INFO]Setup logging from file "None".
[INFO]$ lfs setstripe -c 12 /pylon5/mc5fqip/orca21 >
[WARNING]'lfs setstripe -c 12 /pylon5/mc5fqip/orca21' failed to produce any output.
[INFO]Lustre filesystem detected. This lfs stripe (12) should propagate to subdirs of '/pylon5/mc5fqip/orca21'.
[INFO]fc_run started with configuration fc_run.cfg
[INFO]cfg=
{
  "General": {
    "LA4Falcon_preload": false,
    "avoid_text_file_busy": true,
    "bestn": 12,
    "dazcon": false,
    "falcon_sense_greedy": false,
    "falcon_sense_option": "--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400",
    "falcon_sense_skip_contained": false,
    "fc_ovlp_to_graph_option": " --min-len 1000",
    "genome_size": "1100000000",
    "input_fofn": "fasta.fofn",
    "input_type": "raw",
    "length_cutoff": "-1",
    "length_cutoff_pr": "1000",
    "overlap_filtering_setting": "--max-diff 100 --max-cov 100 --min-cov 2",
    "ovlp_DBdust_option": "",
    "ovlp_DBsplit_option": "-x500 -s200",
    "ovlp_HPCdaligner_option": "-v -M24 -l500",
    "ovlp_daligner_option": "-e.96 -s1000 -h60",
    "pa_DBdust_option": "true",
    "pa_DBsplit_option": "-a -x500 -s200",
    "pa_HPCTANmask_option": "",
    "pa_HPCdaligner_option": "-v -B128 -M24",
    "pa_REPmask_code": "0,300;0,300;0,300",
    "pa_daligner_option": "-e0.8 -l2000 -k18 -h480  -w8 -s100",
    "pa_dazcon_option": "-j 4 -x -l 500",
    "pa_fasta_filter_option": "streamed-median",
    "pa_subsample_coverage": 0,
    "pa_subsample_random_seed": 12345,
    "pa_subsample_strategy": "random",
    "seed_coverage": "30",
    "skip_checks": false,
    "target": "assembly"
  },
  "job.defaults": {
    "JOB_QUEUE": "default",
    "MB": "32768",
    "NPROC": "6",
    "job_type": "slurm",
    "njobs": "32",
    "pwatcher_type": "blocking",
    "submit": "srun --wait=0 -p myqueue -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} --mem-per-cpu=${MB}M --cpus-per-task=${NPROC} ${JOB_SCRIPT}",
    "use_tmpdir": false
  },
  "job.step.asm": {},
  "job.step.cns": {},
  "job.step.da": {},
  "job.step.dust": {},
  "job.step.la": {},
  "job.step.pda": {},
  "job.step.pla": {}
}
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.blocking' from '/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pwatcher/blocking.pyc'>
[INFO]job_type='slurm', (default)job_defaults={'JOB_QUEUE': 'default', 'pwatcher_type': 'blocking', 'use_tmpdir': False, 'MB': '32768', 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} --mem-per-cpu=${MB}M --cpus-per-task=${NPROC} ${JOB_SCRIPT}', 'NPROC': '6', 'njobs': '32'}, use_tmpdir=False, squash=False, job_name_style=0
[INFO]Setting max_jobs to 32; was None
[INFO]Num unsatisfied: 2, graph: 2
[INFO]About to submit: Node(0-rawreads/build)
[INFO]Popen: 'srun --wait=0 -p myqueue -J P2c7b8ae23d7f98 -o /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash.stdout -e /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash.stderr --mem-per-cpu=4000M --cpus-per-task=1 /pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pwatcher/mains/job_start.sh'
[INFO](slept for another 0.0s -- another 1 loop iterations)
[INFO](slept for another 0.3s -- another 2 loop iterations)
[INFO](slept for another 1.2s -- another 3 loop iterations)
[INFO](slept for another 3.0s -- another 4 loop iterations)
[INFO](slept for another 6.0s -- another 5 loop iterations)
[INFO](slept for another 10.5s -- another 6 loop iterations)
[INFO](slept for another 16.8s -- another 7 loop iterations)
[INFO](slept for another 25.2s -- another 8 loop iterations)
[INFO](slept for another 36.0s -- another 9 loop iterations)
srun: error: l006: task 4: Exited with exit code 1
srun: error: l006: task 2: Exited with exit code 1
[INFO](slept for another 49.5s -- another 10 loop iterations)
[INFO](slept for another 66.0s -- another 11 loop iterations)
[INFO](slept for another 85.8s -- another 12 loop iterations)
srun: error: l006: tasks 0,3: Exited with exit code 1
[INFO](slept for another 109.2s -- another 13 loop iterations)
[INFO](slept for another 135.6s -- another 14 loop iterations)
[INFO](slept for another 150.0s -- another 15 loop iterations)
[INFO](slept for another 160.0s -- another 16 loop iterations)
[INFO](slept for another 170.0s -- another 17 loop iterations)
[INFO](slept for another 180.0s -- another 18 loop iterations)
[INFO](slept for another 190.0s -- another 19 loop iterations)
[INFO](slept for another 200.0s -- another 20 loop iterations)
[INFO](slept for another 210.0s -- another 21 loop iterations)
srun: error: l006: task 1: Exited with exit code 1
[ERROR]Task Node(0-rawreads/build) failed with exit-code=1
[ERROR]Some tasks are recently_done but not satisfied: set([Node(0-rawreads/build)])
[ERROR]ready: set([])
	submitted: set([])
[ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
Traceback (most recent call last):
  File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin/fc_run", line 11, in <module>
    load_entry_point('falcon-kit==1.2.6', 'console_scripts', 'fc_run')()
  File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 726, in main
    main1(argv[0], args.config, args.logger)
  File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 76, in main1
    input_fofn_fn=input_fofn_fn,
  File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 242, in run
    dist=Dist(NPROC=4, MB=4000, job_dict=config['job.step.da']),
  File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/pype.py", line 106, in gen_parallel_tasks
    wf.refreshTargets()
  File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 362, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: set([Node(0-rawreads/build)])

Conda list
# packages in environment at /pylon5/mc5fqip/orca21/anaconda3/envs/myenv:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py27_0
avro-python2              1.8.2                      py_1    bioconda
bcftools                  1.9                  h47928c2_2    bioconda
blas                      1.0                         mkl
blasr                     5.3.2                hac9d22c_4    bioconda
blasr_libcpp              5.3.1                hac9d22c_3    bioconda
bzip2                     1.0.6                h14c3975_5
ca-certificates           2019.1.23                     0
certifi                   2019.3.9                 py27_0
cffi                      1.12.1           py27h2e261b9_0
chardet                   3.0.4                    py27_1
cryptography              2.5              py27h1ba5d50_0
curl                      7.63.0            hbc83047_1000
cython                    0.29.5           py27he6710b0_0
decorator                 4.3.2                    py27_0
enum34                    1.1.6                    py27_1
falcon-kit                1.2.6                    pypi_0    pypi
falcon-phase              1.0.0                    pypi_0    pypi
falcon-unzip              1.1.7                    pypi_0    pypi
future                    0.17.1                   py27_0
genomicconsensus          2.3.2                    py27_3    bioconda
h5py                      2.8.0            py27h989c5e5_3
hdf5                      1.10.2               hba1933b_1
htslib                    1.9                  h47928c2_5    bioconda
idna                      2.8                      py27_0
igvtools                  2.3.93                        0    bioconda
intel-openmp              2019.1                      144
ipaddress                 1.0.22                   py27_0
iso8601                   0.1.12                   py27_1
krb5                      1.16.1               h173b8e3_7
libcurl                   7.63.0            h20c2e04_1000
libdeflate                1.0                  h14c3975_1    bioconda
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libgfortran-ng            7.3.0                hdf63c60_0
libssh2                   1.8.0                h1ba5d50_4
libstdcxx-ng              8.2.0                hdf63c60_1
linecache2                1.0.0                    py27_0
mkl                       2019.1                      144
mkl_fft                   1.0.10           py27ha843d7b_0
mkl_random                1.0.2            py27hd81dba3_0
ncurses                   6.1                  he6710b0_1
networkx                  2.2                      py27_1
numpy                     1.16.2           py27h7e9f1db_0
numpy-base                1.16.2           py27hde5b4d6_0
openjdk                   8.0.152              h46b5887_1
openssl                   1.1.1b               h7b6447c_1
pb-dazzler                0.0.0                h470a237_0    bioconda
pb-falcon                 0.2.6            py27hc0aa232_3    bioconda
pbalign                   0.3.2                    py27_0    bioconda
pbbam                     0.19.0               h6678c95_1    bioconda
pbcommand                 1.1.1                    py27_2    bioconda
pbcore                    1.6.5                    py27_0    bioconda
pip                       19.0.3                   py27_0
pycparser                 2.19                     py27_0
pyopenssl                 19.0.0                   py27_0
pysam                     0.15.2           py27h1671916_1    bioconda
pysocks                   1.6.8                    py27_0
python                    2.7.15               h9bab390_6
python-consensuscore      1.1.1            py27h3f2a3d2_3    bioconda
python-consensuscore2     3.4.1                    py27_0    bioconda
python-edlib              1.2.3.post1      py27h470a237_0    bioconda
python-intervaltree       3.0.2                      py_0    bioconda
python-msgpack            0.5.6            py27h470a237_0    bioconda
python-sortedcontainers   2.1.0                      py_0    bioconda
pytz                      2018.9                   py27_0
readline                  7.0                  h7b6447c_5
requests                  2.21.0                   py27_0
samtools                  1.9                  h91753b0_5    bioconda
setuptools                40.8.0                   py27_0
six                       1.12.0                   py27_0
sqlite                    3.26.0               h7b6447c_0
tk                        8.6.8                hbc83047_0
traceback2                1.4.0                    py27_0
unittest2                 1.1.0                    py27_0
urllib3                   1.24.1                   py27_0
wheel                     0.33.1                   py27_0
xz                        5.2.4                h14c3975_4
zlib                      1.2.11               h7b6447c_3

This was originally https://github.com/PacificBiosciences/FALCON/issues/685
I wrote:

You have to look into 0-rawreads/build. But please also post conda list. Someone else posted an odd bug, and I suspect that people are getting an inconsistent version of DAZZ_DB, which we provide via pb-dazzler.

You've got to look into 0-rawreads/build for stderr. You may have to debug this yourself by cd 0-rawreads/build and running parts of user_script.sh yourself. It will be difficult for us to debug this remotely, but with any luck, the answer will be obvious to you when to see stderr or run the commands yourself.

Here is the output of my stderr file, not sure what I should focus on.
executable=${PYPEFLOW_JOB_START_SCRIPT}

executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash ]]
executable=${PYPEFLOW_JOB_START_SCRIPT}
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash ]]
executable=${PYPEFLOW_JOB_START_SCRIPT}
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash ]]

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
executable=${PYPEFLOW_JOB_START_SCRIPT}
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash ]]

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
cd /pylon5/mc5fqip/orca21/0-rawreads/build
eval '/bin/bash run.sh'
++ /bin/bash run.sh
'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
cd /pylon5/mc5fqip/orca21/0-rawreads/build
'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
cd /pylon5/mc5fqip/orca21/0-rawreads/build
eval '/bin/bash run.sh'
++ /bin/bash run.sh
eval '/bin/bash run.sh'
++ /bin/bash run.sh
'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
cd /pylon5/mc5fqip/orca21/0-rawreads/build
eval '/bin/bash run.sh'
++ /bin/bash run.sh
export PATH=$PATH:/bin
export PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
export PATH=$PATH:/bin
/bin/bash task.sh
export PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
cd /pylon5/mc5fqip/orca21/0-rawreads/build
export PATH=$PATH:/bin
/bin/bash task.sh
export PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
/bin/bash task.sh
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
/bin/bash task.sh
export PATH=$PATH:/bin
export PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
/bin/bash task.sh
executable=${PYPEFLOW_JOB_START_SCRIPT}
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash ]]

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
cd /pylon5/mc5fqip/orca21/0-rawreads/build
eval '/bin/bash run.sh'
++ /bin/bash run.sh
export PATH=$PATH:/bin
export PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
/bin/bash task.sh
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
2019-03-11 17:20:44,908 - root - DEBUG - Running "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py /pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-11 17:20:44,908 - root - DEBUG - Running "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py /pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-11 17:20:44,908 - root - DEBUG - Running "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py /pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-11 17:20:44,908 - root - DEBUG - Running "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py /pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-11 17:20:44,909 - root - DEBUG - Running "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py /pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-11 17:20:44,913 - root - DEBUG - Checking existence of '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-11 17:20:44,913 - root - DEBUG - Checking existence of '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-11 17:20:44,913 - root - DEBUG - Loading JSON from '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-11 17:20:44,913 - root - DEBUG - Checking existence of '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-11 17:20:44,913 - root - DEBUG - Loading JSON from '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-11 17:20:44,913 - root - DEBUG - Checking existence of '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-11 17:20:44,913 - root - DEBUG - Loading JSON from '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-11 17:20:44,913 - root - DEBUG - Checking existence of '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-11 17:20:44,913 - root - DEBUG - Loading JSON from '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-11 17:20:44,913 - root - DEBUG - Loading JSON from '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-11 17:20:44,915 - root - DEBUG - {u'bash_template_fn': u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-11 17:20:44,915 - root - DEBUG - {u'bash_template_fn': u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-11 17:20:44,915 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:20:44,915 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:20:44,915 - root - DEBUG - {u'bash_template_fn': u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-11 17:20:44,915 - root - DEBUG - Checking existence of u'../../fasta.fofn' with timeout=30
2019-03-11 17:20:44,915 - root - DEBUG - Checking existence of u'../../fasta.fofn' with timeout=30
2019-03-11 17:20:44,915 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:20:44,915 - root - DEBUG - Checking existence of u'../../fasta.fofn' with timeout=30
2019-03-11 17:20:44,916 - root - DEBUG - Checking existence of u'../../General_config.json' with timeout=30
2019-03-11 17:20:44,916 - root - DEBUG - Checking existence of u'../../General_config.json' with timeout=30
2019-03-11 17:20:44,916 - root - DEBUG - Checking existence of u'../../General_config.json' with timeout=30
2019-03-11 17:20:44,916 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2019-03-11 17:20:44,916 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2019-03-11 17:20:44,916 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2019-03-11 17:20:44,917 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:20:44,918 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:20:44,918 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:20:44,919 - root - DEBUG - {u'bash_template_fn': u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-11 17:20:44,919 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:20:44,919 - root - DEBUG - Checking existence of u'../../fasta.fofn' with timeout=30
2019-03-11 17:20:44,919 - root - DEBUG - Checking existence of u'../../General_config.json' with timeout=30
2019-03-11 17:20:44,920 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2019-03-11 17:20:44,919 - root - DEBUG - {u'bash_template_fn': u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-11 17:20:44,919 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:20:44,920 - root - DEBUG - Checking existence of u'../../fasta.fofn' with timeout=30
2019-03-11 17:20:44,920 - root - DEBUG - Checking existence of u'../../General_config.json' with timeout=30
2019-03-11 17:20:44,920 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2019-03-11 17:20:44,921 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:20:44,921 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:20:45,236 - root - INFO - $('/bin/bash user_script.sh')
2019-03-11 17:20:45,236 - root - INFO - $('/bin/bash user_script.sh')
2019-03-11 17:20:45,237 - root - INFO - $('/bin/bash user_script.sh')
2019-03-11 17:20:45,237 - root - INFO - $('/bin/bash user_script.sh')
2019-03-11 17:20:45,236 - root - INFO - $('/bin/bash user_script.sh')
hostname
hostname
hostname
hostname
hostname
hostname
hostname
hostname
pwd
pwd
hostname
hostname
date
date
pwd
pwd
date
date
pwd
pwd
date
date
pwd
pwd
date
date
pwd
pwd
date
date

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option': u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy': u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100 --min-cov 2', u'length_cutoff_pr': u'1000', u'pa_REPmask_code': u'0,300;0,300;0,300', u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed': 12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'', u'input_type': u'raw', u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u' --min-len 1000', u'input_fofn': u'fasta.fofn', u'genome_size': u'1100000000', u'avoid_text_file_busy': True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'', u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60', u'target': u'assembly', u'falcon_sense_skip_contained': False, u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100', u'length_cutoff': u'-1', u'pa_DBdust_option': u'true', u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False, u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option': u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400', u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24', 'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'true', 'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy': u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0, 'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option': u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy': u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100 --min-cov 2', u'length_cutoff_pr': u'1000', u'pa_REPmask_code': u'0,300;0,300;0,300', u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed': 12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'', u'input_type': u'raw', u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u' --min-len 1000', u'input_fofn': u'fasta.fofn', u'genome_size': u'1100000000', u'avoid_text_file_busy': True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'', u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60', u'target': u'assembly', u'falcon_sense_skip_contained': False, u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100', u'length_cutoff': u'-1', u'pa_DBdust_option': u'true', u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False, u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option': u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400', u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24', 'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'true', 'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy': u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0, 'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]Log-level: INFO
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option': u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy': u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100 --min-cov 2', u'length_cutoff_pr': u'1000', u'pa_REPmask_code': u'0,300;0,300;0,300', u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed': 12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'', u'input_type': u'raw', u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u' --min-len 1000', u'input_fofn': u'fasta.fofn', u'genome_size': u'1100000000', u'avoid_text_file_busy': True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'', u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60', u'target': u'assembly', u'falcon_sense_skip_contained': False, u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100', u'length_cutoff': u'-1', u'pa_DBdust_option': u'true', u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False, u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option': u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400', u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24', 'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'true', 'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy': u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0, 'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option': u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy': u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100 --min-cov 2', u'length_cutoff_pr': u'1000', u'pa_REPmask_code': u'0,300;0,300;0,300', u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed': 12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'', u'input_type': u'raw', u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u' --min-len 1000', u'input_fofn': u'fasta.fofn', u'genome_size': u'1100000000', u'avoid_text_file_busy': True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'', u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60', u'target': u'assembly', u'falcon_sense_skip_contained': False, u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100', u'length_cutoff': u'-1', u'pa_DBdust_option': u'true', u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False, u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option': u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400', u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24', 'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'true', 'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy': u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0, 'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]Log-level: INFO
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option': u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy': u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100 --min-cov 2', u'length_cutoff_pr': u'1000', u'pa_REPmask_code': u'0,300;0,300;0,300', u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed': 12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'', u'input_type': u'raw', u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u' --min-len 1000', u'input_fofn': u'fasta.fofn', u'genome_size': u'1100000000', u'avoid_text_file_busy': True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'', u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60', u'target': u'assembly', u'falcon_sense_skip_contained': False, u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100', u'length_cutoff': u'-1', u'pa_DBdust_option': u'true', u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False, u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option': u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400', u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24', 'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'true', 'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy': u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0, 'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
module () {  eval /usr/bin/modulecmd bash $*
}
module () {  eval /usr/bin/modulecmd bash $*
}
module () {  eval /usr/bin/modulecmd bash $*
}
module () {  eval /usr/bin/modulecmd bash $*
}
module () {  eval /usr/bin/modulecmd bash $*
}
#!/bin/bash
set -vex
set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
echo PBFALCON_ERRFILE=
set -o pipefail
set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
#!/bin/bash
set -vex
set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
echo PBFALCON_ERRFILE=
set -o pipefail
set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
#!/bin/bash
set -vex
set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
echo PBFALCON_ERRFILE=
set -o pipefail
set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
rm -f raw_reads.db raw_reads.dam '.raw_reads.*'
rm -f raw_reads.db raw_reads.dam '.raw_reads.*'
rm -f raw_reads.db raw_reads.dam '.raw_reads.*'
#!/bin/bash
set -vex
set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
echo PBFALCON_ERRFILE=
set -o pipefail
set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
#!/bin/bash
set -vex
set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
echo PBFALCON_ERRFILE=
set -o pipefail
set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
rm -f raw_reads.db raw_reads.dam '.raw_reads.*'
rm -f raw_reads.db raw_reads.dam '.raw_reads.*'
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
zmw_whitelist_option=
use_subsampling=0
use_subsampling=0
zmw_whitelist_option=
use_subsampling=0
use_subsampling=0
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed "12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed "12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
[[ 0 -eq 1 ]]
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
[[ 0 -eq 1 ]]
while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter ${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads -i${fn##/}; done < my.fasta.fofn
while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter ${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads -i${fn##/}; done < my.fasta.fofn
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
read fn
cat ../../m54050R1_180201_210150.subreads.fasta
zmw_whitelist_option=""
zmw_whitelist_option=
use_subsampling=0
use_subsampling=0
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_filter streamed-median -
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed "12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
[[ 0 -eq 1 ]]
while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter ${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads -i${fn##*/}; done < my.fasta.fofn
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
zmw_whitelist_option=
use_subsampling=0
use_subsampling=0
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed "12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
read fn
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
[[ 0 -eq 1 ]]
while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter ${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads -i${fn##*/}; done < my.fasta.fofn
read fn
python -m falcon_kit.mains.fasta_filter streamed-median -
read fn
cat ../../m54050R1_180201_210150.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
cat ../../m54050R1_180201_210150.subreads.fasta
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
zmw_whitelist_option=
use_subsampling=0
use_subsampling=0
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed "12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
[[ 0 -eq 1 ]]
while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter ${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads -i${fn##*/}; done < my.fasta.fofn
read fn
fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
cat ../../m54050R1_180201_210150.subreads.fasta
cat ../../m54050R1_180201_210150.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180201_210150.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180201_210150.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180201_210150.subreads.fasta
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
INFO:root:      #113 count=         48,575
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root:      #113 count=         48,575
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root:      #113 count=         48,575
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root:      #113 count=         48,575
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root:      #317 count=         89,000
INFO:root:      #317 count=         89,000
INFO:root:      #317 count=         89,000
INFO:root:      #317 count=         89,000
INFO:root:      #492 count=        178,009
INFO:root:      #492 count=        178,009
INFO:root:      #492 count=        178,009
INFO:root:      #492 count=        178,009
INFO:root:      #507 count=        355,936
INFO:root:      #507 count=        355,936
INFO:root:      #507 count=        355,936
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:      #612 count=        692,452
INFO:root:      #507 count=        355,936
INFO:root:      #612 count=        692,452
INFO:root:      #612 count=        692,452
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
INFO:root:      #113 count=         48,575
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root:      #317 count=         89,000
INFO:root:      #612 count=        692,452
INFO:root:      #492 count=        178,009
INFO:root:      #687 count=      1,370,718
INFO:root:      #687 count=      1,370,718
INFO:root:      #507 count=        355,936
INFO:root:      #687 count=      1,370,718
INFO:root:      #612 count=        692,452
INFO:root:      #687 count=      1,370,718
INFO:root:      #687 count=      1,370,718
INFO:root:      #816 count=      2,688,290
INFO:root:      #816 count=      2,688,290
INFO:root:      #816 count=      2,688,290
INFO:root:      #816 count=      2,688,290
INFO:root:      #816 count=      2,688,290
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #9,932 count=     41,507,957
INFO:root:    #9,932 count=     41,507,957
INFO:root:    #9,932 count=     41,507,957
INFO:root:    #9,932 count=     41,507,957
INFO:root:    #9,932 count=     41,507,957
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #58,663 count=    331,230,374
INFO:root:   #58,663 count=    331,230,374
INFO:root:   #58,663 count=    331,230,374
INFO:root:   #58,663 count=    331,230,374
INFO:root:   #58,663 count=    331,230,374
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #456,327 count=  2,648,877,672
INFO:root:  #456,327 count=  2,648,877,672
INFO:root:  #456,327 count=  2,648,877,672
INFO:root:  #456,327 count=  2,648,877,672
INFO:root:  #456,327 count=  2,648,877,672
read fn
cat ../../m54050R1_180205_001444.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
read fn
cat ../../m54050R1_180205_001444.subreads.fasta
fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
read fn
cat ../../m54050R1_180205_001444.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
read fn
cat ../../m54050R1_180205_001444.subreads.fasta
fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
read fn
cat ../../m54050R1_180205_001444.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180205_001444.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
INFO:root:        #3 count=         44,205
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 357, in
pypeflow 2.1.1
INFO:root:        #5 count=         99,813
main(sys.argv)          # pragma: no cover
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 354, in main
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)
INFO:root:        #6 count=        164,238
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 140, in write_streamed
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
fp_out.write(str(zrec.subread_record))
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 357, in
main(sys.argv)          # pragma: no cover
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 354, in main
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 140, in write_streamed
fp_out.write(str(zrec.subread_record))
IOError: [Errno 32] Broken pipe
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
INFO:root:        #3 count=         44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:       #12 count=        285,279
INFO:root:        #5 count=         99,813
INFO:root:        #6 count=        164,238
cat: write error: Broken pipe
INFO:root:       #12 count=        285,279
INFO:root:       #36 count=        538,708
cat: write error: Broken pipe
[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
[WARNING]Call 'bash -vex build_db.sh' returned 256.
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1443, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1439, in main
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 985, in cmd_build
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 167, in build_db
INFO:root:       #36 count=        538,708
io.syscall('bash -vex {}'.format(script_fn))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call 'bash -vex build_db.sh' returned 256.
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1443, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1439, in main
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 985, in cmd_build
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 167, in build_db
io.syscall('bash -vex {}'.format(script_fn))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call 'bash -vex build_db.sh' returned 256.
INFO:root:      #128 count=        996,561
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:        #3 count=         44,205
INFO:root:      #128 count=        996,561
2019-03-11 17:23:00,650 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2019-03-11 17:23:00,651 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:23:00,651 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:23:00,652 - root - CRITICAL - Error in /pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
INFO:root:        #5 count=         99,813
INFO:root:        #6 count=        164,238
run(**vars(parsed_args))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
2019-03-11 17:23:00,661 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2019-03-11 17:23:00,661 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:23:00,661 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:23:00,662 - root - CRITICAL - Error in /pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
run_cfg_in_tmpdir(cfg, tmpdir, '.')
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
run_bash(bash_template, myinputs, myoutputs, parameters)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
util.system(cmd)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
INFO:root:       #12 count=        285,279
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
run(**vars(parsed_args))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
run_cfg_in_tmpdir(cfg, tmpdir, '.')
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
run_bash(bash_template, myinputs, myoutputs, parameters)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
util.system(cmd)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
Exception: Call '/bin/bash user_script.sh' returned 256.
INFO:root:       #36 count=        538,708
INFO:root:      #233 count=      1,916,542
+++ pwd
++ echo 'FAILURE. Running top in /pylon5/mc5fqip/orca21/0-rawreads/build (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
INFO:root:      #233 count=      1,916,542
INFO:root:      #128 count=        996,561
+++ pwd
++ echo 'FAILURE. Running top in /pylon5/mc5fqip/orca21/0-rawreads/build (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
INFO:root:      #233 count=      1,916,542
INFO:root:      #359 count=      3,717,003
INFO:root:      #359 count=      3,717,003
++ which python
++ which python
++ which top
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
INFO:root:      #359 count=      3,717,003
INFO:root:    #1,096 count=      7,315,096
INFO:root:    #1,096 count=      7,315,096
INFO:root:    #1,096 count=      7,315,096
++ pstree -apl
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #5,484 count=     28,909,411
INFO:root:    #5,484 count=     28,909,411
INFO:root:    #5,484 count=     28,909,411
INFO:root:   #10,469 count=     57,692,955
INFO:root:   #10,469 count=     57,692,955
INFO:root:   #10,469 count=     57,692,955

real	2m21.757s
user	1m58.810s
sys	0m12.239s

finish
echo 'finish code: 1'
++ pstree -apl
INFO:root:   #18,997 count=    115,259,072

real	2m21.946s
user	1m58.074s
sys	0m12.950s

finish
echo 'finish code: 1'
INFO:root:   #18,997 count=    115,259,072
INFO:root:   #18,997 count=    115,259,072
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #71,452 count=    460,654,500
INFO:root:   #71,452 count=    460,654,500
INFO:root:   #71,452 count=    460,654,500
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #632,056 count=  3,684,320,381
INFO:root:  #632,056 count=  3,684,320,381
INFO:root:  #632,056 count=  3,684,320,381
read fn
python -m falcon_kit.mains.fasta_filter streamed-median -
cat ../../m54050R1_180205_102507.subreads.fasta
fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
read fn
cat ../../m54050R1_180205_102507.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
read fn
cat ../../m54050R1_180205_102507.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            201
INFO:root:        #2 count=            422
INFO:root:Counting bytes from "None"
INFO:root:        #4 count=            884
INFO:root:        #1 count=            201
INFO:root:        #8 count=          1,773
INFO:root:        #2 count=            422
INFO:root:       #15 count=          3,383
INFO:root:        #4 count=            884
INFO:root:        #8 count=          1,773
INFO:root:       #29 count=          6,612
INFO:root:       #15 count=          3,383
INFO:root:       #29 count=          6,612
INFO:root:       #57 count=         13,179
INFO:root:       #61 count=         37,191
INFO:root:       #57 count=         13,179
INFO:root:       #61 count=         37,191
Adding 'm54050R1_180205_102507.subreads.fasta.fasta' ...
INFO:root:       #63 count=         76,347
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 357, in
main(sys.argv)          # pragma: no cover
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 354, in main
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 267, in cmd_run_streamed_median_filter
INFO:root:       #71 count=        134,764
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 140, in write_streamed
fp_out.write(str(zrec.subread_record))
IOError: [Errno 32] Broken pipe
INFO:root:      #425 count=        243,370
pypeflow 2.1.1
cat: write error: Broken pipe
INFO:root:      #442 count=        449,237
[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1443, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1439, in main
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 985, in cmd_build
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 167, in build_db
io.syscall('bash -vex {}'.format(script_fn))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call 'bash -vex build_db.sh' returned 256.
INFO:root:      #657 count=        861,953
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            201
INFO:root:        #2 count=            422
INFO:root:        #4 count=            884
INFO:root:        #8 count=          1,773
INFO:root:       #15 count=          3,383
INFO:root:       #29 count=          6,612
INFO:root:       #57 count=         13,179
INFO:root:       #61 count=         37,191
2019-03-11 17:26:48,511 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
2019-03-11 17:26:48,512 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:26:48,512 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
2019-03-11 17:26:48,512 - root - CRITICAL - Error in /pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 357, in
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
main(sys.argv)          # pragma: no cover
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 354, in main
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in
args.func(args)
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 267, in cmd_run_streamed_median_filter
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
run(**vars(parsed_args))
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
run_cfg_in_tmpdir(cfg, tmpdir, '.')
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 140, in write_streamed
run_bash(bash_template, myinputs, myoutputs, parameters)
fp_out.write(str(zrec.subread_record))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
IOError: [Errno 32] Broken pipe
util.system(cmd)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
cat: write error: Broken pipe
[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
INFO:root:      #890 count=      1,685,294
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1443, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1439, in main
args.func(args)
+++ pwd
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 985, in cmd_build
++ echo 'FAILURE. Running top in /pylon5/mc5fqip/orca21/0-rawreads/build (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 167, in build_db
io.syscall('bash -vex {}'.format(script_fn))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call 'bash -vex build_db.sh' returned 256.
++ which python
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
2019-03-11 17:26:48,569 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2019-03-11 17:26:48,569 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:26:48,569 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:26:48,570 - root - CRITICAL - Error in /pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
run(**vars(parsed_args))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
run_cfg_in_tmpdir(cfg, tmpdir, '.')
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
run_bash(bash_template, myinputs, myoutputs, parameters)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
util.system(cmd)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
+++ pwd
++ echo 'FAILURE. Running top in /pylon5/mc5fqip/orca21/0-rawreads/build (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
INFO:root:    #1,937 count=      3,336,834
++ which python
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
INFO:root:    #2,595 count=      6,653,207
++ pstree -apl
++ pstree -apl

real	6m5.337s
user	6m25.131s
sys	0m40.372s

finish
echo 'finish code: 1'

real	6m5.439s
user	6m27.255s
sys	0m37.017s

finish

echo 'finish code: 1'
INFO:root:    #3,605 count=     13,250,960
INFO:root:    #5,067 count=     26,438,315
INFO:root:    #9,631 count=     52,808,553
INFO:root:   #17,876 count=    105,501,455
INFO:root:   #33,517 count=    210,894,049
INFO:root:   #67,557 count=    421,660,075
INFO:root:  #138,208 count=    843,187,868
INFO:root:  #285,310 count=  1,686,257,172
INFO:root:  #586,353 count=  3,372,369,292

read fn

python -m falcon_kit.mains.fasta_filter streamed-median -

cat ../../m54050R1_180205_203502.subreads.fasta

fasta2DB -v raw_reads -im54050R1_180205_203502.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          7,675
INFO:root:        #2 count=         16,492
Adding 'm54050R1_180205_203502.subreads.fasta.fasta' ...
INFO:root:        #5 count=         58,348
INFO:root:        #9 count=        104,030
INFO:root:       #16 count=        175,720
INFO:root:       #37 count=        316,861
INFO:root:       #61 count=        597,735
INFO:root:      #238 count=      1,149,302
INFO:root:      #441 count=      2,258,656
INFO:root:      #675 count=      4,466,062
INFO:root:    #1,626 count=      8,875,130
INFO:root:    #2,915 count=     17,697,362
INFO:root:    #5,228 count=     35,342,884
INFO:root:   #10,157 count=     70,608,036
INFO:root:   #21,921 count=    141,133,151
INFO:root:   #45,516 count=    282,187,689
INFO:root:   #94,223 count=    564,291,704
INFO:root:  #192,243 count=  1,128,491,266
INFO:root:  #403,896 count=  2,256,894,302
INFO:root:  #842,783 count=  4,513,697,570

read fn

cat ../../m54050R1_180206_064434.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180206_064434.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         22,969
Adding 'm54050R1_180206_064434.subreads.fasta.fasta' ...
INFO:root:       #12 count=         69,965
INFO:root:       #16 count=        136,750
INFO:root:       #25 count=        236,704
INFO:root:      #135 count=        436,572
INFO:root:      #318 count=        838,777
INFO:root:      #408 count=      1,607,103
INFO:root:      #997 count=      3,082,830
INFO:root:    #1,569 count=      6,028,296
INFO:root:    #2,711 count=     11,909,437
INFO:root:    #4,608 count=     23,679,111
INFO:root:    #8,977 count=     47,222,976
INFO:root:   #16,294 count=     94,270,393
INFO:root:   #30,908 count=    188,364,603
INFO:root:   #63,905 count=    376,533,093
INFO:root:  #126,065 count=    752,870,381
INFO:root:  #266,009 count=  1,505,537,109
INFO:root:  #546,986 count=  3,010,835,262

read fn

cat ../../m54050R1_180207_203009.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180207_203009.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         15,528
Adding 'm54050R1_180207_203009.subreads.fasta.fasta' ...
INFO:root:       #15 count=         39,397
INFO:root:       #18 count=         71,078
INFO:root:       #30 count=        134,398
INFO:root:      #130 count=        258,831
INFO:root:      #317 count=        522,529
INFO:root:      #515 count=      1,031,741
INFO:root:    #1,222 count=      2,029,291
INFO:root:    #1,526 count=      4,024,089
INFO:root:    #2,969 count=      8,002,205
INFO:root:    #4,995 count=     15,955,492
INFO:root:    #8,842 count=     31,858,883
INFO:root:   #15,711 count=     63,661,809
INFO:root:   #29,486 count=    127,268,481
INFO:root:   #56,761 count=    254,481,249
INFO:root:  #113,227 count=    508,898,324
INFO:root:  #223,394 count=  1,017,720,119
INFO:root:  #453,921 count=  2,035,373,609
INFO:root:  #921,516 count=  4,070,664,628

read fn

cat ../../m54050R1_180208_063535.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180208_063535.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          8,525
INFO:root:        #2 count=         23,659
Adding 'm54050R1_180208_063535.subreads.fasta.fasta' ...
INFO:root:       #23 count=         41,280
INFO:root:       #80 count=         75,598
INFO:root:      #154 count=        151,110
INFO:root:      #176 count=        303,855
INFO:root:      #212 count=        583,374
INFO:root:      #291 count=      1,139,204
INFO:root:      #795 count=      2,263,452
INFO:root:    #1,724 count=      4,450,316
INFO:root:    #3,024 count=      8,838,183
INFO:root:    #5,973 count=     17,581,829
INFO:root:   #10,012 count=     35,045,103
INFO:root:   #17,359 count=     69,972,353
INFO:root:   #32,883 count=    139,816,348
INFO:root:   #60,679 count=    279,500,311
INFO:root:  #122,165 count=    558,856,717
INFO:root:  #244,681 count=  1,117,552,360
INFO:root:  #488,493 count=  2,234,942,780
INFO:root:  #994,012 count=  4,469,739,199

read fn

cat ../../m54050R1_180208_164509.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180208_164509.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          4,339
Adding 'm54050R1_180208_164509.subreads.fasta.fasta' ...
INFO:root:        #3 count=         15,853
INFO:root:        #4 count=         27,413
INFO:root:        #6 count=         46,287
INFO:root:       #16 count=         91,212
INFO:root:       #28 count=        166,573
INFO:root:       #62 count=        316,189
INFO:root:      #152 count=        599,674
INFO:root:      #384 count=      1,157,061
INFO:root:      #553 count=      2,271,138
INFO:root:    #1,620 count=      4,505,203
INFO:root:    #2,788 count=      8,949,439
INFO:root:    #5,146 count=     17,836,899
INFO:root:    #9,403 count=     35,630,402
INFO:root:   #17,575 count=     71,196,569
INFO:root:   #31,238 count=    142,298,020
INFO:root:   #59,125 count=    284,504,058
INFO:root:  #116,757 count=    568,868,162
INFO:root:  #237,793 count=  1,137,592,686
INFO:root:  #476,839 count=  2,275,040,142
INFO:root:  #985,540 count=  4,549,978,182

read fn

python -m falcon_kit.mains.fasta_filter streamed-median -

cat ../../m54050R1_180209_025449.subreads.fasta

fasta2DB -v raw_reads -im54050R1_180209_025449.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          1,124
INFO:root:        #2 count=         11,891
INFO:root:        #3 count=         25,208
Adding 'm54050R1_180209_025449.subreads.fasta.fasta' ...
INFO:root:        #6 count=         61,981
INFO:root:       #21 count=        109,935
INFO:root:       #41 count=        202,473
INFO:root:       #59 count=        407,049
INFO:root:       #97 count=        755,318
INFO:root:      #275 count=      1,457,424
INFO:root:      #766 count=      2,837,474
INFO:root:    #1,194 count=      5,640,860
INFO:root:    #3,117 count=     11,156,713
INFO:root:    #6,039 count=     22,182,616
INFO:root:   #11,133 count=     44,233,551
INFO:root:   #20,876 count=     88,367,082
INFO:root:   #45,433 count=    176,571,256
INFO:root:   #93,158 count=    352,978,876
INFO:root:  #197,343 count=    705,792,445
INFO:root:  #408,824 count=  1,411,418,611
INFO:root:  #829,918 count=  2,822,672,658

read fn

cat ../../m54050R1_180216_131107.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180216_131107.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          9,719
Adding 'm54050R1_180216_131107.subreads.fasta.fasta' ...
INFO:root:        #4 count=         40,017
INFO:root:        #7 count=         75,435
INFO:root:       #12 count=        141,540
INFO:root:       #28 count=        268,737
INFO:root:       #56 count=        481,809
INFO:root:       #90 count=        894,045
INFO:root:      #153 count=      1,692,205
INFO:root:      #308 count=      3,287,634
INFO:root:      #632 count=      6,477,028
INFO:root:    #1,213 count=     12,864,006
INFO:root:    #2,414 count=     25,595,113
INFO:root:    #4,836 count=     51,071,350
INFO:root:    #9,624 count=    102,031,148
INFO:root:   #19,322 count=    203,873,360
INFO:root:   #39,174 count=    407,562,032
INFO:root:   #79,813 count=    814,919,766
INFO:root:  #161,884 count=  1,629,640,257
INFO:root:  #328,680 count=  3,259,065,121

read fn

cat ../../m54050R1_180218_162755.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180218_162755.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          1,529
INFO:root:        #2 count=          4,406
INFO:root:        #3 count=          8,366
Adding 'm54050R1_180218_162755.subreads.fasta.fasta' ...
INFO:root:        #4 count=         16,652
INFO:root:        #6 count=         30,450
INFO:root:        #9 count=         61,269
INFO:root:       #16 count=        119,461
INFO:root:       #33 count=        229,903
INFO:root:       #58 count=        436,578
INFO:root:       #99 count=        837,979
INFO:root:      #180 count=      1,628,014
INFO:root:      #348 count=      3,199,054
INFO:root:      #724 count=      6,331,018
INFO:root:    #1,416 count=     12,598,488
INFO:root:    #2,818 count=     25,145,555
INFO:root:    #5,536 count=     50,203,980
INFO:root:   #11,332 count=    100,325,476
INFO:root:   #22,391 count=    200,535,278
INFO:root:   #46,441 count=    400,954,212
INFO:root:   #95,439 count=    801,779,282
INFO:root:  #193,205 count=  1,603,425,002
INFO:root:  #390,184 count=  3,206,699,992

read fn

cat ../../m54050R1_180219_023725.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180219_023725.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          3,126
INFO:root:        #2 count=         27,682
Adding 'm54050R1_180219_023725.subreads.fasta.fasta' ...
INFO:root:        #5 count=         55,708
INFO:root:        #9 count=        112,213
INFO:root:       #24 count=        221,574
INFO:root:       #48 count=        420,083
INFO:root:       #84 count=        817,466
INFO:root:      #165 count=      1,606,553
INFO:root:      #354 count=      3,182,603
INFO:root:      #684 count=      6,332,043
INFO:root:    #1,535 count=     12,632,150
INFO:root:    #2,846 count=     25,218,301
INFO:root:    #5,568 count=     50,373,610
INFO:root:   #11,334 count=    100,672,557
INFO:root:   #22,836 count=    201,261,550
INFO:root:   #47,372 count=    402,430,842
INFO:root:   #99,810 count=    804,756,672
INFO:root:  #205,037 count=  1,609,413,862
INFO:root:  #411,538 count=  3,218,716,440

read fn

python -m falcon_kit.mains.fasta_filter streamed-median -

cat ../../m54050R1_180219_124713.subreads.fasta

fasta2DB -v raw_reads -im54050R1_180219_124713.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          2,982
INFO:root:        #2 count=          7,310
INFO:root:        #4 count=         16,448
INFO:root:        #7 count=         29,699
INFO:root:        #9 count=         54,065
Adding 'm54050R1_180219_124713.subreads.fasta.fasta' ...
INFO:root:       #15 count=        116,990
INFO:root:       #29 count=        214,682
INFO:root:       #53 count=        410,232
INFO:root:       #93 count=        792,899
INFO:root:      #185 count=      1,559,939
INFO:root:      #351 count=      3,119,435
INFO:root:      #665 count=      6,179,980
INFO:root:    #1,305 count=     12,292,495
INFO:root:    #2,636 count=     24,517,180
INFO:root:    #5,338 count=     48,970,895
INFO:root:   #10,661 count=     97,846,420
INFO:root:   #21,234 count=    195,569,778
INFO:root:   #43,266 count=    391,008,225
INFO:root:   #88,665 count=    781,871,063
INFO:root:  #180,690 count=  1,563,591,507
INFO:root:  #367,307 count=  3,127,029,280

read fn

cat ../../m54050R1_180219_225651.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180219_225651.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         17,456
Adding 'm54050R1_180219_225651.subreads.fasta.fasta' ...
INFO:root:        #4 count=         43,590
INFO:root:        #6 count=         81,470
INFO:root:       #16 count=        171,029
INFO:root:       #33 count=        311,752
INFO:root:       #64 count=        597,228
INFO:root:      #133 count=      1,155,884
INFO:root:      #234 count=      2,278,516
INFO:root:      #498 count=      4,516,541
INFO:root:    #1,099 count=      8,996,162
INFO:root:    #2,156 count=     17,939,634
INFO:root:    #4,225 count=     35,819,795
INFO:root:    #8,501 count=     71,596,984
INFO:root:   #17,182 count=    143,097,510
INFO:root:   #34,653 count=    286,099,824
INFO:root:   #70,918 count=    572,102,016
INFO:root:  #146,805 count=  1,144,100,517
INFO:root:  #298,705 count=  2,288,097,270

read fn

python -m falcon_kit.mains.fasta_filter streamed-median -

cat ../../m54050R1_180220_090637.subreads.fasta

fasta2DB -v raw_reads -im54050R1_180220_090637.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          3,285
INFO:root:        #3 count=         10,081
Adding 'm54050R1_180220_090637.subreads.fasta.fasta' ...
INFO:root:        #4 count=         17,516
INFO:root:        #9 count=         38,436
INFO:root:       #12 count=         65,282
INFO:root:       #16 count=        138,526
INFO:root:       #25 count=        260,077
INFO:root:       #50 count=        480,285
INFO:root:       #99 count=        920,601
INFO:root:      #199 count=      1,768,106
INFO:root:      #393 count=      3,463,559
INFO:root:      #745 count=      6,833,303
INFO:root:    #1,462 count=     13,590,531
INFO:root:    #3,038 count=     27,052,184
INFO:root:    #6,172 count=     53,985,227
INFO:root:   #12,712 count=    107,818,773
INFO:root:   #25,764 count=    215,485,635
INFO:root:   #53,168 count=    430,779,641
INFO:root:  #108,258 count=    861,357,912
INFO:root:  #216,171 count=  1,722,508,762

read fn
#cat fc.fofn | xargs rm -f
DBdust true raw_reads

DBdust true raw_reads
Usage: DBdust [-b] [-w<int(64)>] [-t<double(2.)>] [-m<int(10)>] path:db|dam
-w: DUST algorithm window size.
-t: DUST algorithm threshold.
-m: Record only low-complexity intervals >= this size.
-b: Take into account base composition bias.

[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1443, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1439, in main
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 985, in cmd_build
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 167, in build_db
io.syscall('bash -vex {}'.format(script_fn))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call 'bash -vex build_db.sh' returned 256.
2019-03-11 17:52:25,210 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2019-03-11 17:52:25,210 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:52:25,211 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-11 17:52:25,211 - root - CRITICAL - Error in /pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
run(**vars(parsed_args))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
run_cfg_in_tmpdir(cfg, tmpdir, '.')
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
run_bash(bash_template, myinputs, myoutputs, parameters)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
util.system(cmd)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
+++ pwd
++ echo 'FAILURE. Running top in /pylon5/mc5fqip/orca21/0-rawreads/build (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
++ which python
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ pstree -apl
real	31m42.079s
user	41m32.251s
sys	6m28.394s

finish
echo 'finish code: 1'

stderr.txt

Here is the actual error, from near the bottom of your stderr file. (I copied your post and created a .txt file in WordPad, then copied that into GitHub for you.)
+ DBdust true raw_reads
Usage: DBdust [-b] [-w<int(64)>] [-t<double(2.)>] [-m<int(10)>] <path:db|dam>

      -w: DUST algorithm window size.
      -t: DUST algorithm threshold.
      -m: Record only low-complexity intervals >= this size.
      -b: Take into account base composition bias.

The current problem is that you are providing true to DBdust_option, rather than actual options. It is not a boolean setting.

Thanks, Chris!
On Thu, Mar 14, 2019 at 9:44 AM Christopher Dunn notifications@github.com
wrote:

Closed #109 https://github.com/PacificBiosciences/pbbioconda/issues/109.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/PacificBiosciences/pbbioconda/issues/109#event-2204079542,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AUPt76mXkrDDYsd6IzUUAIMcJhwiN7MXks5vWnx1gaJpZM4buAy_
.

--
Best,
Nicolas Alexandre
PhD Candidate, Integrative Biology
Whiteman Lab
University of California - Berkeley
nalexandre@berkeley.edu kiv@berkeley.edu

Hi Chris,
This run went a lot longer, but returned another error. I can't seem to
find any usage options suggesting I used the wrong commands anywhere else:
executable=${PYPEFLOW_JOB_START_SCRIPT}

executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default

timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then

    echo "timed out waiting for (${executable})"

    exit 77

fi

echo "not executable: '${executable}', waiting ${timeout}s"

sleep 1

timeout=$((timeout-1))

done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
]]

executable=${PYPEFLOW_JOB_START_SCRIPT}

executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default

timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then

    echo "timed out waiting for (${executable})"

    exit 77

fi

echo "not executable: '${executable}', waiting ${timeout}s"

sleep 1

timeout=$((timeout-1))

done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
]]

executable=${PYPEFLOW_JOB_START_SCRIPT}

executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default

timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then

    echo "timed out waiting for (${executable})"

    exit 77

fi

echo "not executable: '${executable}', waiting ${timeout}s"

sleep 1

timeout=$((timeout-1))

done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
]]

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash

executable=${PYPEFLOW_JOB_START_SCRIPT}

executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default

timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then

    echo "timed out waiting for (${executable})"

    exit 77

fi

echo "not executable: '${executable}', waiting ${timeout}s"

sleep 1

timeout=$((timeout-1))

done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
]]

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash

executable=${PYPEFLOW_JOB_START_SCRIPT}

executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default

timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then

    echo "timed out waiting for (${executable})"

    exit 77

fi

echo "not executable: '${executable}', waiting ${timeout}s"

sleep 1

timeout=$((timeout-1))

done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
]]

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash

'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'

cd /pylon5/mc5fqip/orca21/0-rawreads/build

eval '/bin/bash run.sh'

++ /bin/bash run.sh

'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'

cd /pylon5/mc5fqip/orca21/0-rawreads/build

eval '/bin/bash run.sh'

++ /bin/bash run.sh

'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'

cd /pylon5/mc5fqip/orca21/0-rawreads/build

eval '/bin/bash run.sh'

++ /bin/bash run.sh

'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'

cd /pylon5/mc5fqip/orca21/0-rawreads/build

eval '/bin/bash run.sh'

++ /bin/bash run.sh

'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'

cd /pylon5/mc5fqip/orca21/0-rawreads/build

eval '/bin/bash run.sh'

++ /bin/bash run.sh
export PATH=$PATH:/bin

export
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin

PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build

cd /pylon5/mc5fqip/orca21/0-rawreads/build

/bin/bash task.sh

/bin/bash task.sh

export PATH=$PATH:/bin

export
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin

PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build

cd /pylon5/mc5fqip/orca21/0-rawreads/build

/bin/bash task.sh

/bin/bash task.sh

export PATH=$PATH:/bin

export
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin

PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build

cd /pylon5/mc5fqip/orca21/0-rawreads/build

/bin/bash task.sh

/bin/bash task.sh

export PATH=$PATH:/bin

export
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin

PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build

cd /pylon5/mc5fqip/orca21/0-rawreads/build

/bin/bash task.sh

/bin/bash task.sh

export PATH=$PATH:/bin

export
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin

PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build

cd /pylon5/mc5fqip/orca21/0-rawreads/build

/bin/bash task.sh

/bin/bash task.sh

pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
2019-03-14 17:48:11,255 - root - DEBUG - Running
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py
/pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,255 - root - DEBUG - Running
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py
/pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,255 - root - DEBUG - Running
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py
/pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,255 - root - DEBUG - Running
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py
/pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,255 - root - DEBUG - Running
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py
/pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from
'/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,261 - root - DEBUG - {u'bash_template_fn':
u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
         u'input_fofn': u'../../fasta.fofn'},

u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
             u'pypeflow_nproc': 1,

             u'topdir': u'../..'}}

2019-03-14 17:48:11,261 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,262 - root - DEBUG - Checking existence of
u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,261 - root - DEBUG - {u'bash_template_fn':
u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
         u'input_fofn': u'../../fasta.fofn'},

u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
             u'pypeflow_nproc': 1,

             u'topdir': u'../..'}}

2019-03-14 17:48:11,261 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,262 - root - DEBUG - Checking existence of
u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,262 - root - DEBUG - {u'bash_template_fn':
u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
         u'input_fofn': u'../../fasta.fofn'},

u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
             u'pypeflow_nproc': 1,

             u'topdir': u'../..'}}

2019-03-14 17:48:11,262 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,262 - root - DEBUG - {u'bash_template_fn':
u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
         u'input_fofn': u'../../fasta.fofn'},

u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
             u'pypeflow_nproc': 1,

             u'topdir': u'../..'}}

2019-03-14 17:48:11,262 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,262 - root - DEBUG - Checking existence of
u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,262 - root - DEBUG - Checking existence of
u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,263 - root - DEBUG - {u'bash_template_fn':
u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
         u'input_fofn': u'../../fasta.fofn'},

u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
             u'pypeflow_nproc': 1,

             u'topdir': u'../..'}}

2019-03-14 17:48:11,263 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,263 - root - DEBUG - Checking existence of
u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,263 - root - DEBUG - Checking existence of
u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,263 - root - DEBUG - Checking existence of
u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,263 - root - DEBUG - Checking existence of
u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'template.sh' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'template.sh' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'template.sh' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'template.sh' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of
u'template.sh' with timeout=30
2019-03-14 17:48:11,265 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,265 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,266 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,266 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,266 - root - WARNING - CD:
'/pylon5/mc5fqip/orca21/0-rawreads/build' <-
'/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,599 - root - INFO - $('/bin/bash user_script.sh')
2019-03-14 17:48:11,599 - root - INFO - $('/bin/bash user_script.sh')
2019-03-14 17:48:11,599 - root - INFO - $('/bin/bash user_script.sh')
2019-03-14 17:48:11,599 - root - INFO - $('/bin/bash user_script.sh')
2019-03-14 17:48:11,600 - root - INFO - $('/bin/bash user_script.sh')
hostname

hostname

hostname

hostname

hostname

hostname

hostname

hostname

pwd

pwd

date

date

pwd

pwd

date

date

hostname

hostname

pwd

pwd

date

date

pwd

pwd

date

date

pwd

pwd

date

date

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json
--db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn
--length-cutoff-fn=length_cutoff

falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option':
u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy':
u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100
--min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn',
u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed':
12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'',
u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300',
u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u'
--min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy':
True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'',
u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60',
u'target': u'assembly', u'falcon_sense_skip_contained': False,
u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100',
u'length_cutoff': u'-1', u'pa_DBdust_option': u'',
u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False,
u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option':
u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400',
u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24',
'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8
-s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'',
'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy':
u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt':
u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0,
'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option':
u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy':
u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100
--min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn',
u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed':
12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'',
u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300',
u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u'
--min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy':
True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'',
u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60',
u'target': u'assembly', u'falcon_sense_skip_contained': False,
u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100',
u'length_cutoff': u'-1', u'pa_DBdust_option': u'',
u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False,
u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option':
u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400',
u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24',
'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8
-s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'',
'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy':
u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt':
u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0,
'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option':
u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy':
u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100
--min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn',
u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed':
12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'',
u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300',
u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u'
--min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy':
True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'',
u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60',
u'target': u'assembly', u'falcon_sense_skip_contained': False,
u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100',
u'length_cutoff': u'-1', u'pa_DBdust_option': u'',
u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False,
u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option':
u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400',
u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24',
'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8
-s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'',
'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy':
u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt':
u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0,
'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option':
u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy':
u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100
--min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn',
u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed':
12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'',
u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300',
u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u'
--min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy':
True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'',
u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60',
u'target': u'assembly', u'falcon_sense_skip_contained': False,
u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100',
u'length_cutoff': u'-1', u'pa_DBdust_option': u'',
u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False,
u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option':
u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400',
u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24',
'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8
-s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'',
'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy':
u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt':
u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0,
'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option':
u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy':
u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100
--min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn',
u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed':
12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'',
u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300',
u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u'
--min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy':
True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'',
u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60',
u'target': u'assembly', u'falcon_sense_skip_contained': False,
u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100',
u'length_cutoff': u'-1', u'pa_DBdust_option': u'',
u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False,
u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option':
u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400',
u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24',
'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8
-s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'',
'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy':
u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt':
u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0,
'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
module () {  eval /usr/bin/modulecmd bash $*
}
module () {  eval /usr/bin/modulecmd bash $*
}
#!/bin/bash
set -vex

set -vex

echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"

echo PBFALCON_ERRFILE=

set -o pipefail

set -o pipefail

rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
#!/bin/bash
set -vex

set -vex

echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"

echo PBFALCON_ERRFILE=

set -o pipefail

set -o pipefail

rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
module () {  eval /usr/bin/modulecmd bash $*
}
#!/bin/bash
set -vex

set -vex

echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"

echo PBFALCON_ERRFILE=

set -o pipefail

set -o pipefail

rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
module () {  eval /usr/bin/modulecmd bash $*
}
#!/bin/bash
set -vex

set -vex

echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"

echo PBFALCON_ERRFILE=

set -o pipefail

set -o pipefail

module () {  eval /usr/bin/modulecmd bash $*
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
}
#!/bin/bash
set -vex

set -vex

echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"

echo PBFALCON_ERRFILE=

set -o pipefail

set -o pipefail

rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run

rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx

rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx

rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx

rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx

rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx

#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""

zmw_whitelist_option=

use_subsampling=0
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""

use_subsampling=0

if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed

"12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"

fi

zmw_whitelist_option=

use_subsampling=0

use_subsampling=0

if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed

"12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw

[[ 0 -eq 1 ]]

while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter
${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads
-i${fn##*/}; done < my.fasta.fofn
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"

fi

[[ 0 -eq 1 ]]

zmw_whitelist_option=

use_subsampling=0

use_subsampling=0

while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter
${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads
-i${fn##*/}; done < my.fasta.fofn
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed

"12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"

fi

[[ 0 -eq 1 ]]

while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter
${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads
-i${fn##*/}; done < my.fasta.fofn

read fn

cat ../../m54050R1_180201_210150.subreads.fasta

#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""

zmw_whitelist_option=

use_subsampling=0

use_subsampling=0

if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed

"12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"

fi

[[ 0 -eq 1 ]]

while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter
${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads
-i${fn##*/}; done < my.fasta.fofn

read fn

read fn

read fn

fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta

fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

python -m falcon_kit.mains.fasta_filter streamed-median -

cat ../../m54050R1_180201_210150.subreads.fasta

fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta

#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""

zmw_whitelist_option=

use_subsampling=0

use_subsampling=0

if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed

"12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"

fi

[[ 0 -eq 1 ]]

while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter
${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads
-i${fn##*/}; done < my.fasta.fofn

read fn

cat ../../m54050R1_180201_210150.subreads.fasta

fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta

cat ../../m54050R1_180201_210150.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

cat ../../m54050R1_180201_210150.subreads.fasta

fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180201_210150.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180201_210150.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180201_210150.subreads.fasta
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
INFO:root:      #113 count=         48,575
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
INFO:root:      #113 count=         48,575
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
INFO:root:      #113 count=         48,575
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
INFO:root:      #113 count=         48,575
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
INFO:root:      #113 count=         48,575
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root:      #317 count=         89,000
INFO:root:      #317 count=         89,000
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root:      #317 count=         89,000
INFO:root:      #317 count=         89,000
INFO:root:      #317 count=         89,000
INFO:root:      #492 count=        178,009
INFO:root:      #492 count=        178,009
INFO:root:      #492 count=        178,009
INFO:root:      #492 count=        178,009
INFO:root:      #492 count=        178,009
INFO:root:      #507 count=        355,936
INFO:root:      #507 count=        355,936
INFO:root:      #507 count=        355,936
INFO:root:      #507 count=        355,936
INFO:root:      #507 count=        355,936
INFO:root:      #612 count=        692,452
INFO:root:      #612 count=        692,452
INFO:root:      #612 count=        692,452
INFO:root:      #612 count=        692,452
INFO:root:      #612 count=        692,452
INFO:root:      #687 count=      1,370,718
INFO:root:      #687 count=      1,370,718
INFO:root:      #687 count=      1,370,718
INFO:root:      #687 count=      1,370,718
INFO:root:      #687 count=      1,370,718
INFO:root:      #816 count=      2,688,290
INFO:root:      #816 count=      2,688,290
INFO:root:      #816 count=      2,688,290
INFO:root:      #816 count=      2,688,290
INFO:root:      #816 count=      2,688,290
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #9,932 count=     41,507,957
INFO:root:    #9,932 count=     41,507,957
INFO:root:    #9,932 count=     41,507,957
INFO:root:    #9,932 count=     41,507,957
INFO:root:    #9,932 count=     41,507,957
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #58,663 count=    331,230,374
INFO:root:   #58,663 count=    331,230,374
INFO:root:   #58,663 count=    331,230,374
INFO:root:   #58,663 count=    331,230,374
INFO:root:   #58,663 count=    331,230,374
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #456,327 count=  2,648,877,672
INFO:root:  #456,327 count=  2,648,877,672
INFO:root:  #456,327 count=  2,648,877,672
INFO:root:  #456,327 count=  2,648,877,672
INFO:root:  #456,327 count=  2,648,877,672

read fn

python -m falcon_kit.mains.fasta_filter streamed-median -

cat ../../m54050R1_180205_001444.subreads.fasta

fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta

read fn

read fn

read fn

cat ../../m54050R1_180205_001444.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta

cat ../../m54050R1_180205_001444.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta

fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta

cat ../../m54050R1_180205_001444.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

read fn

cat ../../m54050R1_180205_001444.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta

fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_001444.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_001444.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_001444.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_001444.subreads.fasta
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
INFO:root:        #3 count=         44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
INFO:root:        #3 count=         44,205
INFO:root:        #3 count=         44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:        #5 count=         99,813
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:        #5 count=         99,813
INFO:root:        #6 count=        164,238
INFO:root:        #5 count=         99,813
INFO:root:        #6 count=        164,238
INFO:root:Counting bytes from "None"
INFO:root:        #6 count=        164,238
INFO:root:        #1 count=         14,054
INFO:root:        #3 count=         44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:        #5 count=         99,813
INFO:root:        #6 count=        164,238
INFO:root:       #12 count=        285,279
INFO:root:       #12 count=        285,279
INFO:root:       #12 count=        285,279
INFO:root:       #12 count=        285,279
INFO:root:       #36 count=        538,708
INFO:root:       #36 count=        538,708
pypeflow 2.1.1
INFO:root:       #36 count=        538,708
INFO:root:       #36 count=        538,708
INFO:root:      #128 count=        996,561
INFO:root:      #128 count=        996,561
INFO:root:      #128 count=        996,561
INFO:root:      #128 count=        996,561
INFO:root:      #233 count=      1,916,542
INFO:root:      #233 count=      1,916,542
INFO:root:      #233 count=      1,916,542
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
INFO:root:        #3 count=         44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:        #5 count=         99,813
INFO:root:      #233 count=      1,916,542
INFO:root:        #6 count=        164,238
INFO:root:       #12 count=        285,279
INFO:root:       #36 count=        538,708
INFO:root:      #128 count=        996,561
INFO:root:      #359 count=      3,717,003
INFO:root:      #359 count=      3,717,003
INFO:root:      #359 count=      3,717,003
INFO:root:      #359 count=      3,717,003
INFO:root:      #233 count=      1,916,542
INFO:root:      #359 count=      3,717,003
INFO:root:    #1,096 count=      7,315,096
INFO:root:    #1,096 count=      7,315,096
INFO:root:    #1,096 count=      7,315,096
INFO:root:    #1,096 count=      7,315,096
INFO:root:    #1,096 count=      7,315,096
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #5,484 count=     28,909,411
INFO:root:    #5,484 count=     28,909,411
INFO:root:    #5,484 count=     28,909,411
INFO:root:    #5,484 count=     28,909,411
INFO:root:    #5,484 count=     28,909,411
INFO:root:   #10,469 count=     57,692,955
INFO:root:   #10,469 count=     57,692,955
INFO:root:   #10,469 count=     57,692,955
INFO:root:   #10,469 count=     57,692,955
INFO:root:   #10,469 count=     57,692,955
INFO:root:   #18,997 count=    115,259,072
INFO:root:   #18,997 count=    115,259,072
INFO:root:   #18,997 count=    115,259,072
INFO:root:   #18,997 count=    115,259,072
INFO:root:   #18,997 count=    115,259,072
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #71,452 count=    460,654,500
INFO:root:   #71,452 count=    460,654,500
INFO:root:   #71,452 count=    460,654,500
INFO:root:   #71,452 count=    460,654,500
INFO:root:   #71,452 count=    460,654,500
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #632,056 count=  3,684,320,381
INFO:root:  #632,056 count=  3,684,320,381
INFO:root:  #632,056 count=  3,684,320,381
INFO:root:  #632,056 count=  3,684,320,381
INFO:root:  #632,056 count=  3,684,320,381

read fn

read fn

cat ../../m54050R1_180205_102507.subreads.fasta

cat ../../m54050R1_180205_102507.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta

read fn

cat ../../m54050R1_180205_102507.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

read fn

fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta

cat ../../m54050R1_180205_102507.subreads.fasta

fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

read fn

cat ../../m54050R1_180205_102507.subreads.fasta

python -m falcon_kit.mains.fasta_filter streamed-median -

fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta

fasta2DB: raw_reads.db is corrupted, read failed
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
fasta2DB: [WARNING] Could not delete temporary file
m54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            201
INFO:root:Counting bytes from "None"
INFO:root:        #2 count=            422
INFO:root:        #4 count=            884
INFO:root:        #8 count=          1,773
INFO:root:       #15 count=          3,383
INFO:root:        #1 count=            201
INFO:root:        #2 count=            422
INFO:root:        #4 count=            884
INFO:root:        #8 count=          1,773
INFO:root:       #15 count=          3,383
INFO:root:       #29 count=          6,612
INFO:root:       #57 count=         13,179
INFO:root:       #61 count=         37,191
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
pypeflow 2.1.1
INFO:root:       #29 count=          6,612
INFO:root:       #57 count=         13,179
INFO:root:       #61 count=         37,191
Adding 'm54050R1_180205_102507.subreads.fasta.fasta' ...
INFO:root:       #63 count=         76,347
INFO:root:       #71 count=        134,764
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            201
INFO:root:        #2 count=            422
INFO:root:        #4 count=            884
INFO:root:        #8 count=          1,773
INFO:root:       #15 count=          3,383
INFO:root:       #29 count=          6,612
INFO:root:       #57 count=         13,179
INFO:root:       #61 count=         37,191
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
INFO:root:      #425 count=        243,370
"__main__", fname, loader, pkg_name)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
"__main__", fname, loader, pkg_name)

exec code in run_globals

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 357, in
exec code in run_globals

main(sys.argv)          # pragma: no cover

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 354, in main
args.func(args)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 140, in write_streamed
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 357, in
main(sys.argv)          # pragma: no cover

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 354, in main
args.func(args)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 140, in write_streamed
fp_out.write(str(zrec.subread_record))

fp_out.write(str(zrec.subread_record))

IOError: [Errno 32] Broken pipe
IOError: [Errno 32] Broken pipe
INFO:root:      #442 count=        449,237
cat: write error: Broken pipe
cat: write error: Broken pipe
[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
"__main__", fname, loader, pkg_name)

[WARNING]Call 'bash -vex build_db.sh' returned 256.
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
exec code in run_globals

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1443, in
main()

"__main__", fname, loader, pkg_name)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1439, in main
exec code in run_globals

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1443, in
main()

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1439, in main
args.func(args)

INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            201
INFO:root:        #2 count=            422
INFO:root:        #4 count=            884
INFO:root:        #8 count=          1,773
args.func(args)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 985, in cmd_build
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 167, in build_db
io.syscall('bash -vex {}'.format(script_fn))

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py",
line 29, in syscall
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 985, in cmd_build
INFO:root:       #15 count=          3,383
INFO:root:       #29 count=          6,612
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)

INFO:root:       #57 count=         13,179
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 167, in build_db
INFO:root:       #61 count=         37,191
io.syscall('bash -vex {}'.format(script_fn))

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py",
line 29, in syscall
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
"__main__", fname, loader, pkg_name)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
exec code in run_globals

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 357, in
INFO:root:Counting bytes from "None"
main(sys.argv)          # pragma: no cover

INFO:root:        #1 count=            201
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 354, in main
INFO:root:        #2 count=            422
args.func(args)

INFO:root:        #4 count=            884
INFO:root:        #8 count=          1,773
INFO:root:      #657 count=        861,953
INFO:root:       #15 count=          3,383
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)

INFO:root:       #29 count=          6,612
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 140, in write_streamed
fp_out.write(str(zrec.subread_record))

IOError: [Errno 32] Broken pipe
INFO:root:       #57 count=         13,179
INFO:root:       #61 count=         37,191
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
"__main__", fname, loader, pkg_name)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
exec code in run_globals

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 357, in
main(sys.argv)          # pragma: no cover

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 354, in main
args.func(args)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py",
line 140, in write_streamed
fp_out.write(str(zrec.subread_record))

IOError: [Errno 32] Broken pipe
cat: write error: Broken pipe
[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
"__main__", fname, loader, pkg_name)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
exec code in run_globals

raise Exception(msg)

Exception: Call 'bash -vex build_db.sh' returned 256.
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1443, in
INFO:root:      #890 count=      1,685,294
raise Exception(msg)

Exception: Call 'bash -vex build_db.sh' returned 256.
cat: write error: Broken pipe
[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
174, in _run_module_as_main
"__main__", fname, loader, pkg_name)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line
72, in _run_code
exec code in run_globals

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1443, in
main()

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 1439, in main
args.func(args)

File
"/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py",
line 985, in cmd_build
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)

File
"/pylon5/mc5fqip/orca21/

PacificBiosciences/FALCON#685

This was originally PacificBiosciences/FALCON#685
I wrote:

You have to look into 0-rawreads/build. But please also post conda list. Someone else posted an odd bug, and I suspect that people are getting an inconsistent version of DAZZ_DB, which we provide via pb-dazzler.

You've got to look into 0-rawreads/build for stderr. You may have to debug this yourself by cd 0-rawreads/build and running parts of user_script.sh yourself. It will be difficult for us to debug this remotely, but with any luck, the answer will be obvious to you when to see stderr or run the commands yourself.

Hi,
I got a similar issue to run pb-assembly in cluster.
I checked the stderr file in '0-rawreads/build', and found the following error message:
**
pypeflow 2.1.1
pypeflow 2.1.1
raise Exception(msg)
Exception: Call 'bash -vex build_db.sh' returned 256.
2019-03-16 15:04:37,109 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2019-03-16 15:04:37,109 - root - WARNING - CD: '/BIGDATA1/wbgcas_tfeng_1/project/pb_assembly_test/result/0-rawreads/build' -> '/BIGDATA1/wbgcas_tfeng_1/project/pb_assembly_test/result/0-rawreads/build'
2019-03-16 15:04:37,109 - root - WARNING - CD: '/BIGDATA1/wbgcas_tfeng_1/project/pb_assembly_test/result/0-rawreads/build' -> '/BIGDATA1/wbgcas_tfeng_1/project/pb_assembly_test/result/0-rawreads/build'
2019-03-16 15:04:37,109 - root - CRITICAL - Error in /BIGDATA1/app/anaconda2/5.0.1-gcc-4.8.5/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/BIGDATA1/wbgcas_tfeng_1/project/pb_assembly_test/result/0-rawreads/build/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
File "/BIGDATA1/app/anaconda2/5.0.1-gcc-4.8.5/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/BIGDATA1/app/anaconda2/5.0.1-gcc-4.8.5/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/BIGDATA1/app/anaconda2/5.0.1-gcc-4.8.5/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in
main()
File "/BIGDATA1/app/anaconda2/5.0.1-gcc-4.8.5/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
run(**vars(parsed_args))
File "/BIGDATA1/app/anaconda2/5.0.1-gcc-4.8.5/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
run_cfg_in_tmpdir(cfg, tmpdir, '.')
File "/BIGDATA1/app/anaconda2/5.0.1-gcc-4.8.5/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
run_bash(bash_template, myinputs, myoutputs, parameters)
File "/BIGDATA1/app/anaconda2/5.0.1-gcc-4.8.5/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
util.system(cmd)
File "/BIGDATA1/app/anaconda2/5.0.1-gcc-4.8.5/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
**
How can I fix this ?  Thanks
THIS IS THE DFG FILE:
Input
[General]
input_fofn=/BIGDATA1/wbgcas_tfeng_1/projcet/pb_assembly_test/input.fofn
input_type=raw
pa_DBdust_option=
pa_fasta_filter_option=pass
target=assembly
skip_checks=False
LA4Falcon_preload=false
Data Partitioning
pa_DBsplit_option=-x500 -s200
ovlp_DBsplit_option=-x500 -s200
Repeat Masking
pa_HPCTANmask_option=
pa_REPmask_code=0,300;0,300;0,300
####Pre-assembly
genome_size=0
seed_coverage=20
length_cutoff=1000
pa_HPCdaligner_option=-v -B128 -M24
pa_daligner_option=-e.8 -l2000 -k18 -h480  -w8 -s100
falcon_sense_option=--output-multi --min-idt 0.70 --min-cov 2 --max-n-read 1800
falcon_sense_greedy=False
####Pread overlapping
ovlp_daligner_option=-e.9 -l2500 -k24 -h1024 -w6 -s100
ovlp_HPCdaligner_option=-v -B128 -M24
####Final Assembly
overlap_filtering_setting=--max-diff 100 --max-cov 100 --min-cov 2
fc_ovlp_to_graph_option=
length_cutoff_pr=1000
[job.defaults]
job_type=slurm
pwatcher_type=blocking
JOB_QUEUE = free
JOB_NAME = 200kb_test
MB = 32768
NPROC = 6
njobs = 32
submit = srun --wait=0 -p ${JOB_QUEUE} -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} --mem-per-cpu=${MB}M --cpus-per-task=${NPROC} ${JOB_SCRIPT}
[job.step.da]
NPROC=4
MB=32768
njobs=240
[job.step.la]
NPROC=4
MB=32768
njobs=240
[job.step.cns]
NPROC=8
MB=65536
njobs=240
[job.step.pda]
NPROC=4
MB=32768
njobs=240
[job.step.pla]
NPROC=4
MB=32768
njobs=240
[job.step.asm]
NPROC=24
MB=64608
njobs=1

I copied your message into WordPad and found this:
cat: write error: Broken pipe

There are a variety of ways that can happen, e.g. you could have run out of disk-space, or something could have killed one of the processes. You'll need to debug this at your end. Go into the failing task directory (0-rawreads/build/ I think), and run bash -vex user_script.sh. From there, you might be able to run specific bash commands and figure out what's going on.

Hi Chris,
I was able to narrow it down to this error:
#cat fc.fofn | xargs rm -f
DBdust  raw_reads

DBdust raw_reads
Message from root@br006.pvt.bridges.psc.edu on  at 15:14 ...
pid 17944 (DBdust) killed: rss memory 4194.66 MB exceeded limit 4096 MB
Computations using more than 1800 cpu seconds
and/or 4096 MB of memory should use the batch system.

I have the following batch configuration:
#SBATCH -t 336:00:00
#SBATCH -N 1
#SBATCH --mem=1TB
#SBATCH --mem-per-cpu=5000M
#SBATCH --cpus-per-task=4
Nicolas
On Sat, Mar 16, 2019 at 7:32 AM Christopher Dunn notifications@github.com
wrote:

Reopened #109
https://github.com/PacificBiosciences/pbbioconda/issues/109.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/PacificBiosciences/pbbioconda/issues/109#event-2208196218,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AUPt7-2ukDp1iDfBg3m4BvDGX36tBnwBks5vXQBugaJpZM4buAy_
.

--
Best,
Nicolas Alexandre
PhD Candidate, Integrative Biology
Whiteman Lab
University of California - Berkeley
nalexandre@berkeley.edu kiv@berkeley.edu

Yep, you ran out of memory. And DBdust is single-threaded. You can try different options to DBdust. (You can try to ask thegenemyers questions about that.) Or you can submit to a machine with more RAM. This is definitely not a Falcon error.

With the following Config file:
Input for FALCON 3/18/19
[General]
input_fofn=fasta.fofn
input_type=raw
pa_DBdust_option=
pa_fasta_filter_option=streamed-median
target=assembly
skip_checks=False
LA4Falcon_preload=false
genome_size = 1100000000
Data Partitioning
pa_DBsplit_option=-x500 -s50
ovlp_DBsplit_option=-x500 -s50
Repeat Masking
pa_HPCTANmask_option=
pa_REPmask_code=0,300;0,300;0,300
####Pre-assembly
genome_size=1100000000
seed_coverage=30
length_cutoff=1000
pa_HPCdaligner_option=-v -B128 -M24
pa_daligner_option=-e.8 -l2000 -k18 -h480 -w8 -s100
falcon_sense_option=--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400
falcon_sense_greedy=False
####Pread overlapping
ovlp_daligner_option=-e.9 -l2500 -k24 -h1024 -w6 -s100
ovlp_HPCdaligner_option=-v -B128 -M24
####Final Assembly
overlap_filtering_setting=--max-diff 100 --max-cov 100 --min-cov 2
fc_ovlp_to_graph_option=
length_cutoff_pr=1000
[job.defaults]
pwatcher_type = blocking
submit = srun --wait 2
-p ${JOB_QUEUE}
-J ${JOB_NAME}
-o ${JOB_STDOUT}
-e ${JOB_STDERR}
--mem-per-cpu=${MB}M
--cpus-per-task=${NPROC}
${JOB_SCRIPT}
njobs = 32
And the following script:
#!/bin/bash
#SBATCH -p LM
#SBATCH -t 336:00:00
#SBATCH -N 1
#SBATCH --mem=3000GB
#SBATCH --cpus-per-task=4
source activate myenv
run it!
fc_run fc_run.cfg
I have the following new stderr output: Any help is appreciated.
executable=${PYPEFLOW_JOB_START_SCRIPT}

executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash ]]
executable=${PYPEFLOW_JOB_START_SCRIPT}
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash ]]
executable=${PYPEFLOW_JOB_START_SCRIPT}
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash ]]

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
executable=${PYPEFLOW_JOB_START_SCRIPT}
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash ]]

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
executable=${PYPEFLOW_JOB_START_SCRIPT}
executable=/pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
timeout=60

Wait up to timeout seconds for the executable to become "executable",
then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
if [[ "${timeout}" == "0" ]]; then
echo "timed out waiting for (${executable})"
exit 77
fi
echo "not executable: '${executable}', waiting ${timeout}s"
sleep 1
timeout=$((timeout-1))
done

[[ ! -x /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash ]]

/bin/bash ${executable}

/bin/bash /pylon5/mc5fqip/orca21/0-rawreads/build/run-P2c7b8ae23d7f98.bash
'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
cd /pylon5/mc5fqip/orca21/0-rawreads/build
eval '/bin/bash run.sh'
++ /bin/bash run.sh
'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
cd /pylon5/mc5fqip/orca21/0-rawreads/build
eval '/bin/bash run.sh'
++ /bin/bash run.sh
'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
cd /pylon5/mc5fqip/orca21/0-rawreads/build
eval '/bin/bash run.sh'
++ /bin/bash run.sh
'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
cd /pylon5/mc5fqip/orca21/0-rawreads/build
eval '/bin/bash run.sh'
++ /bin/bash run.sh
'[' '!' -d /pylon5/mc5fqip/orca21/0-rawreads/build ']'
cd /pylon5/mc5fqip/orca21/0-rawreads/build
eval '/bin/bash run.sh'
++ /bin/bash run.sh
export PATH=$PATH:/bin
export PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
/bin/bash task.sh
export PATH=$PATH:/bin
export PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
/bin/bash task.sh
export PATH=$PATH:/bin
export PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
/bin/bash task.sh
export PATH=$PATH:/bin
export PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
/bin/bash task.sh
export PATH=$PATH:/bin
export PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
PATH=/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/bin:/pylon5/mc5fqip/orca21/anaconda3/condabin:/home/orca21/.linuxbrew/Homebrew/sbin:/home/orca21/.linuxbrew/Homebrew/bin:/bin:/sbin:/pylon5/mc5fqip/orca21/anaconda3/bin:/pylon5/mc5fqip/orca21/bin:/usr/lib64/qt-3.3/bin:/opt/packages/xdusage/2.1-1/bin:/opt/intel/advisor_2018.4.0.574144/bin64:/opt/intel/vtune_amplifier_2018.4.0.573462/bin64:/opt/intel/inspector_2018.4.0.574143/bin64:/opt/intel/itac/2018.4.025/intel64/bin:/opt/intel/clck/2018.3/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/bin/intel64:/opt/intel/compilers_and_libraries_2018.5.274/linux/mpi/intel64/bin:/opt/packages/slurm/default/bin:/opt/packages/allocations:/opt/packages/interact/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/puppet/bin:/opt/intel/parallel_studio_xe_2018.4.057/bin:/bin:/sbin:/opt/puppetlabs/bin:/home/orca21/.linuxbrew/Homebrew/bin:/home/orca21/bin:/home/orca21/.local/bin:/home/orca21/bin:/bin
cd /pylon5/mc5fqip/orca21/0-rawreads/build
cd /pylon5/mc5fqip/orca21/0-rawreads/build
/bin/bash task.sh
/bin/bash task.sh
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
2019-03-14 17:48:11,255 - root - DEBUG - Running "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py /pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,255 - root - DEBUG - Running "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py /pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,255 - root - DEBUG - Running "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py /pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,255 - root - DEBUG - Running "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py /pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,255 - root - DEBUG - Running "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py /pylon5/mc5fqip/orca21/0-rawreads/build/task.json"
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,259 - root - DEBUG - Checking existence of '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json' with timeout=30
2019-03-14 17:48:11,260 - root - DEBUG - Loading JSON from '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json'
2019-03-14 17:48:11,261 - root - DEBUG - {u'bash_template_fn': u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-14 17:48:11,261 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,262 - root - DEBUG - Checking existence of u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,261 - root - DEBUG - {u'bash_template_fn': u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-14 17:48:11,261 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,262 - root - DEBUG - Checking existence of u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,262 - root - DEBUG - {u'bash_template_fn': u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-14 17:48:11,262 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,262 - root - DEBUG - {u'bash_template_fn': u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-14 17:48:11,262 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,262 - root - DEBUG - Checking existence of u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,262 - root - DEBUG - Checking existence of u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,263 - root - DEBUG - {u'bash_template_fn': u'template.sh',
u'inputs': {u'config': u'../../General_config.json',
u'input_fofn': u'../../fasta.fofn'},
u'outputs': {u'db': u'raw_reads.db', u'length_cutoff': u'length_cutoff'},
u'parameters': {u'pypeflow_mb': 4000,
u'pypeflow_nproc': 1,
u'topdir': u'../..'}}
2019-03-14 17:48:11,263 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,263 - root - DEBUG - Checking existence of u'../../fasta.fofn' with timeout=30
2019-03-14 17:48:11,263 - root - DEBUG - Checking existence of u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,263 - root - DEBUG - Checking existence of u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,263 - root - DEBUG - Checking existence of u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of u'../../General_config.json' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2019-03-14 17:48:11,264 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2019-03-14 17:48:11,265 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,265 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,266 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,266 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,266 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' <- '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:48:11,599 - root - INFO - $('/bin/bash user_script.sh')
2019-03-14 17:48:11,599 - root - INFO - $('/bin/bash user_script.sh')
2019-03-14 17:48:11,599 - root - INFO - $('/bin/bash user_script.sh')
2019-03-14 17:48:11,599 - root - INFO - $('/bin/bash user_script.sh')
2019-03-14 17:48:11,600 - root - INFO - $('/bin/bash user_script.sh')
hostname
hostname
hostname
hostname
hostname
hostname
hostname
hostname
pwd
pwd
date
date
pwd
pwd
date
date
hostname
hostname
pwd
pwd
date
date
pwd
pwd
date
date
pwd
pwd
date
date

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db  build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff

python -m falcon_kit.mains.dazzler --config-fn=../../General_config.json --db-fn=raw_reads.db build --input-fofn-fn=../../fasta.fofn --length-cutoff-fn=length_cutoff
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]Log-level: INFO
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option': u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy': u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100 --min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn', u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed': 12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'', u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300', u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u' --min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy': True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'', u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60', u'target': u'assembly', u'falcon_sense_skip_contained': False, u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100', u'length_cutoff': u'-1', u'pa_DBdust_option': u'', u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False, u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option': u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400', u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24', 'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'', 'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy': u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0, 'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option': u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy': u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100 --min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn', u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed': 12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'', u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300', u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u' --min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy': True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'', u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60', u'target': u'assembly', u'falcon_sense_skip_contained': False, u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100', u'length_cutoff': u'-1', u'pa_DBdust_option': u'', u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False, u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option': u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400', u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24', 'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'', 'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy': u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0, 'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option': u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy': u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100 --min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn', u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed': 12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'', u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300', u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u' --min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy': True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'', u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60', u'target': u'assembly', u'falcon_sense_skip_contained': False, u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100', u'length_cutoff': u'-1', u'pa_DBdust_option': u'', u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False, u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option': u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400', u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24', 'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'', 'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy': u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0, 'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option': u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy': u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100 --min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn', u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed': 12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'', u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300', u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u' --min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy': True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'', u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60', u'target': u'assembly', u'falcon_sense_skip_contained': False, u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100', u'length_cutoff': u'-1', u'pa_DBdust_option': u'', u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False, u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option': u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400', u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24', 'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'', 'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy': u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0, 'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]config('../../General_config.json'):
{u'falcon_sense_greedy': False, u'pa_fasta_filter_option': u'streamed-median', u'seed_coverage': u'30', u'pa_subsample_strategy': u'random', u'overlap_filtering_setting': u'--max-diff 100 --max-cov 100 --min-cov 2', u'length_cutoff_pr': u'1000', u'input_fofn': u'fasta.fofn', u'pa_HPCdaligner_option': u'-v -B128 -M24', u'pa_subsample_random_seed': 12345, u'ver': u'100', u'bestn': 12, u'pa_HPCTANmask_option': u'', u'input_type': u'raw', u'pa_REPmask_code': u'0,300;0,300;0,300', u'pa_DBsplit_option': u'-a -x500 -s200', u'fc_ovlp_to_graph_option': u' --min-len 1000', u'genome_size': u'1100000000', u'avoid_text_file_busy': True, u'pa_dazcon_option': u'-j 4 -x -l 500', u'ovlp_DBdust_option': u'', u'skip_checks': False, u'ovlp_daligner_option': u'-e.96 -s1000 -h60', u'target': u'assembly', u'falcon_sense_skip_contained': False, u'pa_daligner_option': u'-e0.8 -l2000 -k18 -h480  -w8 -s100', u'length_cutoff': u'-1', u'pa_DBdust_option': u'', u'ovlp_HPCdaligner_option': u'-v -M24 -l500', u'LA4Falcon_preload': False, u'pa_subsample_coverage': 0, u'dazcon': False, u'falcon_sense_option': u'--output-multi --min-idt 0.70 --min-cov 3 --max-n-read 400', u'ovlp_DBsplit_option': u'-x500 -s200'}
[INFO]our subset of config:
{'daligner_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 -v -B128 -M24', 'genome_size': 1100000000, 'TANmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_random_seed': 12345, 'DBdust_opt': u'', 'seed_coverage': 30.0, 'user_length_cutoff': -1, 'subsample_strategy': u'random', 'fasta_filter_option': u'streamed-median', 'REPmask_opt': u'-e0.8 -l2000 -k18 -h480  -w8 -s100 ', 'subsample_coverage': 0, 'DBsplit_opt': u'-a -x500 -s200'}
[INFO]Building rdb from '../../fasta.fofn', to write 'raw_reads.db'
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
[INFO]$('bash -vex build_db.sh')
module () {  eval /usr/bin/modulecmd bash $*
}
module () {  eval /usr/bin/modulecmd bash $*
}
#!/bin/bash
set -vex
set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
echo PBFALCON_ERRFILE=
set -o pipefail
set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
#!/bin/bash
set -vex
set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
echo PBFALCON_ERRFILE=
set -o pipefail
set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
module () {  eval /usr/bin/modulecmd bash $*
}
#!/bin/bash
set -vex
set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
echo PBFALCON_ERRFILE=
set -o pipefail
set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
module () {  eval /usr/bin/modulecmd bash $*
}
#!/bin/bash
set -vex
set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
echo PBFALCON_ERRFILE=
set -o pipefail
set -o pipefail
module () {  eval /usr/bin/modulecmd bash $*
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
}
#!/bin/bash
set -vex
set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
echo PBFALCON_ERRFILE=
set -o pipefail
set -o pipefail
rm -f raw_reads.db raw_reads.dam .raw_reads.* # in case of re-run
rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx
rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx
rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx
rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx
rm -f raw_reads.db raw_reads.dam .raw_reads.bps .raw_reads.idx
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
zmw_whitelist_option=
use_subsampling=0
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
use_subsampling=0
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed "12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
zmw_whitelist_option=
use_subsampling=0
use_subsampling=0
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed "12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
[[ 0 -eq 1 ]]
while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter ${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads -i${fn##*/}; done < my.fasta.fofn
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
[[ 0 -eq 1 ]]
zmw_whitelist_option=
use_subsampling=0
use_subsampling=0
while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter ${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads -i${fn##*/}; done < my.fasta.fofn
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed "12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
[[ 0 -eq 1 ]]
while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter ${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads -i${fn##*/}; done < my.fasta.fofn
read fn
cat ../../m54050R1_180201_210150.subreads.fasta
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
zmw_whitelist_option=
use_subsampling=0
use_subsampling=0
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed "12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
[[ 0 -eq 1 ]]
while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter ${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads -i${fn##*/}; done < my.fasta.fofn
read fn
read fn
read fn
fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
python -m falcon_kit.mains.fasta_filter streamed-median -
cat ../../m54050R1_180201_210150.subreads.fasta
fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
#fc_fasta2fasta < my.fasta.fofn >| fc.fofn
zmw_whitelist_option=""
zmw_whitelist_option=
use_subsampling=0
use_subsampling=0
if [[ $use_subsampling -eq 1 ]]; then
python -m falcon_kit.mains.fasta_subsample --coverage "0" --random-seed "12345" --strategy "random" --genome-size "1100000000" "my.fasta.fofn" zmw
zmw_whitelist_option="--zmw-whitelist-fn zmw.whitelist.json"
fi
[[ 0 -eq 1 ]]
while read fn; do  cat  ${fn} | python -m falcon_kit.mains.fasta_filter ${zmw_whitelist_option} streamed-median - | fasta2DB -v raw_reads -i${fn##*/}; done < my.fasta.fofn
read fn
cat ../../m54050R1_180201_210150.subreads.fasta
fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180201_210150.subreads.fasta
cat ../../m54050R1_180201_210150.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
cat ../../m54050R1_180201_210150.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180201_210150.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180201_210150.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180201_210150.subreads.fasta
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
INFO:root:      #113 count=         48,575
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
INFO:root:      #113 count=         48,575
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
INFO:root:      #113 count=         48,575
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
INFO:root:      #113 count=         48,575
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            224
INFO:root:        #2 count=            839
INFO:root:        #3 count=          2,102
INFO:root:        #4 count=          4,170
INFO:root:        #6 count=          8,348
INFO:root:        #9 count=         14,686
INFO:root:       #11 count=         28,274
INFO:root:      #113 count=         48,575
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root:      #317 count=         89,000
INFO:root:      #317 count=         89,000
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
Adding 'm54050R1_180201_210150.subreads.fasta.fasta' ...
INFO:root:      #317 count=         89,000
INFO:root:      #317 count=         89,000
INFO:root:      #317 count=         89,000
INFO:root:      #492 count=        178,009
INFO:root:      #492 count=        178,009
INFO:root:      #492 count=        178,009
INFO:root:      #492 count=        178,009
INFO:root:      #492 count=        178,009
INFO:root:      #507 count=        355,936
INFO:root:      #507 count=        355,936
INFO:root:      #507 count=        355,936
INFO:root:      #507 count=        355,936
INFO:root:      #507 count=        355,936
INFO:root:      #612 count=        692,452
INFO:root:      #612 count=        692,452
INFO:root:      #612 count=        692,452
INFO:root:      #612 count=        692,452
INFO:root:      #612 count=        692,452
INFO:root:      #687 count=      1,370,718
INFO:root:      #687 count=      1,370,718
INFO:root:      #687 count=      1,370,718
INFO:root:      #687 count=      1,370,718
INFO:root:      #687 count=      1,370,718
INFO:root:      #816 count=      2,688,290
INFO:root:      #816 count=      2,688,290
INFO:root:      #816 count=      2,688,290
INFO:root:      #816 count=      2,688,290
INFO:root:      #816 count=      2,688,290
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #1,375 count=      5,277,506
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #2,367 count=     10,451,057
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #4,437 count=     20,814,122
INFO:root:    #9,932 count=     41,507,957
INFO:root:    #9,932 count=     41,507,957
INFO:root:    #9,932 count=     41,507,957
INFO:root:    #9,932 count=     41,507,957
INFO:root:    #9,932 count=     41,507,957
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #17,666 count=     82,897,567
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #32,172 count=    165,669,678
INFO:root:   #58,663 count=    331,230,374
INFO:root:   #58,663 count=    331,230,374
INFO:root:   #58,663 count=    331,230,374
INFO:root:   #58,663 count=    331,230,374
INFO:root:   #58,663 count=    331,230,374
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #111,773 count=    662,326,916
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #228,302 count=  1,324,520,840
INFO:root:  #456,327 count=  2,648,877,672
INFO:root:  #456,327 count=  2,648,877,672
INFO:root:  #456,327 count=  2,648,877,672
INFO:root:  #456,327 count=  2,648,877,672
INFO:root:  #456,327 count=  2,648,877,672
read fn
python -m falcon_kit.mains.fasta_filter streamed-median -
cat ../../m54050R1_180205_001444.subreads.fasta
fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
read fn
read fn
read fn
cat ../../m54050R1_180205_001444.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
cat ../../m54050R1_180205_001444.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
cat ../../m54050R1_180205_001444.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
read fn
cat ../../m54050R1_180205_001444.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180205_001444.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180205_001444.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180205_001444.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180205_001444.subreads.fasta
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180205_001444.subreads.fasta
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
INFO:root:        #3 count=         44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
INFO:root:        #3 count=         44,205
INFO:root:        #3 count=         44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:        #5 count=         99,813
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:        #5 count=         99,813
INFO:root:        #6 count=        164,238
INFO:root:        #5 count=         99,813
INFO:root:        #6 count=        164,238
INFO:root:Counting bytes from "None"
INFO:root:        #6 count=        164,238
INFO:root:        #1 count=         14,054
INFO:root:        #3 count=         44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:        #5 count=         99,813
INFO:root:        #6 count=        164,238
INFO:root:       #12 count=        285,279
INFO:root:       #12 count=        285,279
INFO:root:       #12 count=        285,279
INFO:root:       #12 count=        285,279
INFO:root:       #36 count=        538,708
INFO:root:       #36 count=        538,708
pypeflow 2.1.1
INFO:root:       #36 count=        538,708
INFO:root:       #36 count=        538,708
INFO:root:      #128 count=        996,561
INFO:root:      #128 count=        996,561
INFO:root:      #128 count=        996,561
INFO:root:      #128 count=        996,561
INFO:root:      #233 count=      1,916,542
INFO:root:      #233 count=      1,916,542
INFO:root:      #233 count=      1,916,542
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         14,054
INFO:root:        #3 count=         44,205
Adding 'm54050R1_180205_001444.subreads.fasta.fasta' ...
INFO:root:        #5 count=         99,813
INFO:root:      #233 count=      1,916,542
INFO:root:        #6 count=        164,238
INFO:root:       #12 count=        285,279
INFO:root:       #36 count=        538,708
INFO:root:      #128 count=        996,561
INFO:root:      #359 count=      3,717,003
INFO:root:      #359 count=      3,717,003
INFO:root:      #359 count=      3,717,003
INFO:root:      #359 count=      3,717,003
INFO:root:      #233 count=      1,916,542
INFO:root:      #359 count=      3,717,003
INFO:root:    #1,096 count=      7,315,096
INFO:root:    #1,096 count=      7,315,096
INFO:root:    #1,096 count=      7,315,096
INFO:root:    #1,096 count=      7,315,096
INFO:root:    #1,096 count=      7,315,096
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #2,198 count=     14,516,279
INFO:root:    #5,484 count=     28,909,411
INFO:root:    #5,484 count=     28,909,411
INFO:root:    #5,484 count=     28,909,411
INFO:root:    #5,484 count=     28,909,411
INFO:root:    #5,484 count=     28,909,411
INFO:root:   #10,469 count=     57,692,955
INFO:root:   #10,469 count=     57,692,955
INFO:root:   #10,469 count=     57,692,955
INFO:root:   #10,469 count=     57,692,955
INFO:root:   #10,469 count=     57,692,955
INFO:root:   #18,997 count=    115,259,072
INFO:root:   #18,997 count=    115,259,072
INFO:root:   #18,997 count=    115,259,072
INFO:root:   #18,997 count=    115,259,072
INFO:root:   #18,997 count=    115,259,072
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #37,811 count=    230,392,828
INFO:root:   #71,452 count=    460,654,500
INFO:root:   #71,452 count=    460,654,500
INFO:root:   #71,452 count=    460,654,500
INFO:root:   #71,452 count=    460,654,500
INFO:root:   #71,452 count=    460,654,500
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #143,311 count=    921,176,072
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #301,804 count=  1,842,225,458
INFO:root:  #632,056 count=  3,684,320,381
INFO:root:  #632,056 count=  3,684,320,381
INFO:root:  #632,056 count=  3,684,320,381
INFO:root:  #632,056 count=  3,684,320,381
INFO:root:  #632,056 count=  3,684,320,381
read fn
read fn
cat ../../m54050R1_180205_102507.subreads.fasta
cat ../../m54050R1_180205_102507.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
read fn
cat ../../m54050R1_180205_102507.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
read fn
fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
cat ../../m54050R1_180205_102507.subreads.fasta
fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
read fn
cat ../../m54050R1_180205_102507.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
fasta2DB: [WARNING] Could not delete temporary file m54050R1_180205_102507.subreads.fasta
fasta2DB: raw_reads.db is corrupted, read failed
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
falcon-kit 1.2.6
pypeflow 2.1.1
pypeflow 2.1.1
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            201
INFO:root:Counting bytes from "None"
INFO:root:        #2 count=            422
INFO:root:        #4 count=            884
INFO:root:        #8 count=          1,773
INFO:root:       #15 count=          3,383
INFO:root:        #1 count=            201
INFO:root:        #2 count=            422
INFO:root:        #4 count=            884
INFO:root:        #8 count=          1,773
INFO:root:       #15 count=          3,383
INFO:root:       #29 count=          6,612
INFO:root:       #57 count=         13,179
INFO:root:       #61 count=         37,191
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
pypeflow 2.1.1
INFO:root:       #29 count=          6,612
INFO:root:       #57 count=         13,179
INFO:root:       #61 count=         37,191
Adding 'm54050R1_180205_102507.subreads.fasta.fasta' ...
INFO:root:       #63 count=         76,347
INFO:root:       #71 count=        134,764
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            201
INFO:root:        #2 count=            422
INFO:root:        #4 count=            884
INFO:root:        #8 count=          1,773
INFO:root:       #15 count=          3,383
INFO:root:       #29 count=          6,612
INFO:root:       #57 count=         13,179
INFO:root:       #61 count=         37,191
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
INFO:root:      #425 count=        243,370
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
"main", fname, loader, pkg_name)
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 357, in
exec code in run_globals
main(sys.argv)          # pragma: no cover
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 354, in main
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 140, in write_streamed
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 357, in
main(sys.argv)          # pragma: no cover
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 354, in main
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 140, in write_streamed
fp_out.write(str(zrec.subread_record))
fp_out.write(str(zrec.subread_record))
IOError: [Errno 32] Broken pipe
IOError: [Errno 32] Broken pipe
INFO:root:      #442 count=        449,237
cat: write error: Broken pipe
cat: write error: Broken pipe
[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
[WARNING]Call 'bash -vex build_db.sh' returned 256.
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1443, in
main()
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1439, in main
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1443, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1439, in main
args.func(args)
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=            201
INFO:root:        #2 count=            422
INFO:root:        #4 count=            884
INFO:root:        #8 count=          1,773
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 985, in cmd_build
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 167, in build_db
io.syscall('bash -vex {}'.format(script_fn))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 985, in cmd_build
INFO:root:       #15 count=          3,383
INFO:root:       #29 count=          6,612
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)
INFO:root:       #57 count=         13,179
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 167, in build_db
INFO:root:       #61 count=         37,191
io.syscall('bash -vex {}'.format(script_fn))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 357, in
INFO:root:Counting bytes from "None"
main(sys.argv)          # pragma: no cover
INFO:root:        #1 count=            201
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 354, in main
INFO:root:        #2 count=            422
args.func(args)
INFO:root:        #4 count=            884
INFO:root:        #8 count=          1,773
INFO:root:      #657 count=        861,953
INFO:root:       #15 count=          3,383
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)
INFO:root:       #29 count=          6,612
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 140, in write_streamed
fp_out.write(str(zrec.subread_record))
IOError: [Errno 32] Broken pipe
INFO:root:       #57 count=         13,179
INFO:root:       #61 count=         37,191
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 357, in
main(sys.argv)          # pragma: no cover
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 354, in main
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 267, in cmd_run_streamed_median_filter
run_streamed_median_filter(fp_in, sys.stdout, whitelist_set)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 147, in run_streamed_median_filter
write_streamed(fp_out, yield_zmwtuple_func, zmw_filter_func)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/fasta_filter.py", line 140, in write_streamed
fp_out.write(str(zrec.subread_record))
IOError: [Errno 32] Broken pipe
cat: write error: Broken pipe
[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
raise Exception(msg)
Exception: Call 'bash -vex build_db.sh' returned 256.
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1443, in
INFO:root:      #890 count=      1,685,294
raise Exception(msg)
Exception: Call 'bash -vex build_db.sh' returned 256.
cat: write error: Broken pipe
[WARNING]Call 'bash -vex build_db.sh' returned 256.
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1443, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1439, in main
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 985, in cmd_build
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 167, in build_db
io.syscall('bash -vex {}'.format(script_fn))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call 'bash -vex build_db.sh' returned 256.
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1439, in main
args.func(args)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 985, in cmd_build
build_db(ours, args.input_fofn_fn, args.db_fn, args.length_cutoff_fn)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 167, in build_db
io.syscall('bash -vex {}'.format(script_fn))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call 'bash -vex build_db.sh' returned 256.
2019-03-14 17:52:53,047 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2019-03-14 17:52:53,047 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:52:53,047 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2019-03-14 17:52:53,047 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:52:53,048 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:52:53,048 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:52:53,048 - root - CRITICAL - Error in /pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
2019-03-14 17:52:53,049 - root - CRITICAL - Error in /pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
"main", fname, loader, pkg_name)
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in
main()
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in
main()
run(**vars(parsed_args))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
run_cfg_in_tmpdir(cfg, tmpdir, '.')
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
run_bash(bash_template, myinputs, myoutputs, parameters)
run(**vars(parsed_args))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
util.system(cmd)
run_cfg_in_tmpdir(cfg, tmpdir, '.')
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
run_bash(bash_template, myinputs, myoutputs, parameters)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
util.system(cmd)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
2019-03-14 17:52:53,055 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2019-03-14 17:52:53,056 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:52:53,056 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:52:53,056 - root - CRITICAL - Error in /pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
run(**vars(parsed_args))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
run_cfg_in_tmpdir(cfg, tmpdir, '.')
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
run_bash(bash_template, myinputs, myoutputs, parameters)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
util.system(cmd)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
2019-03-14 17:52:53,065 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2019-03-14 17:52:53,066 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:52:53,066 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 17:52:53,067 - root - CRITICAL - Error in /pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/pylon5/mc5fqip/orca21/0-rawreads/build/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in
main()
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
run(**vars(parsed_args))
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
run_cfg_in_tmpdir(cfg, tmpdir, '.')
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
run_bash(bash_template, myinputs, myoutputs, parameters)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
util.system(cmd)
File "/pylon5/mc5fqip/orca21/anaconda3/envs/myenv/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
raise Exception(msg)
+++ pwd
++ echo 'FAILURE. Running top in /pylon5/mc5fqip/orca21/0-rawreads/build (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
Exception: Call '/bin/bash user_script.sh' returned 256.
+++ pwd
++ echo 'FAILURE. Running top in /pylon5/mc5fqip/orca21/0-rawreads/build (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
+++ pwd
++ echo 'FAILURE. Running top in /pylon5/mc5fqip/orca21/0-rawreads/build (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
+++ pwd
++ echo 'FAILURE. Running top in /pylon5/mc5fqip/orca21/0-rawreads/build (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
INFO:root:    #1,937 count=      3,336,834
++ which python
++ which python
++ which python
++ which python
++ which top
++ which top
++ which top
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
INFO:root:    #2,595 count=      6,653,207
++ pstree -apl
++ pstree -apl
++ pstree -apl
++ pstree -apl

real	4m44.057s
user	4m5.866s
sys	0m27.214s
real	4m44.060s
user	4m5.665s
sys	0m27.575s

finish
echo 'finish code: 1'
finish
echo 'finish code: 1'
INFO:root:    #3,605 count=     13,250,960

real	4m44.080s
user	4m6.569s
sys	0m26.420s

finish
echo 'finish code: 1'

real	4m44.091s
user	4m7.281s
sys	0m26.221s

finish
echo 'finish code: 1'
INFO:root:    #5,067 count=     26,438,315
INFO:root:    #9,631 count=     52,808,553
INFO:root:   #17,876 count=    105,501,455
INFO:root:   #33,517 count=    210,894,049
INFO:root:   #67,557 count=    421,660,075
INFO:root:  #138,208 count=    843,187,868
INFO:root:  #285,310 count=  1,686,257,172
INFO:root:  #586,353 count=  3,372,369,292
read fn
python -m falcon_kit.mains.fasta_filter streamed-median -
cat ../../m54050R1_180205_203502.subreads.fasta
fasta2DB -v raw_reads -im54050R1_180205_203502.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          7,675
INFO:root:        #2 count=         16,492
Adding 'm54050R1_180205_203502.subreads.fasta.fasta' ...
INFO:root:        #5 count=         58,348
INFO:root:        #9 count=        104,030
INFO:root:       #16 count=        175,720
INFO:root:       #37 count=        316,861
INFO:root:       #61 count=        597,735
INFO:root:      #238 count=      1,149,302
INFO:root:      #441 count=      2,258,656
INFO:root:      #675 count=      4,466,062
INFO:root:    #1,626 count=      8,875,130
INFO:root:    #2,915 count=     17,697,362
INFO:root:    #5,228 count=     35,342,884
INFO:root:   #10,157 count=     70,608,036
INFO:root:   #21,921 count=    141,133,151
INFO:root:   #45,516 count=    282,187,689
INFO:root:   #94,223 count=    564,291,704
INFO:root:  #192,243 count=  1,128,491,266
INFO:root:  #403,896 count=  2,256,894,302
INFO:root:  #842,783 count=  4,513,697,570
read fn
cat ../../m54050R1_180206_064434.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180206_064434.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         22,969
Adding 'm54050R1_180206_064434.subreads.fasta.fasta' ...
INFO:root:       #12 count=         69,965
INFO:root:       #16 count=        136,750
INFO:root:       #25 count=        236,704
INFO:root:      #135 count=        436,572
INFO:root:      #318 count=        838,777
INFO:root:      #408 count=      1,607,103
INFO:root:      #997 count=      3,082,830
INFO:root:    #1,569 count=      6,028,296
INFO:root:    #2,711 count=     11,909,437
INFO:root:    #4,608 count=     23,679,111
INFO:root:    #8,977 count=     47,222,976
INFO:root:   #16,294 count=     94,270,393
INFO:root:   #30,908 count=    188,364,603
INFO:root:   #63,905 count=    376,533,093
INFO:root:  #126,065 count=    752,870,381
INFO:root:  #266,009 count=  1,505,537,109
INFO:root:  #546,986 count=  3,010,835,262
read fn
cat ../../m54050R1_180207_203009.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180207_203009.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         15,528
Adding 'm54050R1_180207_203009.subreads.fasta.fasta' ...
INFO:root:       #15 count=         39,397
INFO:root:       #18 count=         71,078
INFO:root:       #30 count=        134,398
INFO:root:      #130 count=        258,831
INFO:root:      #317 count=        522,529
INFO:root:      #515 count=      1,031,741
INFO:root:    #1,222 count=      2,029,291
INFO:root:    #1,526 count=      4,024,089
INFO:root:    #2,969 count=      8,002,205
INFO:root:    #4,995 count=     15,955,492
INFO:root:    #8,842 count=     31,858,883
INFO:root:   #15,711 count=     63,661,809
INFO:root:   #29,486 count=    127,268,481
INFO:root:   #56,761 count=    254,481,249
INFO:root:  #113,227 count=    508,898,324
INFO:root:  #223,394 count=  1,017,720,119
INFO:root:  #453,921 count=  2,035,373,609
INFO:root:  #921,516 count=  4,070,664,628
read fn
cat ../../m54050R1_180208_063535.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180208_063535.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          8,525
INFO:root:        #2 count=         23,659
Adding 'm54050R1_180208_063535.subreads.fasta.fasta' ...
INFO:root:       #23 count=         41,280
INFO:root:       #80 count=         75,598
INFO:root:      #154 count=        151,110
INFO:root:      #176 count=        303,855
INFO:root:      #212 count=        583,374
INFO:root:      #291 count=      1,139,204
INFO:root:      #795 count=      2,263,452
INFO:root:    #1,724 count=      4,450,316
INFO:root:    #3,024 count=      8,838,183
INFO:root:    #5,973 count=     17,581,829
INFO:root:   #10,012 count=     35,045,103
INFO:root:   #17,359 count=     69,972,353
INFO:root:   #32,883 count=    139,816,348
INFO:root:   #60,679 count=    279,500,311
INFO:root:  #122,165 count=    558,856,717
INFO:root:  #244,681 count=  1,117,552,360
INFO:root:  #488,493 count=  2,234,942,780
INFO:root:  #994,012 count=  4,469,739,199
read fn
cat ../../m54050R1_180208_164509.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180208_164509.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          4,339
Adding 'm54050R1_180208_164509.subreads.fasta.fasta' ...
INFO:root:        #3 count=         15,853
INFO:root:        #4 count=         27,413
INFO:root:        #6 count=         46,287
INFO:root:       #16 count=         91,212
INFO:root:       #28 count=        166,573
INFO:root:       #62 count=        316,189
INFO:root:      #152 count=        599,674
INFO:root:      #384 count=      1,157,061
INFO:root:      #553 count=      2,271,138
INFO:root:    #1,620 count=      4,505,203
INFO:root:    #2,788 count=      8,949,439
INFO:root:    #5,146 count=     17,836,899
INFO:root:    #9,403 count=     35,630,402
INFO:root:   #17,575 count=     71,196,569
INFO:root:   #31,238 count=    142,298,020
INFO:root:   #59,125 count=    284,504,058
INFO:root:  #116,757 count=    568,868,162
INFO:root:  #237,793 count=  1,137,592,686
INFO:root:  #476,839 count=  2,275,040,142
INFO:root:  #985,540 count=  4,549,978,182
read fn
cat ../../m54050R1_180209_025449.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180209_025449.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          1,124
INFO:root:        #2 count=         11,891
INFO:root:        #3 count=         25,208
Adding 'm54050R1_180209_025449.subreads.fasta.fasta' ...
INFO:root:        #6 count=         61,981
INFO:root:       #21 count=        109,935
INFO:root:       #41 count=        202,473
INFO:root:       #59 count=        407,049
INFO:root:       #97 count=        755,318
INFO:root:      #275 count=      1,457,424
INFO:root:      #766 count=      2,837,474
INFO:root:    #1,194 count=      5,640,860
INFO:root:    #3,117 count=     11,156,713
INFO:root:    #6,039 count=     22,182,616
INFO:root:   #11,133 count=     44,233,551
INFO:root:   #20,876 count=     88,367,082
INFO:root:   #45,433 count=    176,571,256
INFO:root:   #93,158 count=    352,978,876
INFO:root:  #197,343 count=    705,792,445
INFO:root:  #408,824 count=  1,411,418,611
INFO:root:  #829,918 count=  2,822,672,658
read fn
cat ../../m54050R1_180216_131107.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180216_131107.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          9,719
Adding 'm54050R1_180216_131107.subreads.fasta.fasta' ...
INFO:root:        #4 count=         40,017
INFO:root:        #7 count=         75,435
INFO:root:       #12 count=        141,540
INFO:root:       #28 count=        268,737
INFO:root:       #56 count=        481,809
INFO:root:       #90 count=        894,045
INFO:root:      #153 count=      1,692,205
INFO:root:      #308 count=      3,287,634
INFO:root:      #632 count=      6,477,028
INFO:root:    #1,213 count=     12,864,006
INFO:root:    #2,414 count=     25,595,113
INFO:root:    #4,836 count=     51,071,350
INFO:root:    #9,624 count=    102,031,148
INFO:root:   #19,322 count=    203,873,360
INFO:root:   #39,174 count=    407,562,032
INFO:root:   #79,813 count=    814,919,766
INFO:root:  #161,884 count=  1,629,640,257
INFO:root:  #328,680 count=  3,259,065,121
read fn
cat ../../m54050R1_180218_162755.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180218_162755.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          1,529
INFO:root:        #2 count=          4,406
INFO:root:        #3 count=          8,366
Adding 'm54050R1_180218_162755.subreads.fasta.fasta' ...
INFO:root:        #4 count=         16,652
INFO:root:        #6 count=         30,450
INFO:root:        #9 count=         61,269
INFO:root:       #16 count=        119,461
INFO:root:       #33 count=        229,903
INFO:root:       #58 count=        436,578
INFO:root:       #99 count=        837,979
INFO:root:      #180 count=      1,628,014
INFO:root:      #348 count=      3,199,054
INFO:root:      #724 count=      6,331,018
INFO:root:    #1,416 count=     12,598,488
INFO:root:    #2,818 count=     25,145,555
INFO:root:    #5,536 count=     50,203,980
INFO:root:   #11,332 count=    100,325,476
INFO:root:   #22,391 count=    200,535,278
INFO:root:   #46,441 count=    400,954,212
INFO:root:   #95,439 count=    801,779,282
INFO:root:  #193,205 count=  1,603,425,002
INFO:root:  #390,184 count=  3,206,699,992
read fn
cat ../../m54050R1_180219_023725.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180219_023725.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          3,126
INFO:root:        #2 count=         27,682
Adding 'm54050R1_180219_023725.subreads.fasta.fasta' ...
INFO:root:        #5 count=         55,708
INFO:root:        #9 count=        112,213
INFO:root:       #24 count=        221,574
INFO:root:       #48 count=        420,083
INFO:root:       #84 count=        817,466
INFO:root:      #165 count=      1,606,553
INFO:root:      #354 count=      3,182,603
INFO:root:      #684 count=      6,332,043
INFO:root:    #1,535 count=     12,632,150
INFO:root:    #2,846 count=     25,218,301
INFO:root:    #5,568 count=     50,373,610
INFO:root:   #11,334 count=    100,672,557
INFO:root:   #22,836 count=    201,261,550
INFO:root:   #47,372 count=    402,430,842
INFO:root:   #99,810 count=    804,756,672
INFO:root:  #205,037 count=  1,609,413,862
INFO:root:  #411,538 count=  3,218,716,440
read fn
cat ../../m54050R1_180219_124713.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180219_124713.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          2,982
INFO:root:        #2 count=          7,310
INFO:root:        #4 count=         16,448
INFO:root:        #7 count=         29,699
INFO:root:        #9 count=         54,065
Adding 'm54050R1_180219_124713.subreads.fasta.fasta' ...
INFO:root:       #15 count=        116,990
INFO:root:       #29 count=        214,682
INFO:root:       #53 count=        410,232
INFO:root:       #93 count=        792,899
INFO:root:      #185 count=      1,559,939
INFO:root:      #351 count=      3,119,435
INFO:root:      #665 count=      6,179,980
INFO:root:    #1,305 count=     12,292,495
INFO:root:    #2,636 count=     24,517,180
INFO:root:    #5,338 count=     48,970,895
INFO:root:   #10,661 count=     97,846,420
INFO:root:   #21,234 count=    195,569,778
INFO:root:   #43,266 count=    391,008,225
INFO:root:   #88,665 count=    781,871,063
INFO:root:  #180,690 count=  1,563,591,507
INFO:root:  #367,307 count=  3,127,029,280
read fn
cat ../../m54050R1_180219_225651.subreads.fasta
python -m falcon_kit.mains.fasta_filter streamed-median -
fasta2DB -v raw_reads -im54050R1_180219_225651.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=         17,456
Adding 'm54050R1_180219_225651.subreads.fasta.fasta' ...
INFO:root:        #4 count=         43,590
INFO:root:        #6 count=         81,470
INFO:root:       #16 count=        171,029
INFO:root:       #33 count=        311,752
INFO:root:       #64 count=        597,228
INFO:root:      #133 count=      1,155,884
INFO:root:      #234 count=      2,278,516
INFO:root:      #498 count=      4,516,541
INFO:root:    #1,099 count=      8,996,162
INFO:root:    #2,156 count=     17,939,634
INFO:root:    #4,225 count=     35,819,795
INFO:root:    #8,501 count=     71,596,984
INFO:root:   #17,182 count=    143,097,510
INFO:root:   #34,653 count=    286,099,824
INFO:root:   #70,918 count=    572,102,016
INFO:root:  #146,805 count=  1,144,100,517
INFO:root:  #298,705 count=  2,288,097,270
read fn
python -m falcon_kit.mains.fasta_filter streamed-median -
cat ../../m54050R1_180220_090637.subreads.fasta
fasta2DB -v raw_reads -im54050R1_180220_090637.subreads.fasta
falcon-kit 1.2.6
pypeflow 2.1.1
INFO:root:Counting bytes from "None"
INFO:root:        #1 count=          3,285
INFO:root:        #3 count=         10,081
Adding 'm54050R1_180220_090637.subreads.fasta.fasta' ...
INFO:root:        #4 count=         17,516
INFO:root:        #9 count=         38,436
INFO:root:       #12 count=         65,282
INFO:root:       #16 count=        138,526
INFO:root:       #25 count=        260,077
INFO:root:       #50 count=        480,285
INFO:root:       #99 count=        920,601
INFO:root:      #199 count=      1,768,106
INFO:root:      #393 count=      3,463,559
INFO:root:      #745 count=      6,833,303
INFO:root:    #1,462 count=     13,590,531
INFO:root:    #3,038 count=     27,052,184
INFO:root:    #6,172 count=     53,985,227
INFO:root:   #12,712 count=    107,818,773
INFO:root:   #25,764 count=    215,485,635
INFO:root:   #53,168 count=    430,779,641
INFO:root:  #108,258 count=    861,357,912
INFO:root:  #216,171 count=  1,722,508,762
read fn
#cat fc.fofn | xargs rm -f
DBdust  raw_reads
DBdust raw_reads

DBsplit -f -a -x500 -s200 raw_reads

DBsplit -f -a -x500 -s200 raw_reads
#LB=$(cat raw_reads.db | LD_LIBRARY_PATH= awk '$1 == "blocks" {print $3}')
#echo -n $LB >| db_block_count

CUTOFF=$(python2.7 -m falcon_kit.mains.calc_cutoff --coverage 30.0 1100000000 <(DBstats -b1 raw_reads))
++ python2.7 -m falcon_kit.mains.calc_cutoff --coverage 30.0 1100000000 /dev/fd/63
+++ DBstats -b1 raw_reads
falcon-kit 1.2.6
pypeflow 2.1.1

CUTOFF=13068
echo -n $CUTOFF >| length_cutoff
echo -n 13068

TODO: Verify that db exists.
#ln -sf length_cutoff length_cutoff
date

date
2019-03-14 19:11:47,694 - root - DEBUG - Call '/bin/bash user_script.sh' returned 0.
2019-03-14 19:11:47,694 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'
2019-03-14 19:11:47,695 - root - DEBUG - Checking existence of u'length_cutoff' with timeout=30
2019-03-14 19:11:47,739 - root - DEBUG - Checking existence of u'raw_reads.db' with timeout=30
2019-03-14 19:11:47,739 - root - WARNING - CD: '/pylon5/mc5fqip/orca21/0-rawreads/build' -> '/pylon5/mc5fqip/orca21/0-rawreads/build'

real	83m38.191s
user	92m3.326s
sys	6m48.399s
touch /pylon5/mc5fqip/orca21/0-rawreads/build/run.sh.done

touch /pylon5/mc5fqip/orca21/0-rawreads/build/run.sh.done
finish
echo 'finish code: 0'

Did you cross-post this? It looks a lot like https://github.com/PacificBiosciences/FALCON/issues/588
If you're going to use GitHub, you need to learn to use GitHub. You can "attach" a file, or you can post it somewhere (e.g. gist.github.com), or you can post it here with markdown formatting if it's small. Other users are able to figure this out. You're just dumping a lot of text.
You need to debug this COMPLETELY OUTSIDE THE WORKFLOW. (See my response to the cross-post.) The bug appears to be specific to your hardware. You should be able to run any task yourself, independent of the workflow, and to learn how much memory it consumes.

We interact with GitHub so we can learn about small bugs directly from users, but this is a free service. Management does not give us much bandwidth to respond to GitHub tickets. Please do everything you can to reduce the time we need to spend. If you prove that we have a bug, we will try to fix it quickly.
If you are running Sequel and using official PB "smrtlink" software, you can get more support through your PacBio representatives.

Hi Chris,
The errors seemed different to me and I’m new to bioinformatics. My
apologies, it won’t happen again. I’ll keep these tips in mind.
Nicolas
Thank you for the info this is helpful.
On Tue, Mar 26, 2019 at 11:53 AM Christopher Dunn notifications@github.com
wrote:

Did you cross-post this? It looks a lot like
PacificBiosciences/FALCON#588
https://github.com/PacificBiosciences/FALCON/issues/588
If you're going to use GitHub, you need to learn to use GitHub. You
can "attach" a file, or you can post it somewhere (e.g. gist.github.com),
or you can post it here with markdown formatting if it's small.
Other users are able to figure this out. You're just dumping a lot of text.
You need to debug this COMPLETELY OUTSIDE THE WORKFLOW. (See my
response to the cross-post.) The bug appears to be specific to your
hardware. You should be able to run any task yourself, independent of the
workflow, and to learn how much memory it consumes.

We interact with GitHub so we can learn about small bugs directly from
users, but this is a free service. Management does not give us much
bandwidth to respond to GitHub tickets. Please do everything you can to
reduce the time we need to spend. If you prove that we have a bug, we will
try to fix it quickly.
If you are running Sequel and using official PB "smrtlink" software, you
can get more support through your PacBio representatives.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/PacificBiosciences/pbbioconda/issues/109#issuecomment-476797629,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AUPt78obhTkmETmUFsfXkDuo-9WN1LLzks5vamyhgaJpZM4buAy_
.

--
Best,
Nicolas Alexandre
PhD Candidate, Integrative Biology
Whiteman Lab
University of California - Berkeley
nalexandre@berkeley.edu kiv@berkeley.edu

Hi nicolasalexandre21,
Have you solved your problems?
I got similar issues with slurm, but sloved by addiing the options : --ntasks 1 --exclusive in srun command. I don't know why, but it works (I learned this when playing with canu). It seems that without --ntasks 1 , thesrun will repeat the job many times simultaneously which causes chaos to the processing.
Let me know whether this helps.

Thanks. I've updated the pypeflow wiki with that helpful advice.

Hi Xiaoyezao,
Thank you for the helpful advice!! It seems to have mitigated the number of
error messages significantly. I am still getting a couple minor internal
allocation errors, but I will give you an update once I figure out the
problem in case it is helpful to anybody.
Nicolas
On Fri, Mar 29, 2019 at 7:05 AM Christopher Dunn notifications@github.com
wrote:

Thanks. I've updated the pypeflow wiki with that helpful advice.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/PacificBiosciences/pbbioconda/issues/109#issuecomment-478009582,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AUPt74L3XLvFiGhHQ2g0rRUL3owBYVIPks5vbh2dgaJpZM4buAy_
.

--
Best,
Nicolas Alexandre
PhD Candidate, Integrative Biology
Whiteman Lab
University of California - Berkeley
nalexandre@berkeley.edu kiv@berkeley.edu

Hi Chris and Xiaoyezao,
It is finally running. We were able to get it to run by adding  ${CMD} in
the srun command since falcon constructs values for ${CMD} that are then
launched by srun.
Thank you both for your help,
Nicolas
On Fri, Mar 29, 2019 at 9:20 AM Nicolas Alexandre nalexandre@berkeley.edu
wrote:

Hi Xiaoyezao,
Thank you for the helpful advice!! It seems to have mitigated the number
of error messages significantly. I am still getting a couple minor internal
allocation errors, but I will give you an update once I figure out the
problem in case it is helpful to anybody.
Nicolas
On Fri, Mar 29, 2019 at 7:05 AM Christopher Dunn notifications@github.com
wrote:

Thanks. I've updated the pypeflow wiki with that helpful advice.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/PacificBiosciences/pbbioconda/issues/109#issuecomment-478009582,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AUPt74L3XLvFiGhHQ2g0rRUL3owBYVIPks5vbh2dgaJpZM4buAy_
.

--
Best,
Nicolas Alexandre
PhD Candidate, Integrative Biology
Whiteman Lab
University of California - Berkeley
nalexandre@berkeley.edu kiv@berkeley.edu

--
Best,
Nicolas Alexandre
PhD Candidate, Integrative Biology
Whiteman Lab
University of California - Berkeley
nalexandre@berkeley.edu kiv@berkeley.edu

I should have mentioned: Always start with a tiny testcase. Integration problems are the worst!
Glad you got it working.

authors: nicolasalexandre21 pb-cdunn xiaoyezao

repo: PacificBiosciences/pbbioconda