text:
Problems with logs and joint VCF file generation in latest dev build

Hello,
After upgrading to the latest development version, the logs and joint VCF file generation don't seem to work properly anymore. Debug messages don't get printed anymore (neither on stdout, nor in the log file), and the bcbio-nextgen-debug.log file is pretty much identical with bcbio-nextgen.log. The only difference is the resource requests messages which appear in the debug log:
[2018-08-02T10:10Z] Resource requests: bwa, sambamba, samtools; memory: 3.00, 3.00, 3.00; cores: 16, 16, 16
[2018-08-02T10:10Z] Configuring 2 jobs to run, using 16 cores each with 48.1g of memory reserved for each job

[2018-08-02T10:10Z] Resource requests: gatk, gatk-haplotype, picard; memory: 3.50, 3.00, 3.00; cores: 1, 16, 16
[2018-08-02T10:10Z] Configuring 32 jobs to run, using 1 cores each with 3.50g of memory reserved for each job

[2018-08-02T10:18Z] Resource requests: bcbio_variation, fastqc, gatk, gatk-vqsr, gemini, kraken, preseq, qsignature, sambamba, samtools; memory: 3.00, 3.00, 3.50, 3.00, 3.00, 3.00, 3.00, 3.00, 3.00, 3.00; cores: 16, 16, 1, 16, 16, 16, 16, 16, 16, 16
[2018-08-02T10:18Z] Configuring 2 jobs to run, using 16 cores each with 56.1g of memory reserved for each job

The multi-sample <batch>-gatk-haplotype-joint-annotated.vcf.gz did not get generated, even though the sample-specific VCF files are where they should be.
Furthermore, bcbio-nextgen-commands.log is completely empty.
To test all of this, I've run a simple variant calling job that worked flawlessly a few days ago, before upgrading Bcbio-nextgen.

Thanks for the report. This is definitely unexpected as we haven't made any changes to logging or anything otherwise extensive that would make everything stop working. What version were you upgrading from? Are you running this in a clean directory or in a pre-existing work directory? Is it possible it's just skipping all the work because the files already exist?

It's a fresh Bcbio install (one day old), upgraded to development. I've had the problem with the log files with a couple of analyses, and they were both done from scratch, in empty or newly-created directories.
This is how I did the installation:
wget https://raw.github.com/chapmanb/bcbio-nextgen/master/scripts/bcbio_nextgen_install.py
python bcbio_nextgen_install.py <bcbio_dir> --tooldir=<bcbio_dir>/tools \
  --genomes sacCer3 --genomes phix --datatarget variation --datatarget rnaseq \
  --datatarget smallrna --datatarget vep --datatarget kraken \
  --aligners bwa --aligners bowtie2 --isolate --cores 32

export PATH=<bcbio_dir>/tools/bin:<bcbio_dir>/anaconda/bin:$PATH

bcbio_nextgen.py upgrade -u development

I tried upgrading to the latest development version again today but it didn't make any difference.

Thanks for following up. I am stumped and now sure what to suggest. What version did you have previously that worked well on your system? Do you get no logging under specific run environments: does it work when running on a single machine and only fail to log on a distributed setup? I'm not exactly sure how best to diagnose this remotely but with more details can keep trying to guess.

The latest version that worked for me was mentioned in this post.
Unfortunately, I don't have that available anymore, because I decided to do a fresh install (after a recent fix) and I deleted all the old installs I had lying around.
I'll make sure to properly test any new install before deleting older (functioning) ones in the future.

Apologies, I'm not sure what to make of this. That looks like a pretty small update distance between two recent development packages. We haven't overhauled any of the logging in that time so I'm not sure why it disappeared on your runs. A couple of questions to help debug:

How are you running bcbio? Single machine? On a cluster?
Did the run environment change between the working and problem runs?
Could you provide the configuration input you're using?

Apologies, but I don't know why you'd lose both logging and joint calling so hoping for some baseline or ideas to reproduce what you're seeing. Thanks again for helping to debug.

I've reinstalled, this time without upgrading to development.
bcbio_conda list | grep bcbio-nextgen
bcbio-nextgen             1.1.0                    py27_0    bioconda
This time I'm getting file system errors:
[2018-08-03T15:34Z] Timing: alignment
[2018-08-03T15:34Z] multiprocessing: process_alignment
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
Traceback (most recent call last):
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/tools/bin/bcbio_nextgen.py", line 242, in <module>
    main(**kwargs)
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/tools/bin/bcbio_nextgen.py", line 47, in main
    run_main(**kwargs)
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 45, in run_main
    fc_dir, run_info_yaml)
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 89, in _run_toplevel
    for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py", line 131, in variant2pipeline
    samples = run_parallel("process_alignment", samples)
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 28, in run_parallel
    return run_multicore(fn, items, config, parallel=parallel)
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py", line 86, in run_multicore
    for data in joblib.Parallel(parallel["num_jobs"], batch_size=1)(joblib.delayed(fn)(x) for x in items):
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 994, in __call__
    self.retrieve()
  File "/storage/brno3-cerit/home/amizeranschi/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py", line 926, in retrieve
    raise exception.unwrap(this_report)
joblib.my_exceptions.JoblibOSError: JoblibOSError

On this page, someone else mentioned getting a similar error and that it was caused by problems with their NFS storage. I'm installing Bcbio-nextgen on an NFS file system, so my issue might be related. No idea what could cause this, though, as I seem to be able to browse the NFS storage without issues and I doubt there was any NFS crash recently.

I'm installing Bcbio on an NFS storage from a VM in an OpenNebula cloud, running Debian Stretch. The working environment did, indeed, change between the working and non-working runs. I was running Ubuntu 14.04 before.
I'm using the following template:
details:
  - analysis: variant2
    genome_build: sacCer3
    algorithm:
      aligner: bwa
      mark_duplicates: true
      recalibrate: false
      realign: false
      effects: false
      tools_off: gemini
      variantcaller: gatk-haplotype
      jointcaller: gatk-haplotype-joint
      #svcaller: [manta]
      ploidy: 1
      variant_regions: ../config/variant_regions.bed

Hi @lpantano
Bcbio is installed on an NFS storage and I'm testing it on a single machine running Debian 9.
I'm setting up samples like this:
bcbio_prepare_samples.py --out ${bcbio_exp_name}-merged --csv ${bcbio_exp_name}.csv
bcbio_nextgen.py --separator R -w template ${bcbio_exp_name}-template.yaml ${bcbio_exp_name}-merged.csv ${bcbio_exp_path}/${bcbio_exp_name}-merged/*fastq*

and then running bcbio like this:
cd ${bcbio_exp_path}/${bcbio_exp_name}-merged/work/
bcbio_nextgen.py ../config/${bcbio_exp_name}-merged.yaml -n ${bcbio_cores}

No, I never managed to solve this issue.

Sorry about these continued problems. We're still not sure how to best debug and fix this issue and I posted some ideas in the discussion in #2520 around running the test suite to see if that also triggers problems. If you're able to follow along and debug, any information about how to replicate on a small sample would help us try and track this down. Thanks much for all the discussion and patience with the issue.

I'm not able to replicate the original problem anymore, because I don't have access to the cluster where I've experienced these issues.
I've installed the latest stable Bcbio version locally, on a Ubuntu machine and things are working fine, as far as I could see, with the latest stable version of Bcbio (1.1.0). The bcbio-nextgen-commands.log and bcbio-nextgen-debug.log are created just as they used to.
However, after upgrading to the most recent development version, deleting the old analysis files/directories and rerunning, the joint VCF file does not get created in the final directory, although it did with the previous (stable) version. I can't tell if the problem is with the creation of the joint VCF file, or if it simply gets omitted during the stage-out process at the end. I am still using the same template as before.

Thanks for following up and glad to hear things are working as expected with the latest version in terms of logging. I am still perplexed as to what is going on here with logging.
For the joint VCF issue, everything looks fine with your template so I'm not sure why you wouldn't see an output VCF file. It's a bit tough to debug remotely from the information we have so far but I can share what it should look like with the bcbio test data. The output directory should have individual sample gVCFs and a joint called shared VCF:
upload/
├── 2018-10-31_project
│   ├── TestBatch1-gatk-haplotype-joint-annotated.vcf.gz
│   └── TestBatch1-gatk-haplotype-joint-annotated.vcf.gz.tbi
├── Test1
│   ├── Test1-gatk-haplotype.vcf.gz
│   ├── Test1-gatk-haplotype.vcf.gz.tbi
└── Test2
    ├── Test2-gatk-haplotype.vcf.gz
    ├── Test2-gatk-haplotype.vcf.gz.tbi

and in the work directory the joint calls should be in joint/gatk-haplotype-joint/your-batch-name/. If you're able to share what you see or error messages we can hopefully identify why you don't see outputs. Hope this helps and thanks again for the work debugging.

The work directory from my previous test seems to contain the joint VCF files:
ls -la /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged__devel_bcbio_01/work/joint/gatk-haplotype-joint/testingVC/
total 156
drwxrwxr-x 19 amizeranschi amizeranschi  4096 Oct 27 11:29 .
drwxrwxr-x  3 amizeranschi amizeranschi  4096 Oct 27 11:05 ..
drwxrwxr-x  6 amizeranschi amizeranschi  4096 Oct 27 11:06 chrI
drwxrwxr-x 14 amizeranschi amizeranschi  4096 Oct 27 11:07 chrII
drwxrwxr-x  8 amizeranschi amizeranschi  4096 Oct 27 11:08 chrIII
drwxrwxr-x 24 amizeranschi amizeranschi  4096 Oct 27 11:11 chrIV
drwxrwxr-x 10 amizeranschi amizeranschi  4096 Oct 27 11:12 chrIX
drwxrwxr-x 11 amizeranschi amizeranschi  4096 Oct 27 11:13 chrV
drwxrwxr-x  5 amizeranschi amizeranschi  4096 Oct 27 11:13 chrVI
drwxrwxr-x 16 amizeranschi amizeranschi  4096 Oct 27 11:15 chrVII
drwxrwxr-x 11 amizeranschi amizeranschi  4096 Oct 27 11:16 chrVIII
drwxrwxr-x 13 amizeranschi amizeranschi  4096 Oct 27 11:18 chrX
drwxrwxr-x 11 amizeranschi amizeranschi  4096 Oct 27 11:19 chrXI
drwxrwxr-x 18 amizeranschi amizeranschi  4096 Oct 27 11:21 chrXII
drwxrwxr-x 16 amizeranschi amizeranschi  4096 Oct 27 11:23 chrXIII
drwxrwxr-x 15 amizeranschi amizeranschi  4096 Oct 27 11:25 chrXIV
drwxrwxr-x 17 amizeranschi amizeranschi  4096 Oct 27 11:27 chrXV
drwxrwxr-x 17 amizeranschi amizeranschi  4096 Oct 27 11:29 chrXVI
drwxrwxr-x  2 amizeranschi amizeranschi  4096 Oct 27 11:05 inprep
-rw-rw-r--  1 amizeranschi amizeranschi  3466 Oct 27 11:29 testingVC-chrI_0_61663-fixheader.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   101 Oct 27 11:29 testingVC-chrI_0_61663-fixheader.vcf.gz.tbi
-rw-rw-r--  1 amizeranschi amizeranschi  4020 Oct 27 11:29 testingVC-joint-combined.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   777 Oct 27 11:29 testingVC-joint-combined.vcf.gz.tbi
-rw-rw-r--  1 amizeranschi amizeranschi  6359 Oct 27 11:29 testingVC-joint-files.list
-rw-rw-r--  1 amizeranschi amizeranschi  4015 Oct 27 11:29 testingVC-joint-indel-filterINDEL.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   260 Oct 27 11:29 testingVC-joint-indel-filterINDEL.vcf.gz.tbi
-rw-rw-r--  1 amizeranschi amizeranschi  3827 Oct 27 11:29 testingVC-joint-indel.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   260 Oct 27 11:29 testingVC-joint-indel.vcf.gz.tbi
-rw-rw-r--  1 amizeranschi amizeranschi  4984 Oct 27 11:29 testingVC-joint-snp-filterSNP.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   810 Oct 27 11:29 testingVC-joint-snp-filterSNP.vcf.gz.tbi
-rw-rw-r--  1 amizeranschi amizeranschi  4779 Oct 27 11:29 testingVC-joint-snp.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   809 Oct 27 11:29 testingVC-joint-snp.vcf.gz.tbi
-rw-rw-r--  1 amizeranschi amizeranschi 11021 Oct 27 11:29 testingVC-joint.vcf.gz
-rw-rw-r--  1 amizeranschi amizeranschi   933 Oct 27 11:29 testingVC-joint.vcf.gz.tbi

In the final directory, the sample-specific VCF files are present, but the main directory (2018-10-27_project) doesn't contain a joint VCF file. This only happens with Bcbio's development version.
After cloning the Bcbio-nexgen repo and executing ./run_tests.sh devel, the test seems to run fine.
========================================================================= 1 passed, 81 deselected in 110.28 seconds ==========================================================================

However, this doesn't appear to run joint variant calling. The main output directory (test_automated_output/upload/2018-10-31_develtest/) contains two VCF files: TestBatch1-precalled-annotated.vcf.gz and TestBatch1-gatk-haplotype-annotated.vcf.gz. And there is only one sample-specific directory called Test1 in the upload directory.

Thanks again for helping debug and providing all the additional information. It does look like things ran through joint calling successfully so I'm not sure why this would be missing from the final output directory. If you re-run bcbio_nextgen.py in the same work directory, does it get transferred? I'm wondering if there was some failure prior to transfer; it just copies these over at the end so if it died for any reason that would prevent that.
If you want to confirm with the tests, ./run_tests.sh joint will run the joint calling integration test. It would be great to know if that successfully replicates what I'm seeing above. That would help us at least confirm bcbio does or doesn't transfer in your environment on the same set of inputs. Thanks again.

./run_tests.sh joint runs fine, the joint VCF files for both HaplotypeCaller and Strelka2 are being copied to the upload directory at the end.
Would it be possible to mention the ./run_tests.sh joint command in the documentation and, possibly, in the header of the run_tests.sh script? It could help others who run into similar issues as mine.
I tried re-running my test in the same work directory and nothing changed. The joint VCF file did not get copied. I also tried running new separate tests using HaplotypeCaller and, respectively, Strelka2 (as I saw this used in the automated tests) and the same thing happened. I don't see any errors in the logs. It seems like the joint VCF file is just not being copied in the end stage, for some reason.
I can't figure out why this only happens during my test, while the "official" Bcbio automatic test runs fine, even with the latest development version. I am using a set of FASTQ files that I created some time ago by sampling 10000 reads from some larger files. I've been using them to test the pipeline for a while and never had any trouble until now.
Finally, I ran my test again with Strelka2 using the stable version of Bcbio and, sure enough, the testingVC-strelka2-joint-annotated.vcf.gz file has been created in the final upload directory, just as it did with the HaplotypeCaller tests, when using the stable Bcbio version. Is there anything that has changed between the last stable version and the development version that could cause this?

The issue persists with the newly-released stable version, as well as the latest development version, as of today.
Running single-threaded did not make a difference.

Sorry you're still having issues and thank you for continuing to test and help debug. If the tests are running cleanly for you, there must be something in your actual run configuration that's causing bcbio to go down a different path and lead to issues. I know you've posted your template above, but I don't see any substantial differences between that and the tests that run fine for you. Is it possible to share your final configuration that doesn't work? Perhaps there is something in the batching we're missing that we could spot by looking at this. Thank you for the help trying to reproduce the problem.

Sure. This is the full config file:
details:
- algorithm:
    aligner: bwa
    effects: false
    jointcaller: strelka2-joint
    mark_duplicates: true
    ploidy: 1
    realign: false
    recalibrate: false
    svcaller:
    - manta
    tools_off: gemini
    variant_regions: ../config/variant_regions.bed
    variantcaller: strelka2
  analysis: variant2
  description: sample1
  files:
  - /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/sample1_R1.fastq.gz
  - /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/sample1_R2.fastq.gz
  genome_build: sacCer3
  metadata:
    batch: testingVC
- algorithm:
    aligner: bwa
    effects: false
    jointcaller: strelka2-joint
    mark_duplicates: true
    ploidy: 1
    realign: false
    recalibrate: false
    svcaller:
    - manta
    tools_off: gemini
    variant_regions: ../config/variant_regions.bed
    variantcaller: strelka2
  analysis: variant2
  description: sample2
  files:
  - /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/sample2_R1.fastq.gz
  - /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/sample2_R2.fastq.gz
  genome_build: sacCer3
  metadata:
    batch: testingVC
fc_name: testingVC-merged
upload:
  dir: ../final

I've had an attempt at applying the logic from Issue #2520. I've installed Docker and bcbio_vm as per documentation and I've pulled several bcbio-vc images from quay.io.
Note: I had to edit the file bcbiovm/docker/install.py as described in this older issue, in order to get bcbio_vm.py to work with a Docker image name of the form image:tag.
My tests run fine with the Docker images bcbio-vc:1.1.0-v1.1.0 and bcbio-vc:1.1.1a-82fb2a0 and the joint VCF file is created in the final directory. However, this stops happening once using the image bcbio-vc:1.1.1a-026700f. I had a look on GitHub (82fb2a0..026700f) for anything that may cause the joint VCF file to not get created but I didn't find anything.

Thanks so much for all the debugging work and help figuring out this issue. I've been struggling to reproduce and understand this for a few days but all your input finally helped me realize the issue. With the changes to do more post-variant annotation, we were implicitly assuming the output joint called VCF would have vcfanno annotations and be uploaded through that path. However, for your genome it likely doesn't get annotated so gets left out of the list of files to upload.
I've added a check for this case that I hope will resolve the issue if you run the latest development version. Please let me know if not and we can investigate further. Thank you again for all your patience and working helping debug this issue.

Thank you, this fixed it alright.
If I may, I also have a couple of last questions about this. Is bcbio_vm still being maintained and will it be in the future? I remember reading somewhere that it's going to get phased out in the future, but I'm not sure whether that was referring to bcbio_vm or something else.
If it's still being maintained, would it be possible to include the following patch? https://github.com/bcbio/bcbio-nextgen-vm/issues/51
This would help quite a bit with reproducibility, allowing one to use any of the old Docker images from quay.io. Alternatively, is there any other (better) way to achieve this?
Thanks a lot for fixing the joint VCF generation bug and for all the great work with bcbio-nextgen and bcbio_vm!

Thank you for following up. I'm glad that fixed the issue for you and thanks again for all the patience as we identified the underlying problem.
For running with Docker containers, we now recommend using the common workflow language (CWL) interface with the Cromwell runner, documented here:
https://bcbio-nextgen.readthedocs.io/en/latest/contents/cwl.html
This still uses bcbio_vm.py for setting up the run environment and launching jobs, but the actually running takes place via external runners to get improved runtime support over what we can develop as part of bcbio.
To lock down to specific tags in your workflows for reproducibility, there is a --add-container-tag for bcbio_vm.py cwl that uses a specific tag from the quay.io releases rather than latest (https://quay.io/repository/bcbio/bcbio-vc?tab=tags). This gets reflected persistently in generated CWL so you have a fully reproducible run tied to a specific instance.
CWL is still a work in progress but improving and being used more often. Hope this works for your use.

I gave this a try but I'm stuck at running bcbio_vm.py cwl because it doesn't find the reference genome files:
bcbio_vm.py cwl   --add-container-tag 1.1.1a-82fb2a0   ../config/${bcbio_exp_name}-merged.yaml
[2018-11-13T18:04Z] INFO: Using input YAML configuration: ../config/testingVC-merged.yaml
[2018-11-13T18:04Z] INFO: Checking sample YAML configuration: ../config/testingVC-merged.yaml
Traceback (most recent call last):
  File "/home/amizeranschi/install/bcbio-vm//anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/main.py", line 11, in run
    world = run_info.organize(dirs, config, run_info_yaml, is_cwl=True, integrations=integrations)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/pipeline/run_info.py", line 77, in organize
    item = add_reference_resources(item, remote_retriever)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/pipeline/run_info.py", line 174, in add_reference_resources
    _check_ref_files(data["reference"], data)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/pipeline/run_info.py", line 231, in _check_ref_files
    raise ValueError(msg + "\n".join(problems) + "\n")
ValueError:
Problems with input reference file None
Did not find fasta reference file for genome sacCer3.
Check tool-data/*.loc files to ensure paths to reference data are correct.

I have previously saved the datadir (bcbio_vm.py --datadir=~/install/bcbio-vm saveconfig). Nevertheless, I tried adding the --datadir to the command (bcbio_vm.py --datadir=~/install/bcbio-vm   cwl --add-container-tag 1.1.1a-82fb2a0 ../config/${bcbio_exp_name}-merged.yaml), but this didn't help.
However, running without CWL works fine:
bcbio_vm.py --datadir=~/install/bcbio-vm run --image quay.io/bcbio/bcbio-vc:1.1.1a-82fb2a0 -n 4 ../config/${bcbio_exp_name}-merged.yaml

The *.loc files are in ~/install/bcbio-vm/galaxy/tool-data/, but the paths inside them seem to point to /usr/local/share, for some reason (e.g. /usr/local/share/bcbio-nextgen/genomes/Scerevisiae/sacCer3/bwa/sacCer3.fa). I'm not sure why this happened. I've installed bcbio_vm with the following command:
bcbio_vm.py --datadir=~/install/bcbio-vm install --data --image quay.io/bcbio/bcbio-vc:1.1.0-v1.1.0 --tools \
  --genomes sacCer3 --aligners bwa

How can I get the correct paths inside the .loc files (without having to edit them manually)? And why isn't this issue affecting bcbio_vm.py when running without CWL?
P.S. Should I open a separate ticket for these issues?

Sorry about the issues and confusion here. The bcbio_vm.py data install is going to be confusing as it works with data both inside and outside the container, which is what leads to the disconnect here. Practically, we've tried to avoid all this by no longer depending on the *.loc files and instead specifying the location of the reference directory in a bcbio_system.yaml file that you pass to bcbio_vm.py cwl:
https://bcbio-nextgen.readthedocs.io/en/latest/contents/cwl.html#generating-cwl-for-input-to-a-tool
This approach should be much cleaner and avoid the need for any custom install tweaks. You point it at the genome directory and it finds all the inputs from that. Once you have the file in place you pass it to the CWL generation command with bcbio_vm.py cwl --systemconfig bcbio_system.yaml (and all your other arguments).
Hope this helps get CWL generation working for you. Thanks for exploring this and all the patience.

Alright, thanks for following up on this. I edited the file bcbio_system.yaml and got that to work, but now bcbio_vm.py cwl gets stuck due to another config (genome_resources__rnaseq__gene_bed). I'm using the same YAML template as previously mentioned on this thread, so I'm not sure why it complains about an rnaseq config, unless this is somehow referring to the sv_regions config that is used for structural VC. However, I don't have this enabled in the template.
I tried adding it and providing a valid BED file, but this didn't make the error go away.  Here's the full output that I get:
bcbio_vm.py cwl --systemconfig ~/install/bcbio-vm/galaxy/bcbio_system.yaml --add-container-tag 1.1.1a-82fb2a0   ../config/${bcbio_exp_name}-merged.yaml
[2018-11-14T19:24Z] INFO: Using input YAML configuration: ../config/testingVC-merged.yaml
[2018-11-14T19:24Z] INFO: Checking sample YAML configuration: ../config/testingVC-merged.yaml
Traceback (most recent call last):
  File "/home/amizeranschi/install/bcbio-vm//anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/main.py", line 12, in run
    create.from_world(world, run_info_yaml, integrations=integrations, add_container_tag=args.add_container_tag)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 37, in from_world
    prep_cwl(samples, workflow_fn, out_dir, out_file, integrations, add_container_tag=add_container_tag)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 427, in prep_cwl
    for cur in workflow.generate(variables, steps, wfoutputs):
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/workflow.py", line 64, in generate
    inputs, parallel_ids, nested_inputs = _get_step_inputs(step, file_vs, std_vs, parallel_ids)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/workflow.py", line 170, in _get_step_inputs
    for orig_input in [_get_variable(x, file_vs) for x in _handle_special_inputs(step.inputs, file_vs)]:
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/workflow.py", line 289, in _get_variable
    raise ValueError("Did not find variable %s in \n%s" % (vid, pprint.pformat(variables)))
ValueError: Did not find variable genome_resources__rnaseq__gene_bed in
[{'id': 'config__algorithm__align_split_size', 'type': ['null', 'string']},
 {'id': 'files', 'type': {'items': 'File', 'type': 'array'}},
 {'id': 'config__algorithm__trim_reads',
  'type': ['string', 'null', 'boolean']},
 {'id': 'reference__fasta__base',
  'secondaryFiles': ['.fai', '^.dict'],
  'type': 'File'},
 {'id': 'config__algorithm__expression_caller', 'type': ['null', 'string']},
 {'id': 'config__algorithm__vcfanno',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'config__algorithm__svprioritize', 'type': ['null', 'string']},
 {'id': 'config__algorithm__variantcaller',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'config__algorithm__adapters',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'config__algorithm__svcaller',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'genome_resources__variation__editing', 'type': ['null', 'string']},
 {'id': 'config__algorithm__coverage_interval', 'type': ['null', 'string']},
 {'id': 'reference__genome_context',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'genome_resources__variation__gc_profile', 'type': ['null', 'string']},
 {'id': 'rgnames__lb', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__clinvar', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__esp', 'type': ['null', 'string']},
 {'id': 'rgnames__rg', 'type': 'string'},
 {'id': 'config__algorithm__realign', 'type': ['string', 'null', 'boolean']},
 {'id': 'metadata__batch', 'type': 'string'},
 {'id': 'rgnames__lane', 'type': 'string'},
 {'id': 'genome_resources__variation__lcr', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__1000g', 'type': ['null', 'string']},
 {'id': 'config__algorithm__min_allele_fraction', 'type': 'double'},
 {'id': 'config__algorithm__nomap_split_targets', 'type': 'long'},
 {'id': 'reference__versions', 'type': ['null', 'string']},
 {'id': 'reference__bwa__indexes',
  'secondaryFiles': ['^.ann', '^.pac', '^.sa', '^.bwt'],
  'type': 'File'},
 {'id': 'config__algorithm__ploidy', 'type': 'long'},
 {'id': 'vrn_file', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__train_hapmap',
  'type': ['null', 'string']},
 {'id': 'config__algorithm__jointcaller',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'config__algorithm__bam_clean', 'type': ['string', 'null', 'boolean']},
 {'id': 'config__algorithm__nomap_split_size', 'type': 'long'},
 {'id': 'config__algorithm__validate', 'type': ['null', 'string']},
 {'id': 'config__algorithm__validate_regions', 'type': ['null', 'string']},
 {'id': 'genome_resources__rnaseq__transcriptome_index__tophat',
  'type': 'string'},
 {'id': 'config__algorithm__aligner', 'type': 'string'},
 {'id': 'rgnames__pl', 'type': 'string'},
 {'id': 'genome_build', 'type': 'string'},
 {'id': 'rgnames__pu', 'type': 'string'},
 {'id': 'genome_resources__variation__germline_het_pon',
  'type': ['null', 'string']},
 {'id': 'genome_resources__variation__exac', 'type': ['null', 'string']},
 {'id': 'config__algorithm__recalibrate',
  'type': ['string', 'null', 'boolean']},
 {'id': 'metadata__phenotype', 'type': 'string'},
 {'id': 'config__algorithm__quality_format', 'type': 'string'},
 {'id': 'config__algorithm__tools_off',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'genome_resources__variation__dbsnp', 'type': ['null', 'string']},
 {'id': 'config__algorithm__mark_duplicates',
  'type': ['string', 'null', 'boolean']},
 {'id': 'reference__twobit', 'type': 'File'},
 {'id': 'genome_resources__variation__polyx', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__encode_blacklist',
  'type': ['null', 'string']},
 {'id': 'reference__viral', 'type': ['null', 'string']},
 {'id': 'genome_resources__version', 'type': 'long'},
 {'id': 'genome_resources__variation__cosmic', 'type': ['null', 'string']},
 {'id': 'config__algorithm__ensemble', 'type': ['null', 'string']},
 {'id': 'config__algorithm__qc', 'type': {'items': 'string', 'type': 'array'}},
 {'id': 'config__algorithm__sv_regions', 'type': 'File'},
 {'id': 'analysis', 'type': 'string'},
 {'id': 'prep_samples/rgnames__sample', 'type': 'string'},
 {'id': 'config__algorithm__tools_on',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'config__algorithm__effects', 'type': ['string', 'null', 'boolean']},
 {'id': 'prep_samples/config__algorithm__variant_regions',
  'type': ['File', 'null']},
 {'id': 'config__algorithm__svvalidate', 'type': ['null', 'string']},
 {'id': 'reference__snpeff__R64-1-1_86', 'type': 'File'},
 {'id': 'genome_resources__aliases__ensembl', 'type': 'string'},
 {'id': 'config__algorithm__exclude_regions',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'reference__rtg', 'type': 'File'},
 {'id': 'genome_resources__variation__train_indels',
  'type': ['null', 'string']},
 {'id': 'genome_resources__aliases__snpeff', 'type': 'string'},
 {'id': 'config__algorithm__archive', 'type': ['null', 'string']},
 {'id': 'alignment_to_rec/alignment_rec',
  'type': {'items': {'fields': [{'name': 'resources', 'type': 'string'},
                                {'name': 'description', 'type': 'string'},
                                {'name': 'config__algorithm__align_split_size',
                                 'type': ['null', 'string']},
                                {'name': 'files',
                                 'type': {'items': 'File',
                                          'type': 'array'}},
                                {'name': 'config__algorithm__trim_reads',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'reference__fasta__base',
                                 'secondaryFiles': ['.fai', '^.dict'],
                                 'type': 'File'},
                                {'name': 'config__algorithm__adapters',
                                 'type': ['null',
                                          {'items': 'null',
                                           'type': 'array'}]},
                                {'name': 'rgnames__lb',
                                 'type': ['null', 'string']},
                                {'name': 'rgnames__rg', 'type': 'string'},
                                {'name': 'rgnames__lane',
                                 'type': 'string'},
                                {'name': 'reference__bwa__indexes',
                                 'secondaryFiles': ['^.ann',
                                                    '^.pac',
                                                    '^.sa',
                                                    '^.bwt'],
                                 'type': 'File'},
                                {'name': 'config__algorithm__bam_clean',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'config__algorithm__aligner',
                                 'type': 'string'},
                                {'name': 'rgnames__pl', 'type': 'string'},
                                {'name': 'rgnames__pu', 'type': 'string'},
                                {'name': 'config__algorithm__mark_duplicates',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'analysis', 'type': 'string'},
                                {'name': 'rgnames__sample',
                                 'type': 'string'},
                                {'name': 'config__algorithm__variant_regions',
                                 'type': 'File'}],
                     'name': 'alignment_rec',
                     'type': 'record'},
           'type': 'array'}},
 {'id': 'alignment/align_bam',
  'secondaryFiles': ['.bai'],
  'type': ['File', 'null']},
 {'id': 'alignment/hla__fastq',
  'type': ['null', {'items': 'File', 'type': 'array'}]},
 {'id': 'alignment/work_bam_plus__disc',
  'secondaryFiles': ['.bai'],
  'type': ['File', 'null']},
 {'id': 'alignment/work_bam_plus__sr',
  'secondaryFiles': ['.bai'],
  'type': ['File', 'null']},
 {'id': 'prep_samples_to_rec/prep_samples_rec',
  'type': {'items': {'fields': [{'name': 'resources', 'type': 'string'},
                                {'name': 'description', 'type': 'string'},
                                {'name': 'reference__fasta__base',
                                 'secondaryFiles': ['.fai', '^.dict'],
                                 'type': 'File'},
                                {'name': 'rgnames__sample',
                                 'type': 'string'},
                                {'name': 'config__algorithm__variant_regions',
                                 'type': 'File'}],
                     'name': 'prep_samples_rec',
                     'type': 'record'},
           'type': 'array'}},
 {'id': 'prep_samples/config__algorithm__variant_regions_merged',
  'type': ['File', 'null']},
 {'id': 'prep_samples/config__algorithm__variant_regions_orig',
  'type': ['File', 'null']},
 {'id': 'prep_samples/config__algorithm__coverage', 'type': ['File', 'null']},
 {'id': 'prep_samples/config__algorithm__coverage_merged',
  'type': ['File', 'null']},
 {'id': 'prep_samples/config__algorithm__coverage_orig',
  'type': ['File', 'null']},
 {'id': 'prep_samples/config__algorithm__seq2c_bed_ready',
  'type': ['File', 'null']}]

Thanks for testing this and apologies about the additional problems. What we're running into here is that bcbio isn't finding transcript files for your sacCer3 genome install, so doesn't have the BED files derived from that which is uses for annotations. In CWL it will fail up front with these type of issues. Are you using the standard bcbio sacCer3 configuration:
https://github.com/bcbio/bcbio-nextgen/blob/master/config/genomes/sacCer3-resources.yaml
or something custom? Do you have the transcript files available in your install? I'd definitely like to get the yeast genome install fixed for you but haven't worked with it in a while so am not entirely sure what might be missing. If you can provide more details I'm happy to help dig into it.

Alright, thanks for looking into this. I'm using the standard configuration. I have installed bcbio-nextgen as such:
python bcbio_nextgen_install.py ${bcbio_path} --tooldir=${bcbio_path}/tools \
  --genomes ${bcbio_genome} --datatarget variation --datatarget rnaseq \
  --datatarget smallrna --datatarget vep --datatarget kraken \
  --aligners bwa --aligners bowtie2 --isolate --cores ${bcbio_cores}

And this is reflected in the file ${bcbio_path}/config/install-params.yaml:
aligners:
- bwa
- bowtie2
- rtg
- twobit
datatarget:
- variation
- rnaseq
- smallrna
- vep
- kraken
genomes:
- sacCer3
isolate: true
tooldir: /home/amizeranschi/bcbio-stuff/bcbio/tools
toolplus: []

However, there is no rnaseq directory in /home/amizeranschi/bcbio-stuff/bcbio/genomes/Scerevisiae/sacCer3/.
ls /home/amizeranschi/bcbio-stuff/bcbio/genomes/Scerevisiae/sacCer3/
bowtie2  bwa  rtg  seq  seq_prep  snpeff  vep

The sacCer3.2bit file is also missing from /home/amizeranschi/bcbio-stuff/bcbio/genomes/Scerevisiae/sacCer3/seq/, although it is mentioned as installable in ${bcbio_path}/config/install-params.yaml:
ls /home/amizeranschi/bcbio-stuff/bcbio/genomes/Scerevisiae/sacCer3/seq/
chromFa.tar.gz  sacCer3.dict  sacCer3.fa  sacCer3.fa.fai  sacCer3-resources.yaml

The contents of the sacCer3-resources.yaml is identical to the one that you've previously linked on GitHub:
version: 16

aliases:
  ensembl: saccharomyces_cerevisiae_vep_94_R64-1-1
  snpeff: R64-1-1.86
rnaseq:
  transcripts: ../rnaseq/ref-transcripts.gtf
  transcripts_mask: ../rnaseq/ref-transcripts-mask.gtf
  transcriptome_index:
    tophat: ../rnaseq/tophat/sacCer3_transcriptome.ver
  dexseq: ../rnaseq/ref-transcripts.dexseq.gff3
  refflat: ../rnaseq/ref-transcripts.refFlat
  rRNA_fa: ../rnaseq/rRNA.fa

The same problem happens with bcbio-vm. I've installed it this way:
bcbio_vm.py --datadir=~/install/bcbio-vm install --data --image quay.io/bcbio/bcbio-vc:1.1.0-v1.1.0 --tools \
  --genomes sacCer3 --aligners bwa

bcbio_vm.py --datadir=~/install/bcbio-vm saveconfig

There are no rnaseq files in /home/amizeranschi/install/bcbio-vm/genomes/Scerevisiae/sacCer3/:
ls /home/amizeranschi/install/bcbio-vm/genomes/Scerevisiae/sacCer3/
bwa  rtg  rtg--sacCer3.sdf-wf.tar.gz  seq  seq_prep  snpeff  snpeff--R64-1-1.86-wf.tar.gz  ucsc-wf.tar.gz  vep

I've installed the hg19 genome to see if the same issues persist, for any reason, and they didn't. The rnaseq directory was created and the hg19.2bit file was created as well.
It looks like whatever is happening is specific to the sacCer3 genome, although I haven't modified the sacCer3-resources.yaml file.

Thanks for checking on this and apologies about sending you down the wrong path. I hadn't realized the sacCer3 genome wasn't up to date with transcripts. I just pushed fixes for this so if you want to do a clean install of the genome it should now have an rnaseq directory and work cleanly:
rm -rf /path/to/bcbio/genomes/Scerevisiae/sacCer3
bcbio_nextgen.py upgrade --data --cwl --genomes sacCer3

We no longer use the 2bit files since GATK has moved away from requiring those, but it should have everything else you need to run the CWL. Please let me know if you run into any issues with the install or run after this. Thanks again for the patience debugging.

Thanks once more for your help. The rnaseq directory has now been created.
One question: is there a way to install the stand-alone bcbio_vm and genome data without having to download the latest Docker image (as it would just waste space)? I have the image already downloaded (quay.io/bcbio/bcbio-vc:1.1.1a-82fb2a0). However, when I try the following:
bcbio_vm.py --datadir=~/install/bcbio-vm install --data --tools --image quay.io/bcbio/bcbio-vc:1.1.1a-82fb2a0 \
  --genomes sacCer3 --aligners bwa

It says it can't find the image locally, as described here: https://github.com/bcbio/bcbio-nextgen-vm/issues/51.
Regarding the previous issues with using CWL, bcbiovm.py now complains about a missing .2bit file:
bcbio_vm.py cwl --systemconfig ~/install/bcbio-vm/galaxy/bcbio_system.yaml --add-container-tag 1.1.1a-82fb2a0   ../config/${bcbio_exp_name}-merged.yaml
[2018-11-20T17:32Z] INFO: Using input YAML configuration: ../config/testingVC-merged.yaml
[2018-11-20T17:32Z] INFO: Checking sample YAML configuration: ../config/testingVC-merged.yaml
Traceback (most recent call last):
  File "/home/amizeranschi/install/bcbio-vm/anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/main.py", line 12, in run
    create.from_world(world, run_info_yaml, integrations=integrations, add_container_tag=args.add_container_tag)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 37, in from_world
    prep_cwl(samples, workflow_fn, out_dir, out_file, integrations, add_container_tag=add_container_tag)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 419, in prep_cwl
    variables, keyvals = _flatten_samples(samples, out_file, get_retriever)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 480, in _flatten_samples
    data["reference"] = _indexes_to_secondary_files(data["reference"], data["genome_build"])
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 515, in _indexes_to_secondary_files
    raise ValueError("Did not find indexes for %s: %s" % (refname, val))
ValueError: Did not find indexes for twobit: {'indexes': []}

My guess is that some checks are still assuming that the .2bit file is in its right spot. And since this file still gets downloaded for e.g. the hg19 genome, this issue doesn't appear when using it.
I've manually downloaded the sacCer3.2bit file into a ucsc directory in the genome location:
mkdir /home/amizeranschi/install/bcbio-vm/genomes/Scerevisiae/sacCer3/ucsc
cd /home/amizeranschi/install/bcbio-vm/genomes/Scerevisiae/sacCer3/ucsc
wget 'https://hgdownload-test.gi.ucsc.edu/goldenPath/sacCer3/bigZips/sacCer3.2bit'

and this got rid of the previous error, but now I get a different one (Did not find variable genome_resources__aliases__human):
bcbio_vm.py cwl --systemconfig ~/install/bcbio-vm/galaxy/bcbio_system.yaml --add-container-tag 1.1.1a-82fb2a0   ../config/${bcbio_exp_name}-merged.yaml
[2018-11-20T17:43Z] INFO: Using input YAML configuration: ../config/testingVC-merged.yaml
[2018-11-20T17:43Z] INFO: Checking sample YAML configuration: ../config/testingVC-merged.yaml
Preparing CWL input tarball: /home/amizeranschi/install/bcbio-vm/genomes/Scerevisiae/sacCer3/rtg--sacCer3.sdf-wf.tar.gz
Preparing CWL input tarball: /home/amizeranschi/install/bcbio-vm/genomes/Scerevisiae/sacCer3/snpeff--R64-1-1.86-wf.tar.gz
Traceback (most recent call last):
  File "/home/amizeranschi/install/bcbio-vm/anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/main.py", line 12, in run
    create.from_world(world, run_info_yaml, integrations=integrations, add_container_tag=args.add_container_tag)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 37, in from_world
    prep_cwl(samples, workflow_fn, out_dir, out_file, integrations, add_container_tag=add_container_tag)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/create.py", line 427, in prep_cwl
    for cur in workflow.generate(variables, steps, wfoutputs):
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/workflow.py", line 64, in generate
    inputs, parallel_ids, nested_inputs = _get_step_inputs(step, file_vs, std_vs, parallel_ids)
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/workflow.py", line 170, in _get_step_inputs
    for orig_input in [_get_variable(x, file_vs) for x in _handle_special_inputs(step.inputs, file_vs)]:
  File "/home/amizeranschi/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/workflow.py", line 289, in _get_variable
    raise ValueError("Did not find variable %s in \n%s" % (vid, pprint.pformat(variables)))
ValueError: Did not find variable genome_resources__aliases__human in
[{'id': 'config__algorithm__align_split_size', 'type': ['null', 'string']},
 {'id': 'files', 'type': {'items': 'File', 'type': 'array'}},
 {'id': 'config__algorithm__trim_reads',
  'type': ['string', 'null', 'boolean']},
 {'id': 'reference__fasta__base',
  'secondaryFiles': ['.fai', '^.dict'],
  'type': 'File'},
 {'id': 'config__algorithm__expression_caller', 'type': ['null', 'string']},
 {'id': 'config__algorithm__vcfanno',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'config__algorithm__svprioritize', 'type': ['null', 'string']},
 {'id': 'reference__snpeff', 'type': 'File'},
 {'id': 'config__algorithm__variantcaller',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'config__algorithm__adapters',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'config__algorithm__svcaller',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'genome_resources__rnaseq__transcripts_mask', 'type': 'File'},
 {'id': 'genome_resources__variation__editing', 'type': ['null', 'string']},
 {'id': 'postprocess_alignment/config__algorithm__coverage_interval',
  'type': ['string', 'null']},
 {'id': 'genome_resources__rnaseq__gene_bed', 'type': 'File'},
 {'id': 'reference__genome_context',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'genome_resources__variation__gc_profile', 'type': ['null', 'string']},
 {'id': 'rgnames__lb', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__clinvar', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__esp', 'type': ['null', 'string']},
 {'id': 'rgnames__rg', 'type': 'string'},
 {'id': 'config__algorithm__realign', 'type': ['string', 'null', 'boolean']},
 {'id': 'metadata__batch', 'type': 'string'},
 {'id': 'rgnames__lane', 'type': 'string'},
 {'id': 'genome_resources__variation__lcr', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__1000g', 'type': ['null', 'string']},
 {'id': 'config__algorithm__min_allele_fraction', 'type': 'double'},
 {'id': 'config__algorithm__nomap_split_targets', 'type': 'long'},
 {'id': 'reference__versions', 'type': 'File'},
 {'id': 'reference__bwa__indexes',
  'secondaryFiles': ['^.ann', '^.pac', '^.sa', '^.bwt'],
  'type': 'File'},
 {'id': 'config__algorithm__ploidy', 'type': 'long'},
 {'id': 'vrn_file', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__train_hapmap',
  'type': ['null', 'string']},
 {'id': 'config__algorithm__jointcaller',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'config__algorithm__bam_clean', 'type': ['string', 'null', 'boolean']},
 {'id': 'config__algorithm__nomap_split_size', 'type': 'long'},
 {'id': 'config__algorithm__validate', 'type': ['null', 'string']},
 {'id': 'genome_resources__rnaseq__dexseq', 'type': 'File'},
 {'id': 'config__algorithm__validate_regions', 'type': ['null', 'string']},
 {'id': 'genome_resources__rnaseq__transcriptome_index__tophat',
  'type': 'string'},
 {'id': 'config__algorithm__aligner', 'type': 'string'},
 {'id': 'rgnames__pl', 'type': 'string'},
 {'id': 'genome_build', 'type': 'string'},
 {'id': 'rgnames__pu', 'type': 'string'},
 {'id': 'genome_resources__variation__germline_het_pon',
  'type': ['null', 'string']},
 {'id': 'genome_resources__variation__exac', 'type': ['null', 'string']},
 {'id': 'config__algorithm__recalibrate',
  'type': ['string', 'null', 'boolean']},
 {'id': 'metadata__phenotype', 'type': 'string'},
 {'id': 'genome_resources__rnaseq__transcripts',
  'secondaryFiles': ['.db'],
  'type': 'File'},
 {'id': 'config__algorithm__quality_format', 'type': 'string'},
 {'id': 'config__algorithm__tools_off',
  'type': {'items': 'string', 'type': 'array'}},
 {'id': 'genome_resources__variation__dbsnp', 'type': ['null', 'string']},
 {'id': 'config__algorithm__mark_duplicates',
  'type': ['string', 'null', 'boolean']},
 {'id': 'reference__twobit', 'type': 'File'},
 {'id': 'genome_resources__variation__polyx', 'type': ['null', 'string']},
 {'id': 'genome_resources__variation__encode_blacklist',
  'type': ['null', 'string']},
 {'id': 'reference__viral', 'type': ['null', 'string']},
 {'id': 'genome_resources__version', 'type': 'long'},
 {'id': 'genome_resources__variation__cosmic', 'type': ['null', 'string']},
 {'id': 'genome_resources__rnaseq__refflat', 'type': 'File'},
 {'id': 'config__algorithm__ensemble', 'type': ['null', 'string']},
 {'id': 'config__algorithm__qc', 'type': {'items': 'string', 'type': 'array'}},
 {'id': 'config__algorithm__sv_regions', 'type': 'File'},
 {'id': 'analysis', 'type': 'string'},
 {'id': 'prep_samples/rgnames__sample', 'type': 'string'},
 {'id': 'config__algorithm__tools_on',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'config__algorithm__effects', 'type': ['string', 'null', 'boolean']},
 {'id': 'postprocess_alignment/config__algorithm__variant_regions',
  'type': ['File', 'null']},
 {'id': 'config__algorithm__svvalidate', 'type': ['null', 'string']},
 {'id': 'genome_resources__aliases__ensembl', 'type': 'string'},
 {'id': 'config__algorithm__exclude_regions',
  'type': ['null', {'items': 'null', 'type': 'array'}]},
 {'id': 'reference__rtg', 'type': 'File'},
 {'id': 'genome_resources__variation__train_indels',
  'type': ['null', 'string']},
 {'id': 'genome_resources__aliases__snpeff', 'type': 'string'},
 {'id': 'config__algorithm__archive', 'type': ['null', 'string']},
 {'id': 'alignment_to_rec/alignment_rec',
  'type': {'items': {'fields': [{'name': 'resources', 'type': 'string'},
                                {'name': 'description', 'type': 'string'},
                                {'name': 'config__algorithm__align_split_size',
                                 'type': ['null', 'string']},
                                {'name': 'files',
                                 'type': {'items': 'File',
                                          'type': 'array'}},
                                {'name': 'config__algorithm__trim_reads',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'reference__fasta__base',
                                 'secondaryFiles': ['.fai', '^.dict'],
                                 'type': 'File'},
                                {'name': 'config__algorithm__adapters',
                                 'type': ['null',
                                          {'items': 'null',
                                           'type': 'array'}]},
                                {'name': 'rgnames__lb',
                                 'type': ['null', 'string']},
                                {'name': 'rgnames__rg', 'type': 'string'},
                                {'name': 'rgnames__lane',
                                 'type': 'string'},
                                {'name': 'reference__bwa__indexes',
                                 'secondaryFiles': ['^.ann',
                                                    '^.pac',
                                                    '^.sa',
                                                    '^.bwt'],
                                 'type': 'File'},
                                {'name': 'config__algorithm__bam_clean',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'config__algorithm__aligner',
                                 'type': 'string'},
                                {'name': 'rgnames__pl', 'type': 'string'},
                                {'name': 'rgnames__pu', 'type': 'string'},
                                {'name': 'config__algorithm__mark_duplicates',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'analysis', 'type': 'string'},
                                {'name': 'rgnames__sample',
                                 'type': 'string'},
                                {'name': 'config__algorithm__variant_regions',
                                 'type': 'File'}],
                     'name': 'alignment_rec',
                     'type': 'record'},
           'type': 'array'}},
 {'id': 'postprocess_alignment/align_bam',
  'secondaryFiles': ['.bai'],
  'type': ['File', 'null']},
 {'id': 'alignment/hla__fastq',
  'type': ['null', {'items': 'File', 'type': 'array'}]},
 {'id': 'alignment/work_bam_plus__disc',
  'secondaryFiles': ['.bai'],
  'type': ['File', 'null']},
 {'id': 'alignment/work_bam_plus__sr',
  'secondaryFiles': ['.bai'],
  'type': ['File', 'null']},
 {'id': 'prep_samples_to_rec/prep_samples_rec',
  'type': {'items': {'fields': [{'name': 'resources', 'type': 'string'},
                                {'name': 'description', 'type': 'string'},
                                {'name': 'reference__fasta__base',
                                 'secondaryFiles': ['.fai', '^.dict'],
                                 'type': 'File'},
                                {'name': 'rgnames__sample',
                                 'type': 'string'},
                                {'name': 'config__algorithm__variant_regions',
                                 'type': 'File'}],
                     'name': 'prep_samples_rec',
                     'type': 'record'},
           'type': 'array'}},
 {'id': 'postprocess_alignment/config__algorithm__variant_regions_merged',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/config__algorithm__variant_regions_orig',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/config__algorithm__coverage',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/config__algorithm__coverage_merged',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/config__algorithm__coverage_orig',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/config__algorithm__seq2c_bed_ready',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment_to_rec/postprocess_alignment_rec',
  'type': {'items': {'fields': [{'name': 'resources', 'type': 'string'},
                                {'name': 'description', 'type': 'string'},
                                {'name': 'reference__fasta__base',
                                 'secondaryFiles': ['.fai', '^.dict'],
                                 'type': 'File'},
                                {'name': 'config__algorithm__coverage_interval',
                                 'type': ['null', 'string']},
                                {'name': 'genome_resources__rnaseq__gene_bed',
                                 'type': 'File'},
                                {'name': 'genome_resources__variation__lcr',
                                 'type': ['null', 'string']},
                                {'name': 'config__algorithm__recalibrate',
                                 'type': ['string', 'null', 'boolean']},
                                {'name': 'genome_resources__variation__dbsnp',
                                 'type': ['null', 'string']},
                                {'name': 'genome_resources__variation__polyx',
                                 'type': ['null', 'string']},
                                {'name': 'genome_resources__variation__encode_blacklist',
                                 'type': ['null', 'string']},
                                {'name': 'config__algorithm__tools_on',
                                 'type': ['null',
                                          {'items': 'null',
                                           'type': 'array'}]},
                                {'name': 'config__algorithm__variant_regions',
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__exclude_regions',
                                 'type': ['null',
                                          {'items': 'null',
                                           'type': 'array'}]},
                                {'name': 'align_bam',
                                 'secondaryFiles': ['.bai'],
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__variant_regions_merged',
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__variant_regions_orig',
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__coverage',
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__coverage_merged',
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__coverage_orig',
                                 'type': ['File', 'null']},
                                {'name': 'config__algorithm__seq2c_bed_ready',
                                 'type': ['File', 'null']}],
                     'name': 'postprocess_alignment_rec',
                     'type': 'record'},
           'type': 'array'}},
 {'id': 'postprocess_alignment/regions__callable', 'type': ['File', 'null']},
 {'id': 'postprocess_alignment/regions__sample_callable',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/regions__nblock', 'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__samtools__stats',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__samtools__idxstats',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__variant_regions__regions',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__variant_regions__dist',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__sv_regions__regions',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__sv_regions__dist',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__coverage__regions',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__coverage__dist',
  'type': ['File', 'null']},
 {'id': 'postprocess_alignment/depth__coverage__thresholds',
  'type': ['File', 'null']},
 {'id': 'combine_sample_regions/config__algorithm__callable_regions',
  'type': 'File'},
 {'id': 'combine_sample_regions/config__algorithm__non_callable_regions',
  'type': 'File'},
 {'id': 'combine_sample_regions/config__algorithm__callable_count',
  'type': 'int'}]

Things are definitely moving forward and I got bcbio_vm.py cwl to run. Thanks for your help. However, there was another error popping up at the next stage. This is how I'm setting up the CWL run (am I using the option --add-container-tag correctly?):
bcbio_vm.py cwl --add-container-tag 1.1.1a-82fb2a0 --systemconfig ~/install/bcbio-vm/galaxy/bcbio_system.yaml  ../config/${bcbio_exp_name}-merged.yaml
bcbio_vm.py cwlrun cromwell ${bcbio_exp_name}-merged-workflow

And below is the outcome:
[2018-11-21 17:16:50,08] [info] Slf4jLogger started
[2018-11-21 17:16:50,14] [info] Pre Processing Workflow...
[2018-11-21 17:16:50,30] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/main-testingVC-merged.cwl
[2018-11-21 17:16:51,98] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/alignment_to_rec.cwl
[2018-11-21 17:16:54,45] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/wf-alignment.cwl
[2018-11-21 17:16:55,53] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/prep_align_inputs.cwl
[2018-11-21 17:16:57,50] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/process_alignment.cwl
[2018-11-21 17:17:01,57] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/merge_split_alignments.cwl
[2018-11-21 17:17:03,52] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/prep_samples_to_rec.cwl
[2018-11-21 17:17:05,35] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/prep_samples.cwl
[2018-11-21 17:17:07,20] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/postprocess_alignment_to_rec.cwl
[2018-11-21 17:17:09,15] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/postprocess_alignment.cwl
[2018-11-21 17:17:11,43] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/combine_sample_regions.cwl
[2018-11-21 17:17:13,75] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/batch_for_variantcall.cwl
[2018-11-21 17:17:15,86] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/wf-variantcall.cwl
[2018-11-21 17:17:17,01] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/get_parallel_regions.cwl
[2018-11-21 17:17:18,98] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/variantcall_batch_region.cwl
[2018-11-21 17:17:21,54] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/concat_batch_variantcalls.cwl
[2018-11-21 17:17:23,75] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/compare_to_rm.cwl
[2018-11-21 17:17:25,65] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/batch_for_jointvc.cwl
[2018-11-21 17:17:27,62] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/wf-jointcall.cwl
[2018-11-21 17:17:28,79] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/get_parallel_regions_jointvc.cwl
[2018-11-21 17:17:30,65] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/run_jointvc.cwl
[2018-11-21 17:17:32,91] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/concat_batch_variantcalls_jointvc.cwl
[2018-11-21 17:17:35,19] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/postprocess_variants.cwl
[2018-11-21 17:17:37,24] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/finalize_jointvc.cwl
[2018-11-21 17:17:39,20] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/summarize_vc.cwl
[2018-11-21 17:17:41,12] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/calculate_sv_bins.cwl
[2018-11-21 17:17:43,04] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/calculate_sv_coverage.cwl
[2018-11-21 17:17:45,01] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/normalize_sv_coverage.cwl
[2018-11-21 17:17:46,84] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/batch_for_sv.cwl
[2018-11-21 17:17:48,82] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/wf-svcall.cwl
[2018-11-21 17:17:49,97] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/detect_sv.cwl
[2018-11-21 17:17:52,28] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/summarize_sv.cwl
[2018-11-21 17:17:54,22] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/qc_to_rec.cwl
[2018-11-21 17:17:56,57] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/pipeline_summary.cwl
[2018-11-21 17:17:58,81] [info] Pre-Processing /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/steps/multiqc_summary.cwl
[2018-11-21 17:18:00,78] [info] Pre Processing Inputs...
[2018-11-21 17:18:00,91] [info] Workflow heartbeat configuration:
{
  "cromwellId" : "cromid-b584c34",
  "heartbeatInterval" : "2 minutes",
  "ttl" : "10 minutes",
  "writeBatchSize" : 10000,
  "writeThreshold" : 10000
}
[2018-11-21 17:18:01,05] [info] Metadata summary refreshing every 2 seconds.
[2018-11-21 17:18:01,07] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-11-21 17:18:01,07] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-11-21 17:18:01,10] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
[2018-11-21 17:18:01,53] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds.
[2018-11-21 17:18:01,54] [info] SingleWorkflowRunnerActor: Version 36-unknown-SNAP
[2018-11-21 17:18:01,55] [info] SingleWorkflowRunnerActor: Submitting workflow
[2018-11-21 17:18:01,59] [info] CWL (Unspecified version) workflow 0856301d-b471-4dd7-bd81-029ac3cf304d submitted
[2018-11-21 17:18:01,59] [info] SingleWorkflowRunnerActor: Workflow submitted 0856301d-b471-4dd7-bd81-029ac3cf304d
[2018-11-21 17:18:01,60] [info] 1 new workflows fetched
[2018-11-21 17:18:01,60] [info] WorkflowManagerActor Starting workflow 0856301d-b471-4dd7-bd81-029ac3cf304d
[2018-11-21 17:18:01,60] [info] WorkflowManagerActor Successfully started WorkflowActor-0856301d-b471-4dd7-bd81-029ac3cf304d
[2018-11-21 17:18:01,60] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2018-11-21 17:18:01,61] [warn] SingleWorkflowRunnerActor: received unexpected message: Done in state RunningSwraData
[2018-11-21 17:18:01,62] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
[2018-11-21 17:18:01,68] [info] MaterializeWorkflowDescriptorActor [0856301d]: Parsing workflow as CWL v1.0
[2018-11-21 17:18:03,16] [info] MaterializeWorkflowDescriptorActor [0856301d]: Call-to-Backend assignments: alignment_to_rec -> Local, calculate_sv_coverage -> Local, batch_for_variantcall -> Local, prep_samples -> Local, get_parallel_regions_jointvc -> Local, pipeline_summary -> Local, finalize_jointvc -> Local, postprocess_alignment -> Local, batch_for_jointvc -> Local, batch_for_sv -> Local, compare_to_rm -> Local, concat_batch_variantcalls_jointvc -> Local, process_alignment -> Local, variantcall_batch_region -> Local, calculate_sv_bins -> Local, prep_samples_to_rec -> Local, postprocess_variants -> Local, combine_sample_regions -> Local, get_parallel_regions -> Local, postprocess_alignment_to_rec -> Local, qc_to_rec -> Local, detect_sv -> Local, merge_split_alignments -> Local, summarize_sv -> Local, multiqc_summary -> Local, prep_align_inputs -> Local, normalize_sv_coverage -> Local, run_jointvc -> Local, summarize_vc -> Local, concat_batch_variantcalls -> Local
[2018-11-21 17:18:13,93] [error] WorkflowManagerActor Workflow 0856301d-b471-4dd7-bd81-029ac3cf304d failed (during InitializingWorkflowState): cromwell.core.CromwellFatalException: java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
	at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:46)
	at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:37)
	at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:413)
	at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
	at com.google.auth.oauth2.DefaultCredentialsProvider.getDefaultCredentials(DefaultCredentialsProvider.java:131)
	at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:127)
	at com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(GoogleCredentials.java:100)
	at cromwell.cloudsupport.gcp.auth.ApplicationDefaultMode$.applicationDefaultCredentials$lzycompute(GoogleAuthMode.scala:276)
	at cromwell.cloudsupport.gcp.auth.ApplicationDefaultMode$.cromwell$cloudsupport$gcp$auth$ApplicationDefaultMode$$applicationDefaultCredentials(GoogleAuthMode.scala:276)
	at cromwell.cloudsupport.gcp.auth.ApplicationDefaultMode.credentials(GoogleAuthMode.scala:282)
	at cromwell.cloudsupport.gcp.auth.ApplicationDefaultMode.credentials(GoogleAuthMode.scala:279)
	at cromwell.cloudsupport.gcp.auth.GoogleAuthMode.pipelinesApiCredentials(GoogleAuthMode.scala:128)
	at cromwell.cloudsupport.gcp.auth.GoogleAuthMode.pipelinesApiCredentials$(GoogleAuthMode.scala:127)
	at cromwell.cloudsupport.gcp.auth.ApplicationDefaultMode.pipelinesApiCredentials(GoogleAuthMode.scala:279)
	at cromwell.filesystems.gcs.GoogleUtil$EnhancedGoogleAuthMode$.credential$1(GoogleUtil.scala:33)
	at cromwell.filesystems.gcs.GoogleUtil$EnhancedGoogleAuthMode$.$anonfun$retryPipelinesApiCredentials$4(GoogleUtil.scala:49)
	at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:654)
	at scala.util.Success.$anonfun$map$1(Try.scala:251)
	at scala.util.Success.map(Try.scala:209)
	at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)
	at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
	at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
	... 12 more

[2018-11-21 17:18:13,93] [info] WorkflowManagerActor WorkflowActor-0856301d-b471-4dd7-bd81-029ac3cf304d is in a terminal state: WorkflowFailedState
[2018-11-21 17:18:21,95] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2018-11-21 17:18:26,22] [info] SingleWorkflowRunnerActor writing metadata to /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-metadata.json
[2018-11-21 17:18:26,25] [info] Workflow polling stopped
[2018-11-21 17:18:26,26] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-11-21 17:18:26,27] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-11-21 17:18:26,27] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-11-21 17:18:26,27] [info] Aborting all running workflows.
[2018-11-21 17:18:26,27] [info] JobExecutionTokenDispenser stopped
[2018-11-21 17:18:26,27] [info] WorkflowStoreActor stopped
[2018-11-21 17:18:26,27] [info] WorkflowLogCopyRouter stopped
[2018-11-21 17:18:26,27] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-11-21 17:18:26,28] [info] WorkflowManagerActor All workflows finished
[2018-11-21 17:18:26,28] [info] WorkflowManagerActor stopped
[2018-11-21 17:18:26,28] [info] Connection pools shut down
[2018-11-21 17:18:26,28] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-11-21 17:18:26,28] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-11-21 17:18:26,28] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-11-21 17:18:26,28] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-11-21 17:18:26,28] [info] SubWorkflowStoreActor stopped
[2018-11-21 17:18:26,28] [info] JobStoreActor stopped
[2018-11-21 17:18:26,28] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-11-21 17:18:26,28] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2018-11-21 17:18:26,28] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-11-21 17:18:26,28] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-11-21 17:18:26,28] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-11-21 17:18:26,28] [info] CallCacheWriteActor stopped
[2018-11-21 17:18:26,29] [info] ServiceRegistryActor stopped
[2018-11-21 17:18:26,29] [info] DockerHashActor stopped
[2018-11-21 17:18:26,29] [info] IoProxy stopped
[2018-11-21 17:18:26,30] [info] Database closed
[2018-11-21 17:18:26,30] [info] Stream materializer shut down
[2018-11-21 17:18:26,30] [info] WDL HTTP import resolver closed
Workflow 0856301d-b471-4dd7-bd81-029ac3cf304d transitioned to state Failed
Unable to find image 'quay.io/bcbio/bcbio-base:latest' locally
latest: Pulling from bcbio/bcbio-base
18d680d61657: Pull complete
0addb6fece63: Pull complete
78e58219b215: Pull complete
eb6959a66df2: Pull complete
f4d326eaeb65: Pull complete
Digest: sha256:4d4f67015e28437cda1c39e5b40926600eedd41b1b87fe69ffe5445c7bf28f84
Status: Downloaded newer image for quay.io/bcbio/bcbio-base:latest
Failed bcbio Cromwell run
-------------------------

Sorry you're still hitting issues here, thank you for all the patience. It looks like your CWL run specification is correct and should be adding the right tags to the output CWL. You can check the dockerImageId in any cwl file in yourproject-workflow/steps to confirm that it's added to the specification. What I'm confused about is why it's pulling bcbio-base from latest. I'd thought this was only used in the build step and should be baked into the actual bcbio-vc container you're using but must not be right. I think the issue is that we need to be able to specify the revision in the FROM tag in the Docker container:
https://github.com/bcbio/bcbio_docker/blob/master/Dockerfile.tools#L3
I'll dig to check if this is now possible and see if we can pin these explicitly. For now if you have bcbio/bcbio-base:latest available it'll use your specific docker container for the actual run. Does having that avoid the Cromwell errors?
I'll also work on not needing to have the Docker image present for downloading data. We should be able to avoid that now. Apologies, this needs some cleanup in the transition to CWL and I'll work on that over the next couple of days and follow up here when there is a better process in place.
Thanks again for helping work through these issues.

It actually did download the bcbio/bcbio-base:latest image during the first run, but it still crashed. On a subsequent run, it skipped over the bcbio/bcbio-base:latest part, and then ended with Failed bcbio Cromwell run.
It seems like the source of the error is the following line:
[2018-11-21 17:18:13,93] [error] WorkflowManagerActor Workflow 0856301d-b471-4dd7-bd81-029ac3cf304d failed (during InitializingWorkflowState): cromwell.core.CromwellFatalException: java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
	at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:46)
	at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:37)
[...]

This is puzzling to me. Why does it error out due to missing Google cloud credentials, when I'm trying to run this locally?

I checked the dockerImageId value and it is correctly set to quay.io/bcbio/bcbio-vc:1.1.1a-82fb2a0. I didn't find the string "bcbio-base" in any of the files inside the CWL project directory.
I ran the CWL test with toil instead of cromwell (bcbio_vm.py cwlrun cromwell somatic-workflow) and this also failed. Apologies for the large block of text:
bcbio_vm.py cwlrun toil somatic-workflow
Running: export PATH=/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin:$PATH && cwltoil --jobStore /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/cwltoil_jobstore --logFile /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/somatic-toil.log --workDir /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/tmpdir --linkImports -- /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic.cwl /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic-samples.json
amizeranschi-Predator-PH317-51 2018-11-22 21:58:35,671 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:35,671 MainThread INFO toil.lib.bioio: Logging to file '/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/somatic-toil.log'.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:35,671 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:35,671 MainThread INFO toil.lib.bioio: Logging to file '/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/somatic-toil.log'.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:35,675 MainThread INFO toil.jobStores.abstractJobStore: The workflow ID is: 'ded58332-e978-4cd6-a654-df89ccaff86b'
Resolved '/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic.cwl' to 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic.cwl'
amizeranschi-Predator-PH317-51 2018-11-22 21:58:35,676 MainThread INFO cwltool: Resolved '/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic.cwl' to 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic.cwl'
../somatic-workflow/steps/alignment_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:37,867 MainThread INFO cwltool: ../somatic-workflow/steps/alignment_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/prep_align_inputs.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:37,984 MainThread INFO cwltool: ../somatic-workflow/steps/prep_align_inputs.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/process_alignment.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:39,706 MainThread INFO cwltool: ../somatic-workflow/steps/process_alignment.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/process_alignment.cwl:83:3: Unknown hint http://arvados.org/cwl#APIRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:39,706 MainThread INFO cwltool: ../somatic-workflow/steps/process_alignment.cwl:83:3: Unknown hint http://arvados.org/cwl#APIRequirement
../somatic-workflow/steps/merge_split_alignments.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:39,778 MainThread INFO cwltool: ../somatic-workflow/steps/merge_split_alignments.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/prep_samples_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:39,832 MainThread INFO cwltool: ../somatic-workflow/steps/prep_samples_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/prep_samples.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:39,867 MainThread INFO cwltool: ../somatic-workflow/steps/prep_samples.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/postprocess_alignment_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:39,942 MainThread INFO cwltool: ../somatic-workflow/steps/postprocess_alignment_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/postprocess_alignment.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,015 MainThread INFO cwltool: ../somatic-workflow/steps/postprocess_alignment.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/postprocess_alignment.cwl:52:3: Unknown hint http://arvados.org/cwl#APIRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,015 MainThread INFO cwltool: ../somatic-workflow/steps/postprocess_alignment.cwl:52:3: Unknown hint http://arvados.org/cwl#APIRequirement
../somatic-workflow/steps/combine_sample_regions.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,067 MainThread INFO cwltool: ../somatic-workflow/steps/combine_sample_regions.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/combine_sample_regions.cwl:40:3: Unknown hint http://arvados.org/cwl#APIRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,067 MainThread INFO cwltool: ../somatic-workflow/steps/combine_sample_regions.cwl:40:3: Unknown hint http://arvados.org/cwl#APIRequirement
../somatic-workflow/steps/batch_for_variantcall.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,222 MainThread INFO cwltool: ../somatic-workflow/steps/batch_for_variantcall.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/get_parallel_regions.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,433 MainThread INFO cwltool: ../somatic-workflow/steps/get_parallel_regions.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/variantcall_batch_region.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,943 MainThread INFO cwltool: ../somatic-workflow/steps/variantcall_batch_region.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/variantcall_batch_region.cwl:112:3: Unknown hint http://arvados.org/cwl#APIRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,944 MainThread INFO cwltool: ../somatic-workflow/steps/variantcall_batch_region.cwl:112:3: Unknown hint http://arvados.org/cwl#APIRequirement
../somatic-workflow/steps/variantcall_batch_region.cwl:113:3: Unknown hint http://arvados.org/cwl#RuntimeConstraints
amizeranschi-Predator-PH317-51 2018-11-22 21:58:40,944 MainThread INFO cwltool: ../somatic-workflow/steps/variantcall_batch_region.cwl:113:3: Unknown hint http://arvados.org/cwl#RuntimeConstraints
../somatic-workflow/steps/concat_batch_variantcalls.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,030 MainThread INFO cwltool: ../somatic-workflow/steps/concat_batch_variantcalls.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/concat_batch_variantcalls.cwl:40:3: Unknown hint http://arvados.org/cwl#APIRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,030 MainThread INFO cwltool: ../somatic-workflow/steps/concat_batch_variantcalls.cwl:40:3: Unknown hint http://arvados.org/cwl#APIRequirement
../somatic-workflow/steps/postprocess_variants.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,098 MainThread INFO cwltool: ../somatic-workflow/steps/postprocess_variants.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/compare_to_rm.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,208 MainThread INFO cwltool: ../somatic-workflow/steps/compare_to_rm.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/batch_for_ensemble.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,357 MainThread INFO cwltool: ../somatic-workflow/steps/batch_for_ensemble.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/combine_calls.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,426 MainThread INFO cwltool: ../somatic-workflow/steps/combine_calls.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/summarize_vc.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,523 MainThread INFO cwltool: ../somatic-workflow/steps/summarize_vc.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/qc_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,623 MainThread INFO cwltool: ../somatic-workflow/steps/qc_to_rec.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/pipeline_summary.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,709 MainThread INFO cwltool: ../somatic-workflow/steps/pipeline_summary.cwl:27:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
../somatic-workflow/steps/pipeline_summary.cwl:76:3: Unknown hint http://arvados.org/cwl#RuntimeConstraints
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,709 MainThread INFO cwltool: ../somatic-workflow/steps/pipeline_summary.cwl:76:3: Unknown hint http://arvados.org/cwl#RuntimeConstraints
../somatic-workflow/steps/multiqc_summary.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:41,760 MainThread INFO cwltool: ../somatic-workflow/steps/multiqc_summary.cwl:26:3: Unknown hint https://www.dnanexus.com/cwl#InputResourceRequirement
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,009 MainThread INFO toil.common: Using the single machine batch system
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,009 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxCores to CPU count of system (8).
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,010 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (8242171904).
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,010 MainThread INFO toil.common: Created the workflow directory at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/tmpdir/toil-ded58332-e978-4cd6-a654-df89ccaff86b-5a229753ce0549c6bf7503d6f103b439
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,010 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxDisk to physically available disk (78591340544).
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,017 MainThread INFO toil.common: User script ModuleDescriptor(dirPath='/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=False) belongs to Toil. No need to hot-deploy it.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,018 MainThread INFO toil.common: No user script to hot-deploy.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,018 MainThread INFO toil.common: Written the environment for the jobs to the environment file
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,018 MainThread INFO toil.common: Caching all jobs in job store
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,019 MainThread INFO toil.common: 0 jobs downloaded.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,162 MainThread INFO toil: Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,163 MainThread INFO toil.realtimeLogger: Real-time logging disabled
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,165 MainThread INFO toil.toilState: (Re)building internal scheduler state
2018-11-22 21:58:42,165 - toil.toilState - INFO - (Re)building internal scheduler state
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,165 MainThread INFO toil.leader: Found 1 jobs to start and 0 jobs with successors to run
2018-11-22 21:58:42,165 - toil.leader - INFO - Found 1 jobs to start and 0 jobs with successors to run
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,165 MainThread INFO toil.leader: Checked batch system has no running jobs and no updated jobs
2018-11-22 21:58:42,165 - toil.leader - INFO - Checked batch system has no running jobs and no updated jobs
amizeranschi-Predator-PH317-51 2018-11-22 21:58:42,166 MainThread INFO toil.leader: Starting the main loop
2018-11-22 21:58:42,166 - toil.leader - INFO - Starting the main loop
amizeranschi-Predator-PH317-51 2018-11-22 21:58:43,084 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi with job batch system ID: 1 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 21:58:43,084 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi with job batch system ID: 1 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 21:58:43,087 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with job batch system ID: 2 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 21:58:43,087 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with job batch system ID: 2 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,722 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
2018-11-22 22:10:13,722 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,723 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
2018-11-22 22:10:13,723 - toil.leader - WARNING - The job seems to have left a log file, indicating failure: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,723 MainThread WARNING toil.leader: D/X/job_AlQdQ    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
2018-11-22 22:10:13,723 - toil.leader - WARNING - D/X/job_AlQdQ    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,723 MainThread WARNING toil.leader: D/X/job_AlQdQ    INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
2018-11-22 22:10:13,723 - toil.leader - WARNING - D/X/job_AlQdQ    INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    ['docker', 'pull', 'quay.io/bcbio/bcbio-vc']
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    ['docker', 'pull', 'quay.io/bcbio/bcbio-vc']
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    INFO:cwltool:['docker', 'pull', 'quay.io/bcbio/bcbio-vc']
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    INFO:cwltool:['docker', 'pull', 'quay.io/bcbio/bcbio-vc']
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    Using default tag: latest
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    Using default tag: latest
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    latest: Pulling from bcbio/bcbio-vc
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    latest: Pulling from bcbio/bcbio-vc
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    18d680d61657: Already exists
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    18d680d61657: Already exists
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    0addb6fece63: Already exists
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    0addb6fece63: Already exists
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    78e58219b215: Already exists
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    78e58219b215: Already exists
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    eb6959a66df2: Already exists
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    eb6959a66df2: Already exists
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    f4d326eaeb65: Already exists
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    f4d326eaeb65: Already exists
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    73f63353b610: Pulling fs layer
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    73f63353b610: Pulling fs layer
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    af6367e4755f: Pulling fs layer
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    af6367e4755f: Pulling fs layer
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    73f63353b610: Download complete
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    73f63353b610: Download complete
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    73f63353b610: Pull complete
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    73f63353b610: Pull complete
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    af6367e4755f: Verifying Checksum
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    af6367e4755f: Verifying Checksum
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    af6367e4755f: Download complete
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    af6367e4755f: Download complete
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,724 MainThread WARNING toil.leader: D/X/job_AlQdQ    af6367e4755f: Pull complete
2018-11-22 22:10:13,724 - toil.leader - WARNING - D/X/job_AlQdQ    af6367e4755f: Pull complete
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ    Digest: sha256:7748650d81ab0b549172846b467f0281c15e523dbf977b2731bda20fad284711
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ    Digest: sha256:7748650d81ab0b549172846b467f0281c15e523dbf977b2731bda20fad284711
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ    Status: Downloaded newer image for quay.io/bcbio/bcbio-vc:latest
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ    Status: Downloaded newer image for quay.io/bcbio/bcbio-vc:latest
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ    Traceback (most recent call last):
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ    Traceback (most recent call last):
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        returnValues = self._run(jobGraph, fileStore)
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        returnValues = self._run(jobGraph, fileStore)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        return self.run(fileStore)
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        return self.run(fileStore)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        (output, status) = cwltool.main.single_job_executor(**opts)
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        (output, status) = cwltool.main.single_job_executor(**opts)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        r.run(**kwargs)
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        r.run(**kwargs)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        self._setup(kwargs)
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        self._setup(kwargs)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,725 MainThread WARNING toil.leader: D/X/job_AlQdQ        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
2018-11-22 22:10:13,725 - toil.leader - WARNING - D/X/job_AlQdQ        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,726 MainThread WARNING toil.leader: D/X/job_AlQdQ    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai) not found or is not a regular file.
2018-11-22 22:10:13,726 - toil.leader - WARNING - D/X/job_AlQdQ    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai) not found or is not a regular file.
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,726 MainThread WARNING toil.leader: D/X/job_AlQdQ    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
2018-11-22 22:10:13,726 - toil.leader - WARNING - D/X/job_AlQdQ    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,726 MainThread WARNING toil.leader: D/X/job_AlQdQ    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ to 1
2018-11-22 22:10:13,726 - toil.leader - WARNING - D/X/job_AlQdQ    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ to 1
amizeranschi-Predator-PH317-51 2018-11-22 22:10:13,726 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with job batch system ID: 3 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 22:10:13,726 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with job batch system ID: 3 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,092 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
2018-11-22 22:10:16,092 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
2018-11-22 22:10:16,093 - toil.leader - WARNING - The job seems to have left a log file, indicating failure: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ    INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ    INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ    Traceback (most recent call last):
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ    Traceback (most recent call last):
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ        returnValues = self._run(jobGraph, fileStore)
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ        returnValues = self._run(jobGraph, fileStore)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,093 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
2018-11-22 22:10:16,093 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ        return self.run(fileStore)
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ        return self.run(fileStore)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ        (output, status) = cwltool.main.single_job_executor(**opts)
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ        (output, status) = cwltool.main.single_job_executor(**opts)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ        r.run(**kwargs)
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ        r.run(**kwargs)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ        self._setup(kwargs)
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ        self._setup(kwargs)
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai) not found or is not a regular file.
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai) not found or is not a regular file.
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,094 MainThread WARNING toil.leader: D/X/job_AlQdQ    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ to 0
2018-11-22 22:10:16,094 - toil.leader - WARNING - D/X/job_AlQdQ    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ to 0
amizeranschi-Predator-PH317-51 2018-11-22 22:10:16,099 MainThread WARNING toil.leader: Job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ is completely failed
2018-11-22 22:10:16,099 - toil.leader - WARNING - Job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ is completely failed
amizeranschi-Predator-PH317-51 2018-11-22 22:10:22,216 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi
2018-11-22 22:10:22,216 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi
amizeranschi-Predator-PH317-51 2018-11-22 22:10:22,833 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl t/Q/jobYX5NcG with job batch system ID: 5 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 22:10:22,833 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl t/Q/jobYX5NcG with job batch system ID: 5 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 22:10:22,837 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl v/z/jobBKnwKQ with job batch system ID: 6 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 22:10:22,837 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl v/z/jobBKnwKQ with job batch system ID: 6 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 22:10:22,838 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl 5/9/jobg3gTOT with job batch system ID: 7 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 22:10:22,838 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl 5/9/jobg3gTOT with job batch system ID: 7 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 22:10:22,838 MainThread INFO toil.leader: Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl i/T/jobIa8lhb with job batch system ID: 8 and cores: 1, disk: 1.0 G, and memory: 2.0 G
2018-11-22 22:10:22,838 - toil.leader - INFO - Issued job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl i/T/jobIa8lhb with job batch system ID: 8 and cores: 1, disk: 1.0 G, and memory: 2.0 G
amizeranschi-Predator-PH317-51 2018-11-22 22:10:32,212 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl 5/9/jobg3gTOT
2018-11-22 22:10:32,212 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl 5/9/jobg3gTOT
amizeranschi-Predator-PH317-51 2018-11-22 22:10:32,527 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl t/Q/jobYX5NcG
2018-11-22 22:10:32,527 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl t/Q/jobYX5NcG
amizeranschi-Predator-PH317-51 2018-11-22 22:10:38,046 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl v/z/jobBKnwKQ
2018-11-22 22:10:38,046 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl v/z/jobBKnwKQ
amizeranschi-Predator-PH317-51 2018-11-22 22:10:40,261 MainThread INFO toil.leader: Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl i/T/jobIa8lhb
2018-11-22 22:10:40,261 - toil.leader - INFO - Job ended successfully: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples.cwl' bcbio_nextgen.py runfn prep_samples cwl i/T/jobIa8lhb
amizeranschi-Predator-PH317-51 2018-11-22 22:10:50,937 MainThread INFO toil.leader: No jobs left to run so exiting.
2018-11-22 22:10:50,937 - toil.leader - INFO - No jobs left to run so exiting.
amizeranschi-Predator-PH317-51 2018-11-22 22:10:50,938 MainThread INFO toil.leader: Finished the main loop
2018-11-22 22:10:50,938 - toil.leader - INFO - Finished the main loop
amizeranschi-Predator-PH317-51 2018-11-22 22:10:50,938 MainThread INFO toil.serviceManager: Waiting for service manager thread to finish ...
2018-11-22 22:10:50,938 - toil.serviceManager - INFO - Waiting for service manager thread to finish ...
amizeranschi-Predator-PH317-51 2018-11-22 22:10:51,346 MainThread INFO toil.serviceManager: ... finished shutting down the service manager. Took 0.406985998154 seconds
2018-11-22 22:10:51,346 - toil.serviceManager - INFO - ... finished shutting down the service manager. Took 0.406985998154 seconds
amizeranschi-Predator-PH317-51 2018-11-22 22:10:51,347 MainThread INFO toil.statsAndLogging: Waiting for stats and logging collator thread to finish ...
2018-11-22 22:10:51,347 - toil.statsAndLogging - INFO - Waiting for stats and logging collator thread to finish ...
amizeranschi-Predator-PH317-51 2018-11-22 22:10:51,464 MainThread INFO toil.statsAndLogging: ... finished collating stats and logs. Took 0.116714000702 seconds
2018-11-22 22:10:51,464 - toil.statsAndLogging - INFO - ... finished collating stats and logs. Took 0.116714000702 seconds
amizeranschi-Predator-PH317-51 2018-11-22 22:10:51,464 MainThread INFO toil.leader: Finished toil run with 5 failed jobs
2018-11-22 22:10:51,464 - toil.leader - INFO - Finished toil run with 5 failed jobs
amizeranschi-Predator-PH317-51 2018-11-22 22:10:51,465 MainThread INFO toil.leader: Failed jobs at end of the run: 'CWLGather' s/0/job8oR0hU 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ 'CWLWorkflow' 9/7/jobPiIsEy 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi 'CWLScatter' u/X/jobMpPwRP
2018-11-22 22:10:51,465 - toil.leader - INFO - Failed jobs at end of the run: 'CWLGather' s/0/job8oR0hU 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ 'CWLWorkflow' 9/7/jobPiIsEy 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi 'CWLScatter' u/X/jobMpPwRP
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/cwltoil", line 11, in <module>
    load_entry_point('toil==3.14.0', 'console_scripts', 'cwltoil')()
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 990, in main
    outobj = toil.start(wf1)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/common.py", line 743, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/common.py", line 1018, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/leader.py", line 226, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/cwltoil_jobstore' contains 5 failed jobs: 'CWLGather' s/0/job8oR0hU, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ, 'CWLWorkflow' 9/7/jobPiIsEy, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl q/Q/jobTevZvi, 'CWLScatter' u/X/jobMpPwRP
=========> Failed job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ
INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
    return self.run(fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
    (output, status) = cwltool.main.single_job_executor(**opts)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
    r.run(**kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
    self._setup(kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
    "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai) not found or is not a regular file.
ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl D/X/job_AlQdQ with ID D/X/job_AlQdQ to 0
<=========

Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 294, in run
    _TOOLS[args.tool](args)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 121, in _run_toil
    _run_tool(cmd, not args.no_container, work_dir)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 42, in _run_tool
    subprocess.check_call(cmd, shell=True)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'export PATH=/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin:$PATH && cwltoil --jobStore /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/cwltoil_jobstore --logFile /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/somatic-toil.log --workDir /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/toil_work/tmpdir --linkImports -- /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic.cwl /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/somatic-workflow/main-somatic-samples.json' returned non-zero exit status 1

Thanks again for all this testing and your patience in working through the issues.
For the Cromwell problem, there was a bug in how we were specifying the filesystem for local Docker runs which made it look for Google permissions. I pushed a fix for this, so if you update with:
/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/conda install -c conda-forge -c bioconda bcbio-nextgen

and re-run it should hopefully work correctly for all your test cases.
For the Toil run issue, this was a problem with the generated test CWL. If you get the latest from the git repository (https://github.com/bcbio/test_bcbio_cwl) it should correctly account for the missing bai index that is causing it to fail.
Hope this gets these running for you. I'll work more on the bcbio-base tag issue, which will require adjusting the Docker bulld process and also on making it easier to install the data directly. Thanks again for testing these.

Update: I created a versions.csv file with the following content (copied from the GRCh37 genome install):
seq,broad-20120813
twobit,broad-20120813
GA4GH_problem_regions,20181016
capture_regions,20161202
MIG,20150730
prioritize,20160215

but this didn't get the tests to run.
In the case of Cromwell:
[...]
[2018-11-24 18:25:29,01] [info] BackgroundConfigAsyncJobExecutionActor [8d7f1694prep_samples:0:1]: 'bcbio_nextgen.py' 'runfn' 'prep_samples' 'cwl' 'sentinel_runtime=cores,1,ram,2048' 'sentinel_parallel=multi-parallel' 'sentinel_outputs=rgnames__sample,config__algorithm__variant_regions,config__algorithm__variant_regions_merged,config__algorithm__variant_regions_orig,config__algorithm__coverage,config__algorithm__coverage_merged,config__algorithm__coverage_orig,config__algorithm__seq2c_bed_ready' 'sentinel_inputs=prep_samples_rec:record' 'run_number=0'
[2018-11-24 18:25:29,02] [info] BackgroundConfigAsyncJobExecutionActor [8d7f1694prep_samples:0:1]: executing: # make sure there is no preexisting Docker CID file
rm -f /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0/execution/docker_cid
# run as in the original configuration without --rm flag (will remove later)
docker run \
  --cidfile /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0/execution/docker_cid \
  -i \
   \
  --entrypoint /bin/bash \
  -v /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0:/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0 \
  quay.io/bcbio/bcbio-vc@sha256:7748650d81ab0b549172846b467f0281c15e523dbf977b2731bda20fad284711 /cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0/execution/script

# get the return code (working even if the container was detached)
rc=$(docker wait cat /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0/execution/docker_cid)

# remove the container after waiting
docker rm cat /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-prep_samples/shard-0/execution/docker_cid

# return exit code
exit $rc
[2018-11-24 18:25:32,54] [info] BackgroundConfigAsyncJobExecutionActor [8d7f1694prep_samples:0:1]: job id: 462
[2018-11-24 18:25:32,54] [info] BackgroundConfigAsyncJobExecutionActor [8d7f1694prep_samples:0:1]: Status change from - to WaitingForReturnCodeFile
[2018-11-24 18:25:35,64] [info] BackgroundConfigAsyncJobExecutionActor [8d7f1694prep_samples:0:1]: Status change from WaitingForReturnCodeFile to Done
[2018-11-24 18:25:37,96] [error] WorkflowManagerActor Workflow 8d7f1694-e3c1-441e-ada8-5ac4c5951bc0 failed (during ExecutingWorkflowState): cromwell.core.CromwellFatalException: java.lang.Exception: Failed command instantiation
	at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:46)
	at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:37)
	at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:413)
	at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: Failed command instantiation
	at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
	at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
	at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
	at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
	at scala.util.Try$.apply(Try.scala:209)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
	at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
	at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
	at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
	... 6 more
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
	:
	Could not localize /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-alignment_to_rec/inputs/66456483/6_100326_FC6107FAAXX.bam.bai:
	/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai doesn't exist
	File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-alignment_to_rec/inputs/66456483/6_100326_FC6107FAAXX.bam.bai -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai
	File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam.bai
	at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
	at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
	at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
	... 35 more

[2018-11-24 18:25:37,96] [info] WorkflowManagerActor WorkflowActor-8d7f1694-e3c1-441e-ada8-5ac4c5951bc0 is in a terminal state: WorkflowFailedState
[2018-11-24 18:26:11,37] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2018-11-24 18:26:12,80] [info] SingleWorkflowRunnerActor writing metadata to /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl-master/somatic/cromwell_work/somatic-metadata.json
[2018-11-24 18:26:12,84] [info] Workflow polling stopped
[2018-11-24 18:26:12,85] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-11-24 18:26:12,85] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-11-24 18:26:12,86] [info] Aborting all running workflows.
[2018-11-24 18:26:12,86] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-11-24 18:26:12,86] [info] JobExecutionTokenDispenser stopped
[2018-11-24 18:26:12,86] [info] WorkflowLogCopyRouter stopped
[2018-11-24 18:26:12,86] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-11-24 18:26:12,86] [info] WorkflowStoreActor stopped
[2018-11-24 18:26:12,86] [info] WorkflowManagerActor All workflows finished
[2018-11-24 18:26:12,86] [info] Connection pools shut down
[2018-11-24 18:26:12,86] [info] WorkflowManagerActor stopped
[2018-11-24 18:26:12,86] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-11-24 18:26:12,86] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-11-24 18:26:12,86] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-11-24 18:26:12,86] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-11-24 18:26:12,86] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-11-24 18:26:12,86] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-11-24 18:26:12,86] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-11-24 18:26:12,86] [info] CallCacheWriteActor stopped
[2018-11-24 18:26:12,86] [info] JobStoreActor stopped
[2018-11-24 18:26:12,87] [info] SubWorkflowStoreActor stopped
[2018-11-24 18:26:12,87] [info] IoProxy stopped
[2018-11-24 18:26:12,87] [info] DockerHashActor stopped
[2018-11-24 18:26:12,87] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2018-11-24 18:26:12,87] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-11-24 18:26:12,87] [info] ServiceRegistryActor stopped
[2018-11-24 18:26:12,88] [info] Database closed
[2018-11-24 18:26:12,88] [info] Stream materializer shut down
[2018-11-24 18:26:12,89] [info] WDL HTTP import resolver closed
Workflow 8d7f1694-e3c1-441e-ada8-5ac4c5951bc0 transitioned to state Failed
Failed bcbio Cromwell run
-------------------------
Failure in step: alignment_to_rec.0
  bcbio log file     : cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-alignment_to_rec/execution/log/bcbio-nextgen-debug.log
  bcbio commands file: cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-alignment_to_rec/execution/log/bcbio-nextgen-commands.log
  Cromwell directory : cromwell_work/cromwell-executions/main-somatic.cwl/8d7f1694-e3c1-441e-ada8-5ac4c5951bc0/call-alignment_to_rec

And for Toil:
[...]
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu        returnValues = self._run(jobGraph, fileStore)
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu        returnValues = self._run(jobGraph, fileStore)
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu        return self.run(fileStore)
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu        return self.run(fileStore)
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu        (output, status) = cwltool.main.single_job_executor(**opts)
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu        (output, status) = cwltool.main.single_job_executor(**opts)
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu        r.run(**kwargs)
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu        r.run(**kwargs)
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu        self._setup(kwargs)
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu        self._setup(kwargs)
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,012 MainThread WARNING toil.leader: 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
2018-11-24 18:32:06,012 - toil.leader - WARNING - 8/u/jobZPBjPu      File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,013 MainThread WARNING toil.leader: 8/u/jobZPBjPu        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
2018-11-24 18:32:06,013 - toil.leader - WARNING - 8/u/jobZPBjPu        "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,013 MainThread WARNING toil.leader: 8/u/jobZPBjPu    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv) not found or is not a regular file.
2018-11-24 18:32:06,013 - toil.leader - WARNING - 8/u/jobZPBjPu    WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv) not found or is not a regular file.
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,013 MainThread WARNING toil.leader: 8/u/jobZPBjPu    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
2018-11-24 18:32:06,013 - toil.leader - WARNING - 8/u/jobZPBjPu    ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,013 MainThread WARNING toil.leader: 8/u/jobZPBjPu    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu with ID 8/u/jobZPBjPu to 0
2018-11-24 18:32:06,013 - toil.leader - WARNING - 8/u/jobZPBjPu    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu with ID 8/u/jobZPBjPu to 0
amizeranschi-Predator-PH317-51 2018-11-24 18:32:06,014 MainThread WARNING toil.leader: Job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu with ID 8/u/jobZPBjPu is completely failed
2018-11-24 18:32:06,014 - toil.leader - WARNING - Job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu with ID 8/u/jobZPBjPu is completely failed
amizeranschi-Predator-PH317-51 2018-11-24 18:32:36,052 MainThread INFO toil.leader: No jobs left to run so exiting.
2018-11-24 18:32:36,052 - toil.leader - INFO - No jobs left to run so exiting.
amizeranschi-Predator-PH317-51 2018-11-24 18:32:36,052 MainThread INFO toil.leader: Finished the main loop
2018-11-24 18:32:36,052 - toil.leader - INFO - Finished the main loop
amizeranschi-Predator-PH317-51 2018-11-24 18:32:36,053 MainThread INFO toil.serviceManager: Waiting for service manager thread to finish ...
2018-11-24 18:32:36,053 - toil.serviceManager - INFO - Waiting for service manager thread to finish ...
amizeranschi-Predator-PH317-51 2018-11-24 18:32:36,745 MainThread INFO toil.serviceManager: ... finished shutting down the service manager. Took 0.692306041718 seconds
2018-11-24 18:32:36,745 - toil.serviceManager - INFO - ... finished shutting down the service manager. Took 0.692306041718 seconds
amizeranschi-Predator-PH317-51 2018-11-24 18:32:36,746 MainThread INFO toil.statsAndLogging: Waiting for stats and logging collator thread to finish ...
2018-11-24 18:32:36,746 - toil.statsAndLogging - INFO - Waiting for stats and logging collator thread to finish ...
amizeranschi-Predator-PH317-51 2018-11-24 18:32:37,254 MainThread INFO toil.statsAndLogging: ... finished collating stats and logs. Took 0.508193016052 seconds
2018-11-24 18:32:37,254 - toil.statsAndLogging - INFO - ... finished collating stats and logs. Took 0.508193016052 seconds
amizeranschi-Predator-PH317-51 2018-11-24 18:32:37,255 MainThread INFO toil.leader: Finished toil run with 18 failed jobs
2018-11-24 18:32:37,255 - toil.leader - INFO - Finished toil run with 18 failed jobs
amizeranschi-Predator-PH317-51 2018-11-24 18:32:37,258 MainThread INFO toil.leader: Failed jobs at end of the run: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/combine_sample_regions.cwl' bcbio_nextgen.py runfn combine_sample_regions cwl 7/M/jobgvGM1D 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/postprocess_alignment_to_rec.cwl' bcbio_nextgen.py runfn postprocess_alignment_to_rec cwl c/L/jobQ2hf2H 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu 'CWLScatter' K/I/jobOfJUCd 'CWLGather' V/R/jobgdDUnB 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/batch_for_variantcall.cwl' bcbio_nextgen.py runfn batch_for_variantcall cwl S/g/jobPabqDO 'CWLScatter' K/K/jobAxKazj 'CWLScatter' i/X/jobRIQRN6 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl 8/i/job6Ha2qc 'CWLScatter' k/M/jobCNmWhn 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl n/3/jobuSm68l 'CWLScatter' v/6/jobDs9Lxc 'CWLGather' 3/p/jobWjBJTL 'CWLGather' V/l/jobyP87Bu 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/summarize_vc.cwl' bcbio_nextgen.py runfn summarize_vc cwl r/U/jobLYCUiV 'CWLGather' m/P/jobd1Emh6 'CWLGather' m/H/jobwpmje7 'CWLWorkflow' 7/8/jobQHebBQ
2018-11-24 18:32:37,258 - toil.leader - INFO - Failed jobs at end of the run: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/combine_sample_regions.cwl' bcbio_nextgen.py runfn combine_sample_regions cwl 7/M/jobgvGM1D 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/postprocess_alignment_to_rec.cwl' bcbio_nextgen.py runfn postprocess_alignment_to_rec cwl c/L/jobQ2hf2H 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu 'CWLScatter' K/I/jobOfJUCd 'CWLGather' V/R/jobgdDUnB 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/batch_for_variantcall.cwl' bcbio_nextgen.py runfn batch_for_variantcall cwl S/g/jobPabqDO 'CWLScatter' K/K/jobAxKazj 'CWLScatter' i/X/jobRIQRN6 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl 8/i/job6Ha2qc 'CWLScatter' k/M/jobCNmWhn 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl n/3/jobuSm68l 'CWLScatter' v/6/jobDs9Lxc 'CWLGather' 3/p/jobWjBJTL 'CWLGather' V/l/jobyP87Bu 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/summarize_vc.cwl' bcbio_nextgen.py runfn summarize_vc cwl r/U/jobLYCUiV 'CWLGather' m/P/jobd1Emh6 'CWLGather' m/H/jobwpmje7 'CWLWorkflow' 7/8/jobQHebBQ
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/cwltoil", line 11, in <module>
    load_entry_point('toil==3.14.0', 'console_scripts', 'cwltoil')()
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 990, in main
    outobj = toil.start(wf1)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/common.py", line 743, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/common.py", line 1018, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/leader.py", line 226, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException: The job store 'file:/home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/toil_work/cwltoil_jobstore' contains 18 failed jobs: 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/combine_sample_regions.cwl' bcbio_nextgen.py runfn combine_sample_regions cwl 7/M/jobgvGM1D, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/postprocess_alignment_to_rec.cwl' bcbio_nextgen.py runfn postprocess_alignment_to_rec cwl c/L/jobQ2hf2H, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu, 'CWLScatter' K/I/jobOfJUCd, 'CWLGather' V/R/jobgdDUnB, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/batch_for_variantcall.cwl' bcbio_nextgen.py runfn batch_for_variantcall cwl S/g/jobPabqDO, 'CWLScatter' K/K/jobAxKazj, 'CWLScatter' i/X/jobRIQRN6, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/alignment_to_rec.cwl' bcbio_nextgen.py runfn alignment_to_rec cwl 8/i/job6Ha2qc, 'CWLScatter' k/M/jobCNmWhn, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/prep_samples_to_rec.cwl' bcbio_nextgen.py runfn prep_samples_to_rec cwl n/3/jobuSm68l, 'CWLScatter' v/6/jobDs9Lxc, 'CWLGather' 3/p/jobWjBJTL, 'CWLGather' V/l/jobyP87Bu, 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/summarize_vc.cwl' bcbio_nextgen.py runfn summarize_vc cwl r/U/jobLYCUiV, 'CWLGather' m/P/jobd1Emh6, 'CWLGather' m/H/jobwpmje7, 'CWLWorkflow' 7/8/jobQHebBQ
=========> Failed job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu
INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
WARNING:salad:unrecognized extension field `http://commonwl.org/cwltool#generation`.  Did you include a $schemas section?
foreign properties set([])
WARNING:salad:foreign properties set([])
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
    return self.run(fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
    (output, status) = cwltool.main.single_job_executor(**opts)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
    r.run(**kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
    self._setup(kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
    "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
WorkflowException: Input file file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv (at /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/testdata/genomes/hg19/versions.csv) not found or is not a regular file.
ERROR:toil.worker:Exiting the worker because of a failed job on host amizeranschi-Predator-PH317-51
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/steps/qc_to_rec.cwl' bcbio_nextgen.py runfn qc_to_rec cwl 8/u/jobZPBjPu with ID 8/u/jobZPBjPu to 0
<=========

Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin/bcbio_vm.py", line 353, in <module>
    args.func(args)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 294, in run
    _TOOLS[args.tool](args)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 121, in _run_toil
    _run_tool(cmd, not args.no_container, work_dir)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbio/cwl/tool.py", line 42, in _run_tool
    subprocess.check_call(cmd, shell=True)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'export PATH=/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin:$PATH && cwltoil --jobStore /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/toil_work/cwltoil_jobstore --logFile /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/toil_work/somatic-toil.log --workDir /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/toil_work/tmpdir --linkImports -- /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/main-somatic.cwl /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/somatic/somatic-workflow/main-somatic-samples.json' returned non-zero exit status 1

Thanks. The two CWL tests are running fine now.
I've also tested on yeast data. I ran into a problem with Toil where it tried to use more CPU cores (16) than what my system actually has available (8). This was due to the default setting (cores: 16) in bcbio_system.yaml.
I remember that the bcbio_nextgen installer automatically detected the CPU cores and available memory and set these up in the system yaml file. Is there a way to do this for bcbio_vm as well?
I tried adding the option -n $cores to bcbio_vm.py cwlrun, but it doesn't seem like this option is supported anymore.
After setting the right number cores (and amount of memory) in bcbio_system.yaml, the Toil run is failing due to problems with one of the intermediary VCF files:
Traceback (most recent call last):
  File "/usr/local/bin/bcbio_nextgen.py", line 227, in <module>
    runfn.process(kwargs["args"])
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/runfn.py", line 58, in process
    out = fn(fnargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 52, in wrapper
    return apply(f, *args, **kwargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 392, in run_jointvc
    return joint.run_jointvc(*args)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 65, in run_jointvc
    joint_out = square_batch_region(data, ready_region, [], [d["vrn_file"] for d in items], out_file)[0]
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 186, in square_batch_region
    merge_gvcfs(data, region, vrn_files, out_file)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 251, in merge_gvcfs
    vcfutils.merge_variant_files([_extract_variants_from_gvcf(f, region, out_file, data) for f in vrn_files],
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 264, in _extract_variants_from_gvcf
    do.run(cmd.format(**locals()), "Extract variants from gVCF %s %s" % (dd.get_sample_name(data), region))
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 23, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 103, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'set -o pipefail; bcftools view -r chrXVI:1-608222 /var/lib/cwl/stg99153a51-3eb9-4b80-b5df-e6c8e87189e8/testingVC.vcf.gz | extract_variants | bgzip -c > /var/spool/cwl/bcbiotx/tmpe2YbZz/testingVC-chrXVI_1-608222-varonly.vcf.gz
ERROR: unexpected number of fields in vcf record:
	line_no: 71
	line: chrXVI	7631	.	T	G	1	LowGQX;LowDepth;NoPassedVariantGTs	SNVHPOL=3;MQ=27	GT:GQ:GQX:DP:DPF:AD:ADF:ADR:SB:FT:PL:AF	.:.:.:0:0:0,0:0,0:0,0:0:LowGQX;LowDepth:.:0,0	0/1:22:1:2:0:1,1:0,1:1,0:0:LowGQX;LowDepth:27,0,27:0,0.5
' returned non-zero exit status 1
[job run_jointvc.cwl] Job error:
Error validating output record. * the `vrn_file_region` field is not valid because
    value is a list, expected File or null

It seems that the problem is with handling the multi-sample VCF file. Recall, I am testing joint variant calling using Strelka2 on two samples, using the YAML file posted above.
I've also tested Cromwell. Interestingly, it didn't fail due to the default CPU-core and memory settings from the bcbio_system.yaml. Does Cromwell automatically adjust to the system settings and ignore the values from bcbio_system.yaml?
However, Cromwell did run into the same VCF-related problems as Toil:
Check the content of stderr for potential additional information: /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/39c3beb2-47f0-4fb1-959a-0de7cfaec61d/call-jointcall/shard-0/wf-jointcall.cwl/808e4e98-9688-4256-a0a7-9727322452cb/call-run_jointvc/shard-24/execution/stderr.
 Traceback (most recent call last):
  File "/usr/local/bin/bcbio_nextgen.py", line 227, in <module>
    runfn.process(kwargs["args"])
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/runfn.py", line 58, in process
    out = fn(fnargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 52, in wrapper
    return apply(f, *args, **kwargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 392, in run_jointvc
    return joint.run_jointvc(*args)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 65, in run_jointvc
    joint_out = square_batch_region(data, ready_region, [], [d["vrn_file"] for d in items], out_file)[0]
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 186, in square_batch_region
    merge_gvcfs(data, region, vrn_files, out_file)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 251, in merge_gvcfs
    vcfutils.merge_variant_files([_extract_variants_from_gvcf(f, region, out_file, data) for f in vrn_files],
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 264, in _extract_variants_from_gvcf
    do.run(cmd.format(**locals()), "Extract variants from gVCF %s %s" % (dd.get_sample_name(data), region))
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 23, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 103, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'set -o pipefail; bcftools view -r chrXVI:1-608222 /cromwell-executions/main-testingVC-merged.cwl/39c3beb2-47f0-4fb1-959a-0de7cfaec61d/call-jointcall/shard-0/wf-jointcall.cwl/808e4e98-9688-4256-a0a7-9727322452cb/call-run_jointvc/shard-24/inputs/1823474195/testingVC.vcf.gz | extract_variants | bgzip -c > /cromwell-executions/main-testingVC-merged.cwl/39c3beb2-47f0-4fb1-959a-0de7cfaec61d/call-jointcall/shard-0/wf-jointcall.cwl/808e4e98-9688-4256-a0a7-9727322452cb/call-run_jointvc/shard-24/execution/bcbiotx/tmpQ0z6Ph/testingVC-chrXVI_1-608222-varonly.vcf.gz
ERROR: unexpected number of fields in vcf record:
	line_no: 71
	line: chrXVI	7631	.	T	G	1	LowGQX;LowDepth;NoPassedVariantGTs	SNVHPOL=3;MQ=27	GT:GQ:GQX:DP:DPF:AD:ADF:ADR:SB:FT:PL:AF	.:.:.:0:0:0,0:0,0:0,0:0:LowGQX;LowDepth:.:0,0	0/1:22:1:2:0:1,1:0,1:1,0:0:LowGQX;LowDepth:27,0,27:0,0.5
' returned non-zero exit status 1

I noticed that there's a CWL test called gvcf_joint on GitHub (https://github.com/bcbio/test_bcbio_cwl.git). I gave this a try, but it ended up looking for input files in /home/chapmanb/drive/work/cwl:
INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.14.0-b91dbf9bf6116879952f0a70f9a2fbbcae7e51b6.
WARNING:toil.fileStore:Starting job h/T/job2oIVgk/g/tmp88jBdX.tmp with less than 10% of disk space remaining.
Traceback (most recent call last):
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/worker.py", line 309, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1328, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/job.py", line 1273, in _run
    return self.run(fileStore)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/toil/cwl/cwltoil.py", line 462, in run
    (output, status) = cwltool.main.single_job_executor(**opts)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/main.py", line 298, in single_job_executor
    r.run(**kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 412, in run
    self._setup(kwargs)
  File "/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/lib/python2.7/site-packages/cwltool/job.py", line 158, in _setup
    "file." % (knownfile, self.pathmapper.mapper(knownfile)[0]))
WorkflowException: Input file file:///home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai (at /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa.fai) not found or is not a regular file.

Glad we're making progress and thank you for this detailed feedback and testing on joint calling with your workflow. I believe we're running into an issue with gvcftools during strelka2 joint calling: sequencing/gvcftools#9 Practically, strelka2 joint calling was pretty primitive and outdated and I've moved it to use the updated gvcfgenotyper (https://github.com/Illumina/gvcfgenotyper) approach the Illumina team has been developing. If you re-try the runs with the latest Docker container (1.1.2a-1ffac4b: https://quay.io/repository/bcbio/bcbio-vc?tab=tags) I hope it will work cleanly for you. Also, going forward all new Docker builds will pin to the version of bcbio-base so you'll avoid the issues you saw was pinning versions in CWL and getting bcbio-base:latest.
Finally, I updated the gvcf_joint test which is working cleanly for me now with Cromwell. I'm not sure what is happening with toil in your example as the missing file it's complaining about appears to be present in the repository. I hope the updated version will work cleanly for you.
Thanks again for all this work and debugging and hope this now runs cleanly for you.

Hi, thanks a lot for the update. I'm afraid I don't quite understand what you meant by "all new Docker builds will pin to the version of bcbio-base so you'll avoid the issues you saw was pinning versions in CWL and getting bcbio-base:latest". Will the bcbio-base:latest image still be needed from now on, or will it suffice to have e.g. bcbio_vc:1.1.2a-1ffac4b installed?
I removed all the Docker images from my system (both bcbio_vc and bcbio_base) and got the latest CWL tests from Git. The Cromwell test failed due to wrong input file paths, and then proceeded to download the Docker image bcbio-base:latest:
cromwell.core.CromwellFatalException: java.lang.Exception: Failed command instantiation
	at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:46)
	at cromwell.core.retry.Retry$$anonfun$withRetry$1.applyOrElse(Retry.scala:37)
	at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:413)
	at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37)
	at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
	at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
	at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
	at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
	at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.Exception: Failed command instantiation
	at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:556)
	at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand$(StandardAsyncExecutionActor.scala:491)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents(StandardAsyncExecutionActor.scala:308)
	at cromwell.backend.standard.StandardAsyncExecutionActor.commandScriptContents$(StandardAsyncExecutionActor.scala:307)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:141)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.writeScriptContents$(SharedFileSystemAsyncJobExecutionActor.scala:140)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:12)
	at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor.writeScriptContents$(BackgroundAsyncJobExecutionActor.scala:11)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute(SharedFileSystemAsyncJobExecutionActor.scala:124)
	at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor.execute$(SharedFileSystemAsyncJobExecutionActor.scala:121)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.standard.StandardAsyncExecutionActor.$anonfun$executeAsync$1(StandardAsyncExecutionActor.scala:619)
	at scala.util.Try$.apply(Try.scala:209)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync(StandardAsyncExecutionActor.scala:619)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeAsync$(StandardAsyncExecutionActor.scala:619)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover(StandardAsyncExecutionActor.scala:934)
	at cromwell.backend.standard.StandardAsyncExecutionActor.executeOrRecover$(StandardAsyncExecutionActor.scala:926)
	at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:193)
	at cromwell.backend.async.AsyncBackendJobExecutionActor.$anonfun$robustExecuteOrRecover$1(AsyncBackendJobExecutionActor.scala:65)
	at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
	at cromwell.core.retry.Retry$$anonfun$withRetry$1.$anonfun$applyOrElse$3(Retry.scala:44)
	at akka.pattern.FutureTimeoutSupport.liftedTree1$1(FutureTimeoutSupport.scala:26)
	at akka.pattern.FutureTimeoutSupport.$anonfun$after$1(FutureTimeoutSupport.scala:26)
	at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
	... 6 more
Caused by: common.exception.AggregatedMessageException: Error(s):
:
:
	:
	Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1635740323/7_100326_FC6107FAAXX.bam:
	/home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam doesn't exist
	File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1635740323/7_100326_FC6107FAAXX.bam -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam
	File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/7_100326_FC6107FAAXX.bam
	:
	Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1635740323/6_100326_FC6107FAAXX.bam:
	/home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam doesn't exist
	File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1635740323/6_100326_FC6107FAAXX.bam -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam
	File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/100326_FC6107FAAXX/6_100326_FC6107FAAXX.bam
:
	Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222581580/hg19.fa.amb:
	/home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb doesn't exist
	File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222581580/hg19.fa.amb -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb
	File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb
	Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222581580/hg19.fa.amb:
	/home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb doesn't exist
	File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222581580/hg19.fa.amb -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb
	File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/bwa/hg19.fa.amb
:
	Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1471069787/variant_regions-bam.bed:
	/home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed doesn't exist
	File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1471069787/variant_regions-bam.bed -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed
	File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed
	Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1471069787/variant_regions-bam.bed:
	/home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed doesn't exist
	File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/1471069787/variant_regions-bam.bed -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed
	File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/automated/variant_regions-bam.bed
:
	Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222597375/hg19.fa:
	/home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa doesn't exist
	File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222597375/hg19.fa -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa
	File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa
	Could not localize /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa -> /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222597375/hg19.fa:
	/home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa doesn't exist
	File not found /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/inputs/222597375/hg19.fa -> /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa
	File not found /home/chapmanb/drive/work/cwl/test_bcbio_cwl/testdata/genomes/hg19/seq/hg19.fa
	at common.validation.Validation$ValidationTry$.toTry$extension1(Validation.scala:68)
	at common.validation.Validation$ValidationTry$.toTry$extension0(Validation.scala:64)
	at cromwell.backend.standard.StandardAsyncExecutionActor.instantiatedCommand(StandardAsyncExecutionActor.scala:554)
	... 35 more

[2018-11-26 20:14:19,15] [info] WorkflowManagerActor WorkflowActor-0ca4b313-c127-45ac-ac0b-820d43943dcf is in a terminal state: WorkflowFailedState
[2018-11-26 20:14:32,93] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2018-11-26 20:14:36,09] [info] SingleWorkflowRunnerActor writing metadata to /home/amizeranschi/bcbio-stuff/bcbio-vm/test_bcbio_cwl/gvcf_joint/cromwell_work/gvcf-joint-metadata.json
[2018-11-26 20:14:36,12] [info] Workflow polling stopped
[2018-11-26 20:14:36,13] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-11-26 20:14:36,13] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-11-26 20:14:36,14] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-11-26 20:14:36,14] [info] Aborting all running workflows.
[2018-11-26 20:14:36,14] [info] JobExecutionTokenDispenser stopped
[2018-11-26 20:14:36,14] [info] WorkflowStoreActor stopped
[2018-11-26 20:14:36,14] [info] WorkflowLogCopyRouter stopped
[2018-11-26 20:14:36,14] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-11-26 20:14:36,14] [info] WorkflowManagerActor All workflows finished
[2018-11-26 20:14:36,14] [info] WorkflowManagerActor stopped
[2018-11-26 20:14:36,14] [info] Connection pools shut down
[2018-11-26 20:14:36,14] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-11-26 20:14:36,14] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-11-26 20:14:36,14] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-11-26 20:14:36,14] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-11-26 20:14:36,14] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-11-26 20:14:36,14] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-11-26 20:14:36,14] [info] SubWorkflowStoreActor stopped
[2018-11-26 20:14:36,14] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-11-26 20:14:36,14] [info] JobStoreActor stopped
[2018-11-26 20:14:36,14] [info] CallCacheWriteActor stopped
[2018-11-26 20:14:36,14] [info] DockerHashActor stopped
[2018-11-26 20:14:36,14] [info] IoProxy stopped
[2018-11-26 20:14:36,15] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2018-11-26 20:14:36,15] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-11-26 20:14:36,15] [info] ServiceRegistryActor stopped
[2018-11-26 20:14:36,16] [info] Database closed
[2018-11-26 20:14:36,16] [info] Stream materializer shut down
[2018-11-26 20:14:36,16] [info] WDL HTTP import resolver closed
Workflow 0ca4b313-c127-45ac-ac0b-820d43943dcf transitioned to state Failed
Unable to find image 'quay.io/bcbio/bcbio-base:latest' locally
latest: Pulling from bcbio/bcbio-base
7b8b6451c85f: Pull complete
ab4d1096d9ba: Pull complete
e6797d1788ac: Pull complete
e25c5c290bde: Pull complete
406293dca006: Pull complete
Digest: sha256:c397057e303fb5018091149e1c3c4167b8f6847e667e82d6e745a0a95d35093c
Status: Downloaded newer image for quay.io/bcbio/bcbio-base:latest
Failed bcbio Cromwell run
-------------------------
Failure in step: alignment_to_rec.0
  bcbio log file     : cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/execution/log/bcbio-nextgen-debug.log
  bcbio commands file: cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec/execution/log/bcbio-nextgen-commands.log
  Cromwell directory : cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-alignment_to_rec

Failure in step: prep_samples_to_rec.0
  bcbio log file     : cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-prep_samples_to_rec/execution/log/bcbio-nextgen-debug.log
  bcbio commands file: cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-prep_samples_to_rec/execution/log/bcbio-nextgen-commands.log
  Cromwell directory : cromwell_work/cromwell-executions/main-gvcf-joint.cwl/0ca4b313-c127-45ac-ac0b-820d43943dcf/call-prep_samples_to_rec

I then removed the bcbio-base:latest image and tried my tests using yeast data, running bcbio_vm.py cwl with the option --add-container-tag 1.1.2a-1ffac4b.
This ended up downloading both the bcbio-vc (with unknown tag) and bcbio-base images:
docker images -a
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
quay.io/bcbio/bcbio-vc     <none>              9c2199cf1749        2 hours ago         7.89GB
quay.io/bcbio/bcbio-base   latest              bdeb18f81e1b        3 hours ago         1.87GB
hello-world                latest              4ab4c602aa5e        2 months ago        1.84kB

Unfortunately, my test also failed. It seems that gvcfgenotyper isn't working properly:
*** Error in gvcfgenotyper': realloc(): invalid pointer: 0x00007fff6b0de6d8 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7ff4240977e5]
/lib/x86_64-linux-gnu/libc.so.6(realloc+0x348)[0x7ff4240a4a08]
gvcfgenotyper[0x46ec81]
gvcfgenotyper[0x41bdbe]
gvcfgenotyper[0x4253c3]
gvcfgenotyper[0x42574d]
gvcfgenotyper[0x425c76]
gvcfgenotyper[0x422e48]
gvcfgenotyper[0x4068c5]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ff424040830]
gvcfgenotyper[0x4099a1]
======= Memory map: ========
00400000-004f6000 r-xp 00000000 fd:00 15473242                           /usr/local/share/bcbio-nextgen/anaconda/bin/gvcfgenotyper
006f5000-006f7000 rw-p 000f5000 fd:00 15473242                           /usr/local/share/bcbio-nextgen/anaconda/bin/gvcfgenotyper
006f7000-006f8000 rw-p 00000000 00:00 0
01948000-019c1000 rw-p 00000000 00:00 0                                  [heap]
7ff420000000-7ff420021000 rw-p 00000000 00:00 0
7ff420021000-7ff424000000 ---p 00000000 00:00 0
7ff424020000-7ff4241e0000 r-xp 00000000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7ff4241e0000-7ff4243e0000 ---p 001c0000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7ff4243e0000-7ff4243e4000 r--p 001c0000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7ff4243e4000-7ff4243e6000 rw-p 001c4000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7ff4243e6000-7ff4243ea000 rw-p 00000000 00:00 0
7ff4243ea000-7ff4244f2000 r-xp 00000000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7ff4244f2000-7ff4246f1000 ---p 00108000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7ff4246f1000-7ff4246f2000 r--p 00107000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7ff4246f2000-7ff4246f3000 rw-p 00108000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7ff4246f3000-7ff42470b000 r-xp 00000000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7ff42470b000-7ff42490a000 ---p 00018000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7ff42490a000-7ff42490b000 r--p 00017000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7ff42490b000-7ff42490c000 rw-p 00018000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7ff42490c000-7ff424910000 rw-p 00000000 00:00 0
7ff424910000-7ff424926000 r-xp 00000000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7ff424926000-7ff424b25000 ---p 00016000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7ff424b25000-7ff424b26000 r--p 00015000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7ff424b26000-7ff424b27000 rw-p 00016000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7ff424b27000-7ff424b4d000 r-xp 00000000 fd:00 15343636                   /lib/x86_64-linux-gnu/ld-2.23.so
7ff424ba9000-7ff424bf0000 rw-p 00000000 00:00 0
7ff424bf0000-7ff424bf3000 r--p 00000000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7ff424bf3000-7ff424c00000 r-xp 00003000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7ff424c00000-7ff424c03000 r--p 00010000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7ff424c03000-7ff424c04000 r--p 00012000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7ff424c04000-7ff424c05000 rw-p 00013000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7ff424c05000-7ff424c06000 rw-p 00000000 00:00 0
7ff424c06000-7ff424c9a000 r--p 00000000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7ff424c9a000-7ff424cff000 r-xp 00094000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7ff424cff000-7ff424d36000 r--p 000f9000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7ff424d36000-7ff424d40000 r--p 0012f000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7ff424d40000-7ff424d44000 rw-p 00139000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7ff424d44000-7ff424d47000 rw-p 00000000 00:00 0
7ff424d4a000-7ff424d4c000 rw-p 00000000 00:00 0
7ff424d4c000-7ff424d4d000 r--p 00025000 fd:00 15343636                   /lib/x86_64-linux-gnu/ld-2.23.so
7ff424d4d000-7ff424d4e000 rw-p 00026000 fd:00 15343636                   /lib/x86_64-linux-gnu/ld-2.23.so
7ff424d4e000-7ff424d4f000 rw-p 00000000 00:00 0
7fff6b0bf000-7fff6b0e0000 rw-p 00000000 00:00 0                          [stack]
7fff6b1c2000-7fff6b1c5000 r--p 00000000 00:00 0                          [vvar]
7fff6b1c5000-7fff6b1c7000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
' returned non-zero exit status -6

Job run_jointvc:9:1 exited with return code 1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-9/execution/stderr.
 Traceback (most recent call last):
  File "/usr/local/bin/bcbio_nextgen.py", line 223, in <module>
    runfn.process(kwargs["args"])
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/runfn.py", line 58, in process
    out = fn(fnargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/utils.py", line 52, in wrapper
    return apply(f, *args, **kwargs)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py", line 396, in run_jointvc
    return joint.run_jointvc(*args)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 64, in run_jointvc
    joint_out = square_batch_region(data, ready_region, [], [d["vrn_file"] for d in items], out_file)[0]
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/joint.py", line 185, in square_batch_region
    strelka2.run_gvcfgenotyper(data, region, vrn_files, out_file)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/strelka2.py", line 343, in run_gvcfgenotyper
    for r in regions]
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/variation/strelka2.py", line 358, in _run_gvcfgenotyper
    do.run(cmd, "gvcfgenotyper: %s %s" % (dd.get_sample_name(data), region))
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 23, in run
    _do_run(cmd, checks, log_stdout, env=env)
  File "/usr/local/share/bcbio-nextgen/anaconda/lib/python2.7/site-packages/bcbio/provenance/do.py", line 103, in _do_run
    raise subprocess.CalledProcessError(exitcode, error_msg)
subprocess.CalledProcessError: Command 'gvcfgenotyper -f /cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-9/inputs/-1642066049/sacCer3.fa -l /cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-9/execution/bcbiotx/tmpjPGnlh/testingVC-strelka2-chrVII_609522-1090940-chrVII_618649_618650-inputs.txt -r chrVII:618649-618650 -O z -o /cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-9/execution/bcbiotx/tmpjPGnlh/testingVC-strelka2-chrVII_609522-1090940-chrVII_618649_618650.vcf.gz
Logging output to gvcfgenotyper.20181126_185656.80.log
*** Error in gvcfgenotyper': realloc(): invalid pointer: 0x00007ffda7dc0b48 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fc4230937e5]
/lib/x86_64-linux-gnu/libc.so.6(realloc+0x348)[0x7fc4230a0a08]
gvcfgenotyper[0x46ec81]
gvcfgenotyper[0x41bdbe]
gvcfgenotyper[0x4253c3]
gvcfgenotyper[0x42574d]
gvcfgenotyper[0x425c76]
gvcfgenotyper[0x422e48]
gvcfgenotyper[0x4068c5]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fc42303c830]
gvcfgenotyper[0x4099a1]
======= Memory map: ========
00400000-004f6000 r-xp 00000000 fd:00 15473242                           /usr/local/share/bcbio-nextgen/anaconda/bin/gvcfgenotyper
006f5000-006f7000 rw-p 000f5000 fd:00 15473242                           /usr/local/share/bcbio-nextgen/anaconda/bin/gvcfgenotyper
006f7000-006f8000 rw-p 00000000 00:00 0
01b3c000-01bb5000 rw-p 00000000 00:00 0                                  [heap]
7fc41c000000-7fc41c021000 rw-p 00000000 00:00 0
7fc41c021000-7fc420000000 ---p 00000000 00:00 0
7fc42301c000-7fc4231dc000 r-xp 00000000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7fc4231dc000-7fc4233dc000 ---p 001c0000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7fc4233dc000-7fc4233e0000 r--p 001c0000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7fc4233e0000-7fc4233e2000 rw-p 001c4000 fd:00 15343656                   /lib/x86_64-linux-gnu/libc-2.23.so
7fc4233e2000-7fc4233e6000 rw-p 00000000 00:00 0
7fc4233e6000-7fc4234ee000 r-xp 00000000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7fc4234ee000-7fc4236ed000 ---p 00108000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7fc4236ed000-7fc4236ee000 r--p 00107000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7fc4236ee000-7fc4236ef000 rw-p 00108000 fd:00 15343688                   /lib/x86_64-linux-gnu/libm-2.23.so
7fc4236ef000-7fc423707000 r-xp 00000000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7fc423707000-7fc423906000 ---p 00018000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7fc423906000-7fc423907000 r--p 00017000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7fc423907000-7fc423908000 rw-p 00018000 fd:00 15343724                   /lib/x86_64-linux-gnu/libpthread-2.23.so
7fc423908000-7fc42390c000 rw-p 00000000 00:00 0
7fc42390c000-7fc423922000 r-xp 00000000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7fc423922000-7fc423b21000 ---p 00016000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7fc423b21000-7fc423b22000 r--p 00015000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7fc423b22000-7fc423b23000 rw-p 00016000 fd:00 15361813                   /usr/local/share/bcbio-nextgen/anaconda/lib/libz.so.1.2.11
7fc423b23000-7fc423b49000 r-xp 00000000 fd:00 15343636                   /lib/x86_64-linux-gnu/ld-2.23.so
7fc423ba5000-7fc423bec000 rw-p 00000000 00:00 0
7fc423bec000-7fc423bef000 r--p 00000000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7fc423bef000-7fc423bfc000 r-xp 00003000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7fc423bfc000-7fc423bff000 r--p 00010000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7fc423bff000-7fc423c00000 r--p 00012000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7fc423c00000-7fc423c01000 rw-p 00013000 fd:00 15474523                   /usr/local/share/bcbio-nextgen/anaconda/lib/libgcc_s.so.1
7fc423c01000-7fc423c02000 rw-p 00000000 00:00 0
7fc423c02000-7fc423c96000 r--p 00000000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7fc423c96000-7fc423cfb000 r-xp 00094000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7fc423cfb000-7fc423d32000 r--p 000f9000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7fc423d32000-7fc423d3c000 r--p 0012f000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7fc423d3c000-7fc423d40000 rw-p 00139000 fd:00 15361701                   /usr/local/share/bcbio-nextgen/anaconda/lib/libstdc++.so.6.0.25
7fc423d40000-7fc423d43000 rw-p 00000000 00:00 0
7fc423d46000-7fc423d48000 rw-p 00000000 00:00 0
7fc423d48000-7fc423d49000 r--p 00025000 fd:00 15343636                   /lib/x86_64-linux-gnu/ld-2.23.so
7fc423d49000-7fc423d4a000 rw-p 00026000 fd:00 15343636                   /lib/x86_64-linux-gnu/ld-2.23.so
7fc423d4a000-7fc423d4b000 rw-p 00000000 00:00 0
7ffda7da2000-7ffda7dc3000 rw-p 00000000 00:00 0                          [stack]
7ffda7dd2000-7ffda7dd5000 r--p 00000000 00:00 0                          [vvar]
7ffda7dd5000-7ffda7dd7000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
' returned non-zero exit status -6

[2018-11-26 20:56:59,18] [info] WorkflowManagerActor WorkflowActor-20f07690-6792-4bf5-80af-52b937112b35 is in a terminal state: WorkflowFailedState
[2018-11-26 20:57:26,44] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2018-11-26 20:57:28,02] [info] SingleWorkflowRunnerActor writing metadata to /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-metadata.json
[2018-11-26 20:57:28,09] [info] Workflow polling stopped
[2018-11-26 20:57:28,10] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-11-26 20:57:28,10] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-11-26 20:57:28,10] [info] Aborting all running workflows.
[2018-11-26 20:57:28,10] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-11-26 20:57:28,10] [info] WorkflowStoreActor stopped
[2018-11-26 20:57:28,10] [info] JobExecutionTokenDispenser stopped
[2018-11-26 20:57:28,11] [info] WorkflowLogCopyRouter stopped
[2018-11-26 20:57:28,11] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-11-26 20:57:28,11] [info] WorkflowManagerActor All workflows finished
[2018-11-26 20:57:28,11] [info] WorkflowManagerActor stopped
[2018-11-26 20:57:28,11] [info] Connection pools shut down
[2018-11-26 20:57:28,11] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-11-26 20:57:28,11] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-11-26 20:57:28,11] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-11-26 20:57:28,11] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-11-26 20:57:28,11] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-11-26 20:57:28,11] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-11-26 20:57:28,11] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-11-26 20:57:28,11] [info] JobStoreActor stopped
[2018-11-26 20:57:28,11] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2018-11-26 20:57:28,11] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-11-26 20:57:28,11] [info] CallCacheWriteActor stopped
[2018-11-26 20:57:28,11] [info] DockerHashActor stopped
[2018-11-26 20:57:28,11] [info] IoProxy stopped
[2018-11-26 20:57:28,11] [info] ServiceRegistryActor stopped
[2018-11-26 20:57:28,11] [info] SubWorkflowStoreActor stopped
[2018-11-26 20:57:28,12] [info] Database closed
[2018-11-26 20:57:28,12] [info] Stream materializer shut down
[2018-11-26 20:57:28,13] [info] WDL HTTP import resolver closed
Workflow 20f07690-6792-4bf5-80af-52b937112b35 transitioned to state Failed
Unable to find image 'quay.io/bcbio/bcbio-base:latest' locally
latest: Pulling from bcbio/bcbio-base
7b8b6451c85f: Already exists
ab4d1096d9ba: Already exists
e6797d1788ac: Already exists
e25c5c290bde: Already exists
406293dca006: Already exists
Digest: sha256:c397057e303fb5018091149e1c3c4167b8f6847e667e82d6e745a0a95d35093c
Status: Downloaded newer image for quay.io/bcbio/bcbio-base:latest
Running: export PATH=/home/amizeranschi/bcbio-stuff/bcbio-vm/anaconda/bin:$PATH && cromwell -Xms1g -Xmx3g run --type CWL -Dconfig.file=/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/bcbio-cromwell.conf --metadata-output /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-metadata.json --options /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-options.json --inputs /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/main-testingVC-merged-samples.json /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/testingVC-merged-workflow/main-testingVC-merged.cwl 2>&1 | tee -a /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/testingVC-merged-cromwell.log
Failed bcbio Cromwell run
-------------------------
Failure in step: jointcall.0.subWorkflowMetadata.calls.run_jointvc.0
  bcbio log file     : cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-0/execution/log/bcbio-nextgen-debug.log
  bcbio commands file: cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-0/execution/log/bcbio-nextgen-commands.log
  Cromwell directory : cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-0
[...]

Thanks much for the updates and continued testing. I'll try to tackle all 3 of the issue from your last message:

For the Docker images, bcbio-base is still required/expected, but if you pin to a specific version it should pull that tag rather than latest now. So you'll expect bcbio-base and bcbio-vc with a specific tag if you request when generating the CWL.
For the gVCF joint call tests, I pushed a fix which I think will resolve this. I hadn't put relative links for the input files so they were referencing the build on my local machine. You can update and re-run and it should now work cleanly for you.
For the gvcfgenotyper issues to fix your actual analysis, we'll have to try and put together a test case as it seems like something in your inputs is uniquely causing it to fail. If you're able to share the input gVCFs for that region that cause the failure, I'm happy to try and help to put together a bug report and try to get it working for you.

Hope this all helps.

The latest joint call tests are running fine on my system now.
However, regarding my failed tests with yeast data, the error I pasted above doesn't seem to be related to a specific genomic region. There are many similar error blocks, related to all of the analyzed genomic regions, as far as I could tell. I've attached the full output here: https://gist.githubusercontent.com/amizeranschi/48204a751510880b20211cc5e7f07a9d/raw/abb1d685e17a13c6302a4da7cfae3a7b3985aca4/gistfile1.txt.
Unfortunately, I couldn't find any of the gVCF files in the temporary directories mentioned in the gvcfgenotyper errors. All of the temporary directories have been automatically deleted:
ls -la /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-*/execution/bcbiotx/
/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-0/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:53 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:53 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:53 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-10/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-11/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-12/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-13/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-14/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:53 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:53 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:53 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-15/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:53 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:53 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:53 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-16/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-17/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-18/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-19/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-1/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-20/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-21/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-22/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-23/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-24/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-25/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:54 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:54 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:54 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-2/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-3/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-4/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-5/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:53 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:53 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:53 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-6/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:53 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:53 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:53 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-7/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-8/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:55 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:55 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:55 .file

/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-9/execution/bcbiotx/:
total 8
drwxr-xr-x 2 amizeranschi root         4096 Nov 26 20:56 .
drwxrwxrwx 6 amizeranschi amizeranschi 4096 Nov 26 20:56 ..
-rw-r--r-- 1 amizeranschi root            0 Nov 26 20:56 .file

Thanks for clarifying the location of the input files. I've just sent them by e-mail. I hope they will be useful.

Thanks for passing these along; I appreciate the help debugging. Are these the input to one of the failing steps using gvcfgenotyper? The only input VCF there is not a gVCF but rather a combined and called VCF file with two samples, so I'm confused. I wouldn't expect you to have these if the gvcfgenotyper step failed earlier. Are these the input files from the path I referenced or a different one? Also, could you share your configuration for the actual run? Maybe I'm missing something in reproducing and understanding the issue. Thanks again for all the help.

Yes, those are the files from the directory you mentioned in your previous post (/home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged__failed-strelka2/work/cromwell_work/cromwell-executions/main-testingVC-merged.cwl/20f07690-6792-4bf5-80af-52b937112b35/call-jointcall/shard-0/wf-jointcall.cwl/680304fd-104e-4ab4-ae35-a692ffd75652/call-run_jointvc/shard-7/inputs/).
This is the YAML template:
# Template for whole genome Illumina variant calling with GATK pipeline
---
details:
  - analysis: variant2
    genome_build: sacCer3
    algorithm:
      aligner: bwa
      mark_duplicates: true
      recalibrate: false
      realign: false
      effects: false
      tools_off: gemini
      variantcaller: strelka2
      jointcaller: strelka2-joint
      svcaller: [manta]
      ploidy: 1
      variant_regions: ../config/variant_regions.bed

This is the CSV file:
samplename,description,batch
/home/amizeranschi/bcbio-stuff/automated-VC-test/fastq-files/sample1-R1-L001.fastq.gz,sample1,testingVC
/home/amizeranschi/bcbio-stuff/automated-VC-test/fastq-files/sample1-R1-L002.fastq.gz,sample1,testingVC
/home/amizeranschi/bcbio-stuff/automated-VC-test/fastq-files/sample1-R2-L001.fastq.gz,sample1,testingVC
/home/amizeranschi/bcbio-stuff/automated-VC-test/fastq-files/sample1-R2-L002.fastq.gz,sample1,testingVC
/home/amizeranschi/bcbio-stuff/automated-VC-test/fastq-files/sample2-R1-L001.fastq.gz,sample2,testingVC
/home/amizeranschi/bcbio-stuff/automated-VC-test/fastq-files/sample2-R1-L002.fastq.gz,sample2,testingVC
/home/amizeranschi/bcbio-stuff/automated-VC-test/fastq-files/sample2-R2-L001.fastq.gz,sample2,testingVC
/home/amizeranschi/bcbio-stuff/automated-VC-test/fastq-files/sample2-R2-L002.fastq.gz,sample2,testingVC

And this is the YAML config file:
details:
- algorithm:
    aligner: bwa
    effects: false
    jointcaller: strelka2-joint
    mark_duplicates: true
    ploidy: 1
    realign: false
    recalibrate: false
    svcaller:
    - manta
    tools_off: gemini
    variant_regions: ../config/variant_regions.bed
    variantcaller: strelka2
  analysis: variant2
  description: sample1
  files:
  - /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/sample1_R1.fastq.gz
  - /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/sample1_R2.fastq.gz
  genome_build: sacCer3
  metadata:
    batch: testingVC
- algorithm:
    aligner: bwa
    effects: false
    jointcaller: strelka2-joint
    mark_duplicates: true
    ploidy: 1
    realign: false
    recalibrate: false
    svcaller:
    - manta
    tools_off: gemini
    variant_regions: ../config/variant_regions.bed
    variantcaller: strelka2
  analysis: variant2
  description: sample2
  files:
  - /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/sample2_R1.fastq.gz
  - /home/amizeranschi/bcbio-stuff/automated-VC-test/testingVC-merged/sample2_R2.fastq.gz
  genome_build: sacCer3
  metadata:
    batch: testingVC
fc_name: testingVC-merged
upload:
  dir: ../final

I tried running the same test (Yeast data + CWL with Cromwell), but I've replaced Strelka2 with GATK's HaplotypeCaller. It's been running for almost 7 hours now and I'm not sure if it's still working or just stuck somehow.
The last lines of the output so far are:
# return exit code
exit $rc
[2018-11-30 12:00:56,75] [info] WorkflowExecutionActor-254691bd-cc5e-4c38-beeb-2f3f5f8b9a4f [254691bd]: Starting normalize_sv_coverage
[2018-11-30 12:00:57,23] [info] BackgroundConfigAsyncJobExecutionActor [30fc7c81variantcall_batch_region:0:1]: job id: 10055
[2018-11-30 12:00:57,24] [info] BackgroundConfigAsyncJobExecutionActor [30fc7c81variantcall_batch_region:0:1]: Status change from - to WaitingForReturnCodeFile

Is there a way to check if this is still running or just stuck?
For comparison, the same workflow (with Strelka2) took 20 minutes when running on a local Bcbio_nextgen copy, and around an hour when running with Bcbio_vm and Docker (no CWL). Strelka2 has run aprox. 2X faster than GATK's HaplotypeCaller for me, locally, in Bcbio_nextgen (tested a couple of weeks ago).
Is it expected for Bcbio_vm+Docker+CWL to run significantly slower than Bcbio_nextgen or the older Bcbio_vm+Docker workflow (now CWL)?

Thanks much for the example, this was a big help to identify the issue. If you add tools_on: [gvcf] to your configuration, it should now do the right thing. In CWL you can drop the jointcaller input approach and just add this for do gVCF based joint calling. This seemed like a cleaner approach as the dual variantcaller and jointcaller approach ended up being confusing.
However, the way you specified should work and not fail, so I've also pushed fixes to make that happen. Those should be in place by tomorrow but in the short term just removing jointcaller and adding tools_on: [gvcf] should proceed as you expect.
For the GATK issue, I'm not sure if this is a symptom of the same underlying problem or if something else is happening. You should be able to use top to check if there is anything running, which is the most straightforward way. In general, we don't expect the CWL approach to be significantly slower than the older approach so taking this much longer doesn't seem right. For smaller runs on a single machine CWL might be slower depending on the core allocation, but should be relatively similar.
Hope this all helps and gets your analysis finally finished. Thanks again for all the work and patience.

I've been checking htop and I can see lines showing that stuff is happening (e.g. /usr/bin/containerd or java lines running cromwell.jar). I'm just surprised at the surprisingly long runtime for this.
Just to check, should the jointcaller option only be taken out for CWL, or is it still useful for Bcbio_nextgen and Bcbio_vm+Docker?
I'll test this tomorrow and report back.

Adding the option tools_on: [gvcf] got Strelka2 to run fine for me, both with and without the jointcaller option. Furthermore, the wall time was approx. the same as the one I had with Bcbio_nextgen, which is great news (around 20 minutes). GATK's HaplotypeCaller also ran fine (only tested with tools_on: gvcf and also jointcaller: gatk-haplotype-joint) and its wall time was around 30 minutes (50% slower than Strelka2).
Is there any good reason to still use HaplotypeCaller instead of Strelka2?
One thing I noticed was that the upload directory seems to have changed when using CWL, compared with what I've expected from Bcbio_nextgen. This is what the directory structure looks like for my test:
├── testingVC-merged
│   ├── config
│   ├── final
│   ├── work
│   │   ├── cromwell_work
│   │   │   ├── cromwell-executions
│   │   │   ├── cromwell-workflow-logs
│   │   │   ├── final
│   │   │   │   ├── project
│   │   │   │   ├── sample1
│   │   │   │   ├── sample2
│   │   │   ├── persist
│   │   ├── testingVC-merged-workflow

To run both Bcbio_nextgen and Bcbio_vm+CWL, I cd into testingVC-merged/work and run either
bcbio_nextgen.py ../config/<config>.yaml -n <cores>

or
bcbio_vm.py cwlrun cromwell testingVC-merged-workflow

Bcbio_nextgen seems to set the upload directory to testingVC-merged/final. However, Bcbio_vm+CWL copies files to testingVC-merged/work/cromwell_work/final.
In addition, the name of the directory where the joint VCF file and multiqc results are copied differs between Bcbio_nextgen and Bcbio_vm+CWL. With the former, that directory was called something like <run_date>_<exp_name>, e.g. 2018-11-12_testingVC-merged in my example. Meanwhile, for Bcbio_vm+CWL, this is simply called project.
I also noticed a file called testingVC-manta.vcf.gz inside testingVC-merged/work/cromwell_work/final/project. Is this meant to be a multi-sample a joint-VCF file for sv calls, similarly to what's done for small variants called by Strelka2? The file was empty for my test case, (as I never bothered to set up a realistic test case for large variants) and I only wanted to see if Manta actually ran successfully.
WIth Bcbio_nextgen, I expected Manta VCFs to be in the testingVC-merged/final/sample1 and testingVC-merged/final/sample1 directories. With Bcbio_vm+CWL, I see that all the VCF files are in testingVC-merged/work/cromwell_work/final/project and only the BAM files and other metadata are in testingVC-merged/work/cromwell_work/final/sample1 and testingVC-merged/work/cromwell_work/final/sample2.
What's the plan with the two workflows (Bcbio_nextgen and Bcbio_vm+CWL)? Are they going to diverge in terms of directory set-up and output file locations? Is Bcbio_vm+CWL going to replace Bcbio_nextgen in the future?

It's brilliant to hear this is now working for you. Thanks for all the patience getting it running and for the usage report. In terms of GATK HaplotypeCaller and strelka2, both are good choices for joint variant calling. They do have different intermediate gVCF formats so you might also make a decision based on other platforms or analyses you want to integrate with.
Thanks also for the detailed investigation of the output directory. It's expected to be non-identical for the CWL case. Replicating some of what we did previously was difficult since we get the results in a different manner from the CWL runner so can't allocate to sample and project directories in the exact same way. We were also hoping to clean up some of the historical cruft and confusion in the upload directory. Please let us know if anything is problematic and we can investigate.
For manta, you're right that is should be a merged VCF of the batched samples. It's not joint called, but rather called together within a group. Hope that works as expected when you're able to put together a realistic case for it.
Thanks again for all the feedback and work on getting this going.

OK, great. Thanks for all your help!

authors: amizeranschi chapmanb

repo: bcbio/bcbio-nextgen