Jenkins, retest this please

Jenkins, test this please

@b-wyss Looks like you have some PEP8 violations. make autopep8 should clear them up.

Keep on going, you've made good progress!

Scripts to update:

• normalize-by-median (multiple files);
• count-median;
• extract-partitions;
• extract-long-sequences;
• extract-paired-reads;
• fastq-to-fasta;
• filter-abund;
• filter-abund-single;
• sample-reads-randomly;
• split paired-reads;
• interleave-reads;
• trim-low-abund;

Looking at http://khmer.readthedocs.org/en/v1.3/user/scripts.html, I think:

filter-abund and filter-abund-single;
count-median;
extract-partitions;
extract-long-sequences;
extract-paired-reads;
fastq-to-fasta
interleave-reads
sample-reads-randomly
split-paired-reads

and eventually trim-low-abund when #759 is merged :)

@ctb First pass is for single outputs ( -o or similar)

@b-wyss Mind if I vulture this?

Go for it

On Tue, Jul 21, 2015 at 10:52 AM Jake Fenton notifications@github.com
wrote:

@b-wyss https://github.com/b-wyss Mind if I vulture this?

—
Reply to this email directly or view it on GitHub
#747 (comment).

Michael R. Crusoe: Programmer & Bioinformatician crusoe@ucdavis.edu
The lab for Data Intensive Biology; University of California, Davis
https://impactstory.org/MichaelRCrusoe http://twitter.com/biocrusoe

Please do! Sorry I fell off the face of the earth, things got really crazy
towards the end of the spring, centered around a bit of a family situation.
I meant to officially leave and try to make it on good terms, but it never
really happened. You guys deserved to at least know what was going on with
me, and I didn't let anyone know. Seems like the lab has transferred over
pretty successfully - congratulations!

On Tue, Jul 21, 2015 at 1:57 PM Michael R. Crusoe notifications@github.com
wrote:

Go for it

On Tue, Jul 21, 2015 at 10:52 AM Jake Fenton notifications@github.com
wrote:

@b-wyss https://github.com/b-wyss Mind if I vulture this?

—
Reply to this email directly or view it on GitHub
#747 (comment).

Michael R. Crusoe: Programmer & Bioinformatician crusoe@ucdavis.edu
The lab for Data Intensive Biology; University of California, Davis
https://impactstory.org/MichaelRCrusoe http://twitter.com/biocrusoe

—
Reply to this email directly or view it on GitHub
#747 (comment).

All good! Stuff happens. Hope everything is okay and you're doing well!

TO-DO:

• Generalize enable compression: currently it takes args and converts args.output to a gz/bz writer. as per @ctb reccomendation, will shift to a func that takes a file handle and returns an FP. This will enable use in scripts a la normalize-by-median with multiple files.
• Enable compression in normalize-by-median
  • determine which outputs in normalize-by-median should be compressed
• figure out what happens if we do -o - --gzip in normalize-by-median (probably nothing good)

question: should the func that adds the compression args be moved to khmer_args?

https://github.com/dib-lab/khmer/pull/747/files#diff-46d242f8c7f4ca46fd5c784bd24184c7R165

normalize-by-median's stdout 3 test is leaking output

(It's my doing, I just need to remember to fi x it)

"revert" to argparse handling opening files:

• normalize-by-median (multiple files);
• count-median;
• extract-partitions;
• extract-long-sequences;
• extract-paired-reads;
• fastq-to-fasta;
• filter-abund;
• filter-abund-single;
• sample-reads-randomly;
• split paired-reads;
• interleave-reads;
• trim-low-abund;

split-paired-reads has this:

parser.add_argument('infile', nargs='?', default='/dev/stdin')

we should make than an argparse handled file open

retest this, please

Good question. No for now, but file an issue.

Titus Brown, ctbrown@ucdavis.edu

On Jul 30, 2015, at 11:22 AM, Jake Fenton notifications@github.com wrote:

question re trim-low-abund: we set aside stuff in a file, then go back to that file, pull things from it and then write to output. Currently, I don't compress the aside_file--Should I?

—
Reply to this email directly or view it on GitHub.

Right. Everything is in place except for more extensive test coverage.

Probably gonna make a test_output_compression.py or summuch

I would suggest adding streaming tests to the new test_streaming_io.

Titus Brown, ctbrown@ucdavis.edu

On Jul 30, 2015, at 11:31 AM, Jake Fenton notifications@github.com wrote:

Right. Everything is in place except for more extensive test coverage.

Probably gonna make a test_output_compression.py or summuch

—
Reply to this email directly or view it on GitHub.

Agreed, but my concern is that for nearly every case where we have an output file we could have a compressed output file, and we should be testing all of those--but that's like, 50 tests.

from high-bandwidth conversation; We'll employ stupidity-driven-testing here. No point in going through the combinatorial matrix (Well, there is a point, but the cost/benefit isn't there). So if somebody finds something that's super borked we'll fix it and add a test for it but as we stand we're probably on solid footing.

On that note, @ctb Merge?

• The number of changed files and the number of files mentioned in the ChangeLog entry are different;
  what's up?
• Is is_block really only used in one place in the codebase? Is there nowhere else it belongs?
• More generally, in the 'Wrote output to" section of fastq-to-fasta - is there a need for a general way to
  provide a human-readable output name for things that may-or-may-not-be-files? I would prefer that
  logic like if output_is_block be placed somewhere central and reusable (kfile.py?) - perhaps a
  function named 'describe_filehandle` or some such that returns a string?

Overall looks good. Double-check your diff-cover, fix issues above, ask for re-review :)

@ctb Cleaned up, ready for merge.

LGTM.

Thanks!