Using HMMER2 models with HMMER3: don’t do that.

Thanks to our panopticon-like ability to scan the Internets for people benchmarking the HMMER3 alpha release even though we told you not to, it’s come to the attention of our Watchers that people are doing benchmarks using existing HMMER2 files (Pfam, SMART), taking advantage of HMMER3’s ability to read or convert HMMER2 format profile HMM files. There is something important you need to know, if you’re doing this.

The hmmbuild program is responsible for parameterization; and parameterization is what determines most of the sensitivity/specificity of a profile HMM.

The profile search programs (hmmsearch, hmmscan) take an already parameterized profile HMM as input. The search program is what determines the speed of a profile HMM search.

HMMER3 search programs (and HMMER3’s hmmconvert program) can read HMMER2 format profile HMM files, but all that means is that HMMER3 can read the file format. It doesn’t mean it can convert the parameterization. If you run a HMMER3 search program with a HMMER2 profile HMM file, all you’re getting is a close approximation of HMMER2 scoring, using HMMER3’s computational speed.

If you want to benchmark HMMER3 sensitivity and specificity, you absolutely must use HMMER3’s hmmbuild and create new profiles from multiple alignments. While it is trivial to merely interconvert H2 and H3 file formats, it is not possible to convert a H2 profile parameterization to an H3 parameterization. If you do run a benchmark of H3 using H2-generated profiles, please do not be surprised to see “performance” close to H2’s sensitivity/specificity performance — and possibly even a little degraded, because all you’re doing is a fast emulation of H2.

If you report such a benchmark as a benchmark of HMMER3 performance, we will send the Watchers to collect your corpse. They know where you are.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s