Thanks to our panopticon-like ability to scan the Internets for people benchmarking the HMMER3 alpha release even though we told you not to, it's come to the attention of our Watchers that people are doing benchmarks using existing HMMER2 files (Pfam, SMART), taking advantage of HMMER3's ability to read or convert HMMER2 format profile HMM files. There is something important you need to know, if you're doing this.
The hmmbuild program is responsible for parameterization; and parameterization is what determines most of the sensitivity/specificity of a profile HMM.
The profile search programs (hmmsearch, hmmscan) take an already parameterized profile HMM as input. The search program is what determines the speed of a profile HMM search.
HMMER3 search programs (and HMMER3's hmmconvert program) can read HMMER2 format profile HMM files, but all that means is that HMMER3 can read the file format. It doesn't mean it can convert the parameterization. If you run a HMMER3 search program with a HMMER2 profile HMM file, all you're getting is a close approximation of HMMER2 scoring, using HMMER3's computational speed.
If you want to benchmark HMMER3 sensitivity and specificity, you absolutely must use HMMER3's hmmbuild and create new profiles from multiple alignments. While it is trivial to merely interconvert H2 and H3 file formats, it is not possible to convert a H2 profile parameterization to an H3 parameterization. If you do run a benchmark of H3 using H2-generated profiles, please do not be surprised to see "performance" close to H2's sensitivity/specificity performance -- and possibly even a little degraded, because all you're doing is a fast emulation of H2.
If you report such a benchmark as a benchmark of HMMER3 performance, we will send the Watchers to collect your corpse. They know where you are.