This was cut from the dynamic_fmt.c file, and placed here.

I will STILL comment known issues, while I am working on the, within

the dynamic_fmt.c file. But once done, they will get moved here.

* Generic MD5 hashes cracker

*
* It uses the openSSL md5 for x86, and MMX/SSE2 md5 from md5-mmx.S
*
* TODO's
* All done (right now)!

*
* recently DONE's
* 1. in the MMX/SSE2 builds, ALSO make sure native x86 (using OpenSSL), is
built.
* Then if the format is a NON-SSE safe format, link the functions to the
non-x86 versions.
* In x86 mode, we DO NOT need to have the SSE2 code, since it is not valid
to call anyway.
* 2. Add SSE Intrisic code. MD5_PARA (as built by simon), IS now working.
* 3. Start making 'thin' _fmt.c files for other 'formats'.
* Easy formats:
* DMD5 (Is a challenge/response, so may need to simply be
'redone'
* MD5_apache_fmt.c (DONE)
* md5_fmt.c (DONE)
* HDAA_fmt.c (DONE)
* pixMD5_fmt.c (DONE)
* raw_md5_fmt.c (DONE)
* PO_fmt.c (DONE)
* phpassMD5_fmt.c (DONE)
* PHPS_fmt.c (DONE)
* IPB2_fmt.c (DONE)
* 4. Made 10 primitives which allow us to switch back and forth from SSE (1
MD5 buffer), to X86, multiple buffers. Thus, we
* can write SSE formats, and when they overflow, switch to x86, perform the
MD5, and switch back. A good example is type
* dynamic_13 md5(md5($p).md5($s)) Here, we preload $s, so it does not
play a part in the runtime. However, the
* first md5($p) CAN be done in SSE mode. Then we switch to x86, and perform
the MD5 of the 64 byte string. This increases
* throughput about 150%.
* 5. Found and fixed a bug in the MMX string concatentation code.
* 6. Generated a 'test-suite' format.
* 7. Add in 'real' utf8 to unicode conversion. Added MGF_UTF8, and set
FMT_UTF8 and FMT_UNICODE into the format's structure
* if this flag (or the uncode switch function), is used.
* 8. Put some ability into the formats, so that test strings can be switched
out in utf8 mode,
* and so that any 8859-1 which would 'fail' in utf8 mode could be switched
out. In the preloads, U= and A= are used at the
* start of the cipher strings. In the john.conf, there are now Test= TestA=
and TestU= for always use, using only if non-utf8
* and use ONLY if utf8. The TestA= and TestU= were documented. A= and U=
in the preload.c file are internal, and need
* NO documentation to end users.
* 9. Added raw-md5-unicode.
 * 10. max salt len not working (not a fixed size, just max len, i.e. saltlen =
neg number).
* 11. upcase of user name (currently only up/lo case we do), is not handled
with code found in unicode.c.
* 12. Allow upcasing and low casing of strings (like passwords). Do this using
unicode
* upcasing from unicode.c
* 13. Fix x86-64 'broken' formats. Right now, they 'work' but only with SSE
turned off.
* The broken formats are 12/13/21-26
*
* Only valid up to 54 bytes max string length (salts, rehashes,
* passwords, etc) if using SSE2. 96 byte keys (appended salts,
* keys, re-hashes, etc), if running under x86 mode. NOTE some
* hashes will NOT run properly under SSE2. A hash such as
* md5(md5($p).md5($s)) would fail under SSE2, since it always
* would need at least 64 bytes to complete but even md5($p) would
* fail in SSE2 if the password is over 54 bytes. NOTE no run-time
* checks are made so if you provide data too large, it will not find
* the hash, and will 'overwrite' some of the hashes being worked on,
* and cause them to have invalid results. This is a bug that 'might'
* be worked on, but I do not want do slow the cracking down performing
* checks.
*
* This code has gone through a few iterations, and now is quite a bit
* more mature. It has been designed with an array for keys (which
* is optionally used), a slot for the current salt, 2 arrays for
* input buffers (there is optional loading that loads keys directly
* into input buffer #1 as an optimization for certain formats), and
* a pair of arrays for crypt outputs. The 'first' output buffer array
* is used to return the final results. There is also 2 arrays of lengths
* of input buffers. There are then 'primitive' functions. These can
* append keys, append salts, blank out keys, move from input 1 to input
* 2, crypt input 1 -> output 1, (or from 1->2 or 2->2 or 2->1, etc).
* There are functions that do base 16 conversions of the outputs back
* into inputs (O1->I1 in base 16, 1->2 2->2 2->1, etc). There are
* functions that over write the start of an input buffer from outputs
* without 'adjusting' the lengths. There are a few special functions
* to do phpass work.
*
* Then there are helper functions which allow another format to 'use'
* the generic MD5 code. So, we can make a VERY thin raw-md5 (or phpass
* md5), where it simply has a format structure (which does not need to be
* 'heavily' filled out, and that format only needs to implement a few
* functions on its own. It would need to implement init, valid, salt
* and binary. Then there needs to be a 'conversion' function that
* converts from the 'native' format, into the native GENERIC format.
* Then, within the init function, that format would hook into the
* generic md5, by calling the dynamic_RESET_LINK() function, passing
* in its Format structure to have functions pointed into the md5 generic
* stuff. The conversion function is likely very trivial. For phpass, we
* convert from
* $H$9aaaaaSXBjgypwqm.JsMssPLiS8YQ00
* to
* $dynamic_17$jgypwqm.JsMssPLiS8YQ00$9aaaaaSXB
*
* Here is that convert function:
* static char *Convert(char *Buf, char *ciphertext) {
* sprintf(Buf, "$dynamic_17$%s%10.10s", &ciphertext[3+8+1], &ciphertext[2]);
* return Buf;
* }
*
*
* Generic MD5 can now be user expanded. The first 1000 dynamic_# are
* reserved as 'built-in' functions for john. Above 1000 is free to use
* for anyone wanting to do so. NO programming changes are needed to
* add a format. All that is needed is modifcations to john.conf. There is
* FULL documentation about how to do this in doc/DYNAMIC. There is
* no parser 'generation' logic. A person would have to understand the
* primitive functions and how they work. But the format can be added
* without a rebuild of john. There are 7 (or 8) examples already done
* in john.conf at this time, which should make it pretty easy for someone
* wanting to do a new or obscure format.
*
* Recent additions / fixes
*
* SSE2 intrinsics
* Faster (double speed) for a couple formats
* Addition of PO format (Native and using constants)
* Addition of 8 'constant' fields'
* In MD5_COEF builds, the not-SSE-safe formats now work (using x86 code)
* MD5_go functions used (10% faster than OpenSSL).
*
* Renamed and changed from md5_gen* to dynamic*. We handle MD5 and SHA1
* at the present time. More crypt types 'may' be added later.
*
* Spring 2012:
* Added many new crypts. SHA2 (sha224,256,384,512), GOST, WHIRLPOOL
*
* Summer 2012:
* Added (or changed) the larger hash output methods. Original code ONLY
* allowed lowercase base16 writing. Now, after the crypt, the overwrite
* or append output can be done in: base-16, base-16-uc, base-64 (MIME)
* base-64 with no trailing '=' chars, or raw. Adding new output 'types'
* should be easy to do. All of the original primitive dynamic functions
* have been kept (depricated), and new functions were added. There are
* a few output format setters also added. The default output format
* (at the start of crypt_all) is base-16, which is what it was before.
* however, the new code allows that to be changed, within a format.
*
* Winter 2013
* Added OMP support. Each thread gets a group of candidats, and walks
* each script step on its set of candidates. Thus many function had to
* be changed, to work from start to end range, instead of from 0 to count.
* Non omp still runs from 0 to count.
* added 4 'standard' functions for each large hash. Found some SERIOUS
* bugs in the original large hash functions. Only appending base-16
* was working. The overwrite, and things like upcase, raw, base64, etc
* were NON functional. The new functions test some of these items.
*
* Spring 2013
* Added many additional large formats: Whirlpool for older oSSL versions.
* Tiger, RIPEMD (sizes 128, 160, 256, 320). The RIPEMD256/320 were in a
* JimF update to the ripemd_plug.c (SPH library source only did 128 and 160)
*