Native bindings to xxhash.

xxHash is an extremely fast Hash algorithm, running at RAM speed limits. It also successfully passes all tests from the SMHasher suite.

A 64-bit version, named XXH64, is available since r35. It offers much better speed, but for 64-bit applications only.

Streaming

Streaming functions generate the xxHash value from an incremental input. This method is slower than single-call functions, due to state management. For small inputs, prefer 32 and 64, which are better optimized.

XXH state must first be allocated, using 32_createState.

Start a new hash by initializing state with a seed, using 32_reset.

Then, feed the hash state by calling 32_update as many times as necessary. Obviously, input must be allocated and read accessible. The function returns an error code, with 0 meaning OK, and any other value meaning there is an error.

Finally, a hash value can be produced anytime, by using 32_digest. This function returns the 32-bits hash as an int.

It's still possible to continue inserting input into the hash state after a digest, and generate some new hash values later on, by calling again 32_digest.

When done, release the state, using 32_freeState.

Example code for incrementally hashing a file:

#include <stdio.h>
#include <xxhash.h>
#define BUFFER_SIZE 256

// Note: XXH64 and XXH3 use the same interface.
XXH32_hash_t
hashFile(FILE* stream)
{
    XXH32_state_t* state;
    unsigned char buf[BUFFER_SIZE];
    size_t amt;
    XXH32_hash_t hash;

    state = XXH32_createState();       // Create a state
    assert(state != NULL);             // Error check here
    XXH32_reset(state, 0xbaad5eed);    // Reset state with our seed
    while ((amt = fread(buf, 1, sizeof(buf), stream)) != 0) {
        XXH32_update(state, buf, amt); // Hash the file in chunks
    }
    hash = XXH32_digest(state);        // Finalize the hash
    XXH32_freeState(state);            // Clean up
    return hash;
}

Canonical representation

The default return values from XXH functions are unsigned 32 and 64 bit integers. This the simplest and fastest format for further post-processing.

However, this leaves open the question of what is the order on the byte level, since little and big endian conventions will store the same number differently.

The canonical representation settles this issue by mandating big-endian convention, the same convention as human-readable numbers (large digits first).

When writing hash values to storage, sending them over a network, or printing them, it's highly recommended to use the canonical representation to ensure portability across a wider range of systems, present and future.

XXH3

XXH3 is a more recent hash algorithm featuring:

• Improved speed for both small and large inputs
• True 64-bit and 128-bit outputs
• SIMD acceleration
• Improved 32-bit viability

Speed analysis methodology is explained here:

https://fastcompression.blogspot.com/2019/03/presenting-xxh3.html

Compared to XXH64, expect XXH3 to run approximately ~2x faster on large inputs and >3x faster on small ones, exact differences vary depending on platform.

XXH3's speed benefits greatly from SIMD and 64-bit arithmetic, but does not require it. Most 32-bit and 64-bit targets that can run XXH32 smoothly can run XXH3 at competitive speeds, even without vector support. Further details are explained in the implementation.

Optimized implementations are provided for AVX512, AVX2, SSE2, NEON, POWER8, ZVector and scalar targets. This can be controlled via the XXH_VECTOR macro.

XXH3 implementation is portable:

• it has a generic C90 formulation that can be compiled on any platform,
• all implementations generage exactly the same hash value on all platforms.
• Starting from v0.8.0, it's also labelled "stable", meaning that any future version will also generate the same hash value.

XXH3 offers 2 variants, _64bits and _128bits. When only 64 bits are needed, prefer invoking the _64bits variant, as it reduces the amount of mixing, resulting in faster speed on small inputs. It's also generally simpler to manipulate a scalar return type than a struct.

The API supports one-shot hashing, streaming mode, and custom secrets.

*_withSecretandSeed()

These variants generate hash values using either seed for "short" keys (< XXH3_MIDSIZE_MAX = 240 bytes) or secret for "large" keys (≥ XXH3_MIDSIZE_MAX).

This generally benefits speed, compared to _withSeed() or _withSecret(). _withSeed() has to generate the secret on the fly for "large" keys. It's fast, but can be perceptible for "not so large" keys (< 1 KB). _withSecret() has to generate the masks on the fly for "small" keys, which requires more instructions than _withSeed() variants. Therefore, _withSecretandSeed variant combines the best of both worlds.

When secret has been generated by 3_generateSecret_fromSeed, this variant produces exactly the same results as _withSeed()` variant, hence offering only a pure speed benefit on "large" input, by skipping the need to regenerate the secret for every large input.

Another usage scenario is to hash the secret to a 64-bit hash value, for example with 3_64bits, which then becomes the seed, and then employ both the seed and the secret in _withSecretandSeed(). On top of speed, an added benefit is that each bit in the secret has a 50% chance to swap each bit in the output, via its impact to the seed. This is not guaranteed when using the secret directly in "small data" scenarios, because only portions of the secret are employed for small data.