Release Notes: This release fixes a few bugs and provides several improvements in efficiency and performance. The Similarity detection effectiveness for similarity based near-exact deduplication has been improved. At the same time memory requirements for the index has been reduced. Accuracy of data partitioning between threads has been improved. Chunking and indexing performance have been improved and the KMV Sketch computation is now more accurate. This release moves all the core functionality into a shared library in preparation for an API interface that will be introduced in future releases.
Release Notes: This release adds many bugfixes and performance improvements. Accuracy in finding duplicates in Global Dedupe has been improved. SHA256 is now the default block hash algorithm for dedupe, with the ability to change it separately from the chunk verification hash. Overall, many performance improvements have been made, with better parallelism, more SSE vectorization, and faster sorting and improving the segment hash list file handling, resulting in smaller I/O and fewer random accesses. Bugs in calculating in-memory index size has been fixed to avoid overflowing free RAM and swapping to disk.
Release Notes: This is a performance-focused release with improvements across the board. Extensive x86 SSE2/3/4/AVX vectorization has been done with runtime CPU feature detection. Deduplication performance is increased 3X. The Delta Compression algorithm has been tweaked for better performance and effectiveness with reduced memory usage. xxHash has been vectorized. Support for BLAKE2 checksum has been included. AES CTR mode has been vectorized. Intel's optimized SHA512, 512/256 is included for leading edge SHA2 performance in addition to parallel modes. LZMA performance is slightly boosted.