Buy Percona ServicesBuy Now!

ZSTD compression support for Percona Server for MongoDB 3.4 (working patch)

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • ZSTD compression support for Percona Server for MongoDB 3.4 (working patch)

    Hi,

    I've been able to enable support for facebooks (amazing) Zstandard compression in Percona MongoDB, by simply modifying the wiredtiger SConscript file (using apt-get source percona-server-mongodb-34).

    I did various benchmarks (with that patch) that have proven that ZSTD offers a way better compression/performance ratio, than snappy or zlib (see https://github.com/facebook/zstd for more details)

    ​​​​​​Looking forward to see it default supported in Percona Server for MongoDB :-)


    Code:
    --- percona-server-mongodb-34-3.4.4.orig/src/third_party/wiredtiger/SConstruct
    +++ percona-server-mongodb-34-3.4.4/src/third_party/wiredtiger/SConstruct
    @@ -114,6 +114,7 @@ env['STATIC_AND_SHARED_OBJECTS_ARE_THE_S
    
     useZlib = GetOption("zlib")
     useSnappy = GetOption("snappy")
    +useZstd = true
     useLz4 = GetOption("lz4")
     useBdb = GetOption("bdb")
     useTcmalloc = GetOption("tcmalloc")
    @@ -144,6 +145,15 @@ if useSnappy:
             print 'snappy-c.h must be installed!'
             Exit(1)
    
    +if useZstd:
    +    if conf.CheckCHeader('zstd.h'):
    +       env.Append(CPPDEFINES=['HAVE_BUILTIN_EXTENSION_ZSTD'])
    +        wtlibs.append("zstd")
    +    else:
    +        print 'zstd.h must be installed!'
    +        Exit(1)
    +
    +
     if useLz4:
         conf.env.Append(CPPPATH=[useLz4 + "/include"])
         conf.env.Append(LIBPATH=[useLz4 + "/lib"])
    @@ -270,6 +280,9 @@ if useZlib:
     if useSnappy:
         wtsources.append("ext/compressors/snappy/snappy_compress.c")
    
    +if useZstd:
    +    wtsources.append("ext/compressors/zstd/zstd_compress.c")
    +
     if useLz4:
         wtsources.append("ext/compressors/lz4/lz4_compress.c")
    To compile (tested on Ubuntu) you need to install: libzstd-dev libzstd1


    Since zstd is not native supported in MongoDB, only directly via wiredtiger you need to use:

    Code:
    db.createCollection( "myNewCol", { storageEngine: { wiredTiger: { configString: "block_compressor=zstd" }}})
    or

    Code:
    wiredTiger:
      engineConfig:
        cacheSizeGB: 1
        checkpointSizeMB: 1000
        statisticsLogDelaySecs: 0
        journalCompressor: none
        directoryForIndexes: false
      collectionConfig:
        blockCompressor: zlib
        configString: "block_compressor=zstd"
      indexConfig:
        prefixCompression: true
        configString: "block_compressor=zstd"
    optional you can also set "compression_level" which is defautl set to "3" (I've not yet tested)
    Last edited by mipa; 05-12-2017, 05:22 AM.

  • #2
    Hi Mipa,

    Is there any way I could convince you to make a pull request on our GITHUB (https://github.com/percona/percona-server-mongodb) and file a PSMDB-JIRA (https://jira.percona.com/projects/PSMDB/issues) ticket? If the performance and simplicity prove out, this might be a good option for both WT & Mongo Rocks engines.


    Cheers
    David Murphy

    Comment


    • #3
      Hi David,

      thanks for the fast reply.

      There are already two other pending pull requests since January, and one is also regarding ZSTD. I think you guys will find a way better/cleaner way to enable zstd for wiredtiger than my patch, and maybe even support using it as native blockCompressor parameter from mongodb.conf, instead of wiredtiger hack ;-)

      If you first merge the pending ZSTD static pull request, you can even include it static :-)

      Thx,
      Michael

      Comment

      Working...
      X