Java release for 9.6.1, JVM crash when read Blob file, opening as secondary.

[eric2877]

Expected behavior

primary db opened by a C kernel，a secondary opening was operated by JVM，usually JVM crashed when read values in a blob file. Two different Errors were thrown like below:

Current thread (0x00007f9b3002ef40): JavaThread "reactor-http-epoll-3" daemon [_thread_in_native, id=363308, >stack(0x00007f9b7fe00000,0x00007f9b7ff00000)]

Stack: [0x00007f9b7fe00000,0x00007f9b7ff00000], sp=0x00007f9b7fefc1d0, free space=1008k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni16124523717842761313.so+0x80cb7d] rocksdb::GetSstInternalUniqueId(std::string const&, std::string const&, unsigned long, rocksdb::UniqueIdPtr, bool)+0x15d
C [librocksdbjni16124523717842761313.so+0x3791e2] rocksdb::OffsetableCacheKey::OffsetableCacheKey(std::string const&, std::string const&, unsigned long)+0x32
C [librocksdbjni16124523717842761313.so+0x3c4fbd] rocksdb::BlobSource::GetBlob(rocksdb::ReadOptions const&, rocksdb::Slice const&, unsigned long, unsigned long, unsigned long, unsigned long, rocksdb::CompressionType, rocksdb::FilePrefetchBuffer*, rocksdb::PinnableSlice*, unsigned long*)+0x4d
C [librocksdbjni16124523717842761313.so+0x5d60f6] rocksdb::Version::GetBlob(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::BlobIndex const&, rocksdb::FilePrefetchBuffer*, rocksdb::PinnableSlice*, unsigned long*) const+0x1a6
C [librocksdbjni16124523717842761313.so+0x5e06e8] rocksdb::Version::GetBlob(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::FilePrefetchBuffer*, rocksdb::PinnableSlice*, unsigned long*) const+0xc8
C [librocksdbjni16124523717842761313.so+0x50a1ff] rocksdb::DBIter::SetBlobValueIfNeeded(rocksdb::Slice const&, rocksdb::Slice const&)+0x12f
C [librocksdbjni16124523717842761313.so+0x50ef40] rocksdb::DBIter::FindNextUserEntryInternal(bool, rocksdb::Slice const*)+0xe00
C [librocksdbjni16124523717842761313.so+0x50f0a1] rocksdb::DBIter::FindNextUserEntry(bool, rocksdb::Slice const*)+0x91
C [librocksdbjni16124523717842761313.so+0x50fd9f] rocksdb::DBIter::Next()+0x58f
J 6248 org.rocksdb.RocksIterator.next0Jni(J)V [email protected] (0 bytes) @ 0x00007f9bf535465c [0x00007f9bf53545a0+0x00000000000000bc]
J 6247 c1 org.rocksdb.RocksIterator.next0(J)V [email protected] (5 bytes) @ 0x00007f9bee412bbc [0x00007f9bee412b40+0x000000000000007c]
J 6246 c1 org.rocksdb.AbstractRocksIterator.next()V [email protected] (30 bytes) @ 0x00007f9bee5ca55c [0x00007f9bee5ca440+0x000000000000011c]

the other one:

Stack: [0x00007ff436bd8000,0x00007ff436cd8000], sp=0x00007ff436cd3fd0, free space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni5483512105604404713.so+0x85032c]
C [librocksdbjni5483512105604404713.so+0x854855] ROCKSDB_XXH3_128bits_withSeed+0x4d5
C [librocksdbjni5483512105604404713.so+0x832772] rocksdb::Hash2x64(char const*, unsigned long, unsigned long, unsigned long*, unsigned long*)+0x12
C [librocksdbjni5483512105604404713.so+0x80cb86] rocksdb::GetSstInternalUniqueId(std::string const&, std::string const&, unsigned long, rocksdb::UniqueIdPtr, bool)+0x166
C [librocksdbjni5483512105604404713.so+0x3791e2] rocksdb::OffsetableCacheKey::OffsetableCacheKey(std::string const&, std::string const&, unsigned long)+0x32
C [librocksdbjni5483512105604404713.so+0x3c4fbd] rocksdb::BlobSource::GetBlob(rocksdb::ReadOptions const&, rocksdb::Slice const&, unsigned long, unsigned long, unsigned long, unsigned long, rocksdb::CompressionType, rocksdb::FilePrefetchBuffer*, rocksdb::PinnableSlice*, unsigned long*)+0x4d
C [librocksdbjni5483512105604404713.so+0x5d60f6] rocksdb::Version::GetBlob(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::BlobIndex const&, rocksdb::FilePrefetchBuffer*, rocksdb::PinnableSlice*, unsigned long*) const+0x1a6
C [librocksdbjni5483512105604404713.so+0x5e06e8] rocksdb::Version::GetBlob(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::FilePrefetchBuffer*, rocksdb::PinnableSlice*, unsigned long*) const+0xc8
C [librocksdbjni5483512105604404713.so+0x50a1ff] rocksdb::DBIter::SetBlobValueIfNeeded(rocksdb::Slice const&, rocksdb::Slice const&)+0x12f
C [librocksdbjni5483512105604404713.so+0x50ef40] rocksdb::DBIter::FindNextUserEntryInternal(bool, rocksdb::Slice const*)+0xe00
C [librocksdbjni5483512105604404713.so+0x50f0a1] rocksdb::DBIter::FindNextUserEntry(bool, rocksdb::Slice const*)+0x91
C [librocksdbjni5483512105604404713.so+0x50fd9f] rocksdb::DBIter::Next()+0x58f
j org.rocksdb.RocksIterator.next0Jni(J)V+0 [email protected]
j org.rocksdb.RocksIterator.next0(J)V+1 [email protected]
j org.rocksdb.AbstractRocksIterator.next()V+26 [email protected]

Secondary DB Configuration

 final  DBOptions options = new DBOptions();
        options.setMaxOpenFiles(-1);
        return RocksDB.openAsSecondary(options, db_path, ConstLimitData.DATA_SECOND_BASE, columnFamilyDescriptors,
                                         columnFamilyHandles);

Addtion

It works well，if the values were not in BLOB file. What strange is that a very small probability JVM can read values in Blob file correctly. So is there any configuration can fix it?