Dockerfile needed

[AlexanderPavlenko]

FROM golang:1.14-stretch
RUN apt-get update && apt-get install -y \
      build-essential git wget pkg-config libzmq3-dev libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev liblz4-dev
RUN git clone --depth 1 https://github.com/facebook/rocksdb.git /rocksdb; \
    cd /rocksdb; CFLAGS=-fPIC CXXFLAGS=-fPIC make release
RUN git clone --depth 1 https://github.com/nbcorg/blockbook.git /blockbook; \
    cd /blockbook; CGO_CFLAGS="-I/rocksdb/include" CGO_LDFLAGS="-L/rocksdb -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4" go build

This attempt failed to go build:

go: downloading github.com/juju/errors v0.0.0-20170703010042-c7d06af17c68
go: downloading github.com/bsm/go-vlq v0.0.0-20150828105119-ec6e8d4f5f4e
go: downloading github.com/nbcorg/btcutil v0.0.0-20201106055106-dcdcc4d0007a
go: downloading github.com/prometheus/client_golang v1.1.0
go: downloading github.com/pebbe/zmq4 v1.0.0
go: downloading github.com/nbcorg/btcd v0.0.0-20200622133330-8ba93b4a5029
go: downloading github.com/tecbot/gorocksdb v0.0.0-20180907100951-214b6b7bc0f0
go: downloading github.com/prometheus/common v0.9.1
go: downloading github.com/gobuffalo/packr v1.13.7
go: downloading github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f
go: downloading github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
go: downloading github.com/beorn7/perks v1.0.1
go: downloading golang.org/x/crypto v0.0.0-20201016220609-9e8e0b390897
go: downloading github.com/martinboehm/golang-socketio v0.0.0-20180414165752-f60b0a8befde
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading github.com/golang/protobuf v1.3.2
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading github.com/prometheus/procfs v0.0.10
go: downloading github.com/gogo/protobuf v1.1.1
go: downloading github.com/pkg/errors v0.8.1
go: downloading github.com/gorilla/websocket v1.4.2
# github.com/tecbot/gorocksdb
/rocksdb/librocksdb.a(env_posix.o): In function `rocksdb::(anonymous namespace)::PosixDynamicLibrary::~PosixDynamicLibrary()':
/rocksdb/env/env_posix.cc:103: undefined reference to `dlclose'
/rocksdb/librocksdb.a(env_posix.o): In function `~PosixDynamicLibrary':
/rocksdb/env/env_posix.cc:103: undefined reference to `dlclose'
/rocksdb/env/env_posix.cc:103: undefined reference to `dlclose'
/rocksdb/librocksdb.a(env_posix.o): In function `rocksdb::(anonymous namespace)::PosixDynamicLibrary::LoadSymbol(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void**)':
/rocksdb/env/env_posix.cc:107: undefined reference to `dlerror'
/rocksdb/env/env_posix.cc:108: undefined reference to `dlsym'
/rocksdb/env/env_posix.cc:112: undefined reference to `dlerror'
/rocksdb/librocksdb.a(env_posix.o): In function `rocksdb::(anonymous namespace)::PosixEnv::LoadLibrary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<rocksdb::DynamicLibrary>*)':
/rocksdb/env/env_posix.cc:168: undefined reference to `dlopen'
/rocksdb/env/env_posix.cc:206: undefined reference to `dlerror'
/rocksdb/env/env_posix.cc:185: undefined reference to `dlopen'
/rocksdb/env/env_posix.cc:196: undefined reference to `dlopen'
collect2: error: ld returned 1 exit status

[CryptoManiac]

-ldl

[CryptoManiac]

trezor/blockbook#441 (comment)

[AlexanderPavlenko]

Thanks, build works.
Noticed some connectivity issue, is this a problem?

blockbook_1  | I1223 16:36:22.620955       1 mq.go:60] MQ listening to tcp://0.0.0.0:10000

# docker-compose exec russianbitcoind bash
root@ca0dcb686fcf:/# telnet blockbook 10000
Trying 172.18.0.3...
telnet: Unable to connect to remote host: Connection refused

russianbitcoind_1  | 2020-12-23T16:27:43Z Config file arg: zmqpubhashblock="tcp://blockbook:10000"
russianbitcoind_1  | 2020-12-23T16:27:43Z Config file arg: zmqpubhashtx="tcp://blockbook:10000"
russianbitcoind_1  | 2020-12-23T16:27:43Z zmq: Outbound message high water mark for pubhashblock at tcp://blockbook:10000 is 1000
russianbitcoind_1  | 2020-12-23T16:27:43Z zmq: Notifier pubhashblock failed (address = tcp://blockbook:10000)
russianbitcoind_1  | 2020-12-23T16:27:43Z zmq: Shutdown notifier pubhashblock at tcp://blockbook:10000
russianbitcoind_1  | 2020-12-23T16:27:43Z zmq: Shutdown notifier pubhashtx at tcp://blockbook:10000

[CryptoManiac]

Yes. MQ interface must be available for the daemon, or you will not see actual state of the blockchain and there always will be a delay.

[CryptoManiac]

Does it work for 127.0.0.1?

[AlexanderPavlenko]

ss -ln inside the container does not show port 10000 as listening despite the MQ listening log message.

[CryptoManiac]

Looks like it's not able to recognize 0.0.0.0 as endpoint IP address. Probably, it would be simpler to bind it at 127.0.0.1 and then redirect traffic from desired interface.

[AlexanderPavlenko]

ss -ln inside the container does not show port 10000 as listening despite the MQ listening log message.

Same holds for "message_queue_binding": "tcp://127.0.0.1:10000".

[CryptoManiac]

Is this possible that port 10000 is used by some other application? Maybe this app is not really listening but merely holding this port for itself (i.e. called bind() without subsequent listen() call).

P.S. From ss -ln output on current production node:

tcp    LISTEN  0       100                                       127.0.0.1:10000                0.0.0.0:*

[AlexanderPavlenko]

Build not working:

go: downloading github.com/juju/errors v0.0.0-20170703010042-c7d06af17c68
go: downloading github.com/prometheus/client_golang v1.1.0
go: downloading github.com/martinboehm/golang-socketio v0.0.0-20180414165752-f60b0a8befde
go: downloading github.com/nbcorg/btcutil v0.0.0-20201106055106-dcdcc4d0007a
go: downloading github.com/nbcorg/btcd v0.0.0-20200622133330-8ba93b4a5029
go: downloading github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
go: downloading github.com/bsm/go-vlq v0.0.0-20150828105119-ec6e8d4f5f4e
go: downloading github.com/gorilla/websocket v1.4.2
go: downloading github.com/gobuffalo/packr v1.13.7
go: downloading github.com/tecbot/gorocksdb v0.0.0-20180907100951-214b6b7bc0f0
go: downloading github.com/golang/protobuf v1.3.2
go: downloading github.com/pebbe/zmq4 v1.0.0
go: downloading golang.org/x/crypto v0.0.0-20201016220609-9e8e0b390897
go: downloading github.com/gogo/protobuf v1.1.1
go: downloading github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading github.com/prometheus/common v0.9.1
go: downloading github.com/prometheus/procfs v0.0.10
go: downloading github.com/pkg/errors v0.8.1
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
# github.com/tecbot/gorocksdb
/go/pkg/mod/github.com/tecbot/gorocksdb@v0.0.0-20180907100951-214b6b7bc0f0/db.go:552:29: not enough arguments in call to _Cfunc_rocksdb_approximate_sizes
	have (*_Ctype_struct_rocksdb_t, _Ctype_int, **_Ctype_char, *_Ctype_ulong, **_Ctype_char, *_Ctype_ulong, *_Ctype_ulong)
	want (*_Ctype_struct_rocksdb_t, _Ctype_int, **_Ctype_char, *_Ctype_ulong, **_Ctype_char, *_Ctype_ulong, *_Ctype_ulong, **_Ctype_char)
/go/pkg/mod/github.com/tecbot/gorocksdb@v0.0.0-20180907100951-214b6b7bc0f0/db.go:586:32: not enough arguments in call to _Cfunc_rocksdb_approximate_sizes_cf
	have (*_Ctype_struct_rocksdb_t, *_Ctype_struct_rocksdb_column_family_handle_t, _Ctype_int, **_Ctype_char, *_Ctype_ulong, **_Ctype_char, *_Ctype_ulong, *_Ctype_ulong)
	want (*_Ctype_struct_rocksdb_t, *_Ctype_struct_rocksdb_column_family_handle_t, _Ctype_int, **_Ctype_char, *_Ctype_ulong, **_Ctype_char, *_Ctype_ulong, *_Ctype_ulong, **_Ctype_char)
ERROR: Service 'blockbook' failed to build : The command '/bin/sh -c cd $GOPATH/src; git clone --depth 1 https://github.com/nbcorg/blockbook.git /blockbook; cd /blockbook;     CGO_CFLAGS="-I/rocksdb/include" CGO_LDFLAGS="-L/rocksdb -lrocksdb -lstdc++ -lm -lz -ldl -lbz2 -lsnappy -llz4" go build' returned a non-zero code: 2

[AlexanderPavlenko]

I pinned rocksdb to v6.14.6 and build succeeded, but the resulting binary fails to run: Illegal instruction (core dumped). Maybe there is some flag to increase binary portability? Building on Github Actions, running on x86_64 GNU/Linux.

[CryptoManiac]

It's quite expected. By default gcc uses instruction set which may not be supported on a target machine. You can add something like -mtune=generic to CFLAGS to override this behaviour but that will create slower code.

[CryptoManiac]

It's likely that your building machine has AVX support while target machine doesn't. It is the most common case. Simplest option without using generic instructions set would be to disable AVX support.

-mno-avx -mno-avx2

But there is a possibility that target platform lacks some other instructions. So, there is no guarantee that disabling AVX would be helpful for your case.

[CryptoManiac]

Maybe the best option would be to generate few images to ensure that all used machines will be running optimal code. Otherwise you will have no option but to sacrifice some performance to achieve portability.