log: fix potential overflow with long log messages (#490)

qb_vsnprintf_serialize was called with 'max_size' as the
limiting number for the length of the formatted log
message. But the buffer also needs to contain the
log header (given by 'actual_size'), so we now pass
't->max_line_length' as the maximum length of the
formatted log message to limit space to the actual
bytes left

Also added error checks to the blackbox calls at
the end of the test, as these now provide a proper
test that the BB is functioning. Before they were
masking failures.