Storing / retrieving binary data

Von: Ben Engbers via BaseX-Talk <basex-talk at mailman.uni-konstanz.de> Gesendet: Dienstag, 25. August 2026 23:46 An: Basex Mail-lijst Betreff: [basex-talk] Storing / retrieving binary data Hi, My code for the C++ version of libBasexCpp has been largely rewritten after I received the necessary suggestions for improvements from Google Gemini and Mistral. The current code has ultimately become much ‘neater’ and therefore easier to maintain. At the moment, I’m writing a C wrapper around the C++ functions. The last point of uncertainty I’ve come across in the documentation so far relates to storing and retrieving binary data. The documentation for the client/server protocol contains the following sentence: “{...}: UTF-8 strings or raw data, suffixed with a \00 byte. To avoid confusion with this end-of-string byte, all transferred \00 and \FF bytes are prefixed by an additional \FF byte". For \00-terminated strings, it is clear how the \FF byte should be handled. However, when dealing with the 'Binary get' or 'Putbinary' commands, to me it is unclear what the specifications for ‘binary’ data are, or what the sources of binary data are. Are binary data provided as a binary file? Are binary files read ‘as-is’, or are they encoded by adding a \FF byte? Is an extra \00 byte added? Ben

Op 26-08-2026 om 14:34 schreef Christian Grün via BaseX-Talk: > Hi Ben, > > There is no separate binary format: the escaping is the whole payload format. Every payload, in both directions and for every command, is the same byte stream: each \00 and \FF byte gets an extra \FF prefix, and a single unescaped \00 ends the stream. > > For BINARY PUT (0x0D): send the code byte, the path as a \00-terminated string, then the file’s bytes as-is (no charset conversion, no base64) escaped byte by byte, followed by the terminating \00. BINARY GET is just an ordinary database command; read its result until you hit an unescaped \00, dropping every \FF that precedes a \00 or \FF. > > There is no length prefix, so both directions have to be streamed until the terminator. See ServerOutput (encode) and ServerInput (decode) in basex-core. > > Hope this helps, > Christian

Von: Ben Engbers <ben.engbers at gmail.com> Gesendet: Mittwoch, 26. August 2026 22:13 An: Christian Grün; Basex Mail-lijst; Ben Engbers Betreff: Re: [basex-talk] Re: Storing / retrieving binary data Hi Christian, Thank you for the explanation; this makes things a lot easier. Compared to the standard ‘Command’ commands, I have currently made an exception for command("Binary get <bin>"). This exception is therefore no longer necessary. Years ago, whilst developing the library, I was once faced with a situation where the input contained 0x00 bytes. To prevent this from causing errors again, I have since added 0xFF to all instances of 0x00 and 0xFF in the input. I therefore no longer need to make an exception for ‘PutBinary’. This means that, apart from code byte 0x0D, there is no difference between ‘Command’ and ‘PutBinary’. What, then, is the rationale behind defining a separate ‘PutBinary’ function? Ben Op 26-08-2026 om 14:34 schreef Christian Grün via BaseX-Talk: > Hi Ben, > > There is no separate binary format: the escaping is the whole payload format. Every payload, in both directions and for every command, is the same byte stream: each \00 and \FF byte gets an extra \FF prefix, and a single unescaped \00 ends the stream. > > For BINARY PUT (0x0D): send the code byte, the path as a \00-terminated string, then the file’s bytes as-is (no charset conversion, no base64) escaped byte by byte, followed by the terminating \00. BINARY GET is just an ordinary database command; read its result until you hit an unescaped \00, dropping every \FF that precedes a \00 or \FF. > > There is no length prefix, so both directions have to be streamed until the terminator. See ServerOutput (encode) and ServerInput (decode) in basex-core. > > Hope this helps, > Christian