[EXTERNAL] - Re: Re. Encoding support for the BaseX server

On Thu, Sep 3, 2020 at 4:43 PM Sharath Babu Pulumati <spulumat at opentext.com> wrote: > Hi Christian, > > The detailed steps are below, > > 1. Installed the BaseX and started the BaseX server, with default > configuration. > 2. Using the xqj client, for communicating with the basex sever. > 3. Using db::replace(), I have tried pushing and pulling of the > document in basex. > > > Can we have a quick call at any time, to discuss on this issue. > > Thanks & Regards, > Sharath. > ------------------------------ > *From:* Christian Grün <christian.gruen at gmail.com> > *Sent:* Thursday, September 3, 2020 5:29 PM > *To:* Sharath Babu Pulumati <spulumat at opentext.com> > *Cc:* James Ball <basex-talk at jamesball.co.uk>; BaseX < > basex-talk at mailman.uni-konstanz.de> > *Subject:* [EXTERNAL] - Re: Re. Encoding support for the BaseX server > > Hi Sharath, > > It's difficult to tell what might go wrong in your setup. Could you please > provide us with a step-by-step procedure that allows us to reproduce the > problem? > > Thanks in advance, > Christian > > > > Sharath Babu Pulumati <spulumat at opentext.com> schrieb am Do., 3. Sep. > 2020, 13:55: > > Hi, > > I'm facing an issue with the basex when trying to import documents, > > 1. With specified and un-specified encoding values. > 2. Importing the document which are exported from basex server. > > Getting the below error when trying to import the exported document from > basex server with below content: > > *Imported Document Content to BaseX: * > <?xml version="1.0" encoding="ISO-8859-1"?> > <!-- Edited by XMLSpy® --> > <root> > <name>mahesh</name> > </root> > > *Received Document Content from BaseX:* > <!-- Edited by XMLSpy® --> > <root> > <name>mahesh</name> > </root> > > > *NOTE: Data loss of encoding is occurred. * > > > *Error:* Invalid byte 1 of 1-byte UTF-8 sequence. > *Stack trace:* > 2020-09-03 16:33:35,911 INFO [stdout] (default task-3) 16:33:35,895 DEBUG > [default task-3] com.xmlstore.service.BaseXXmlStoreService - id: > 0601e16280001ada slot: 0101e162800009130 > > 2020-09-03 16:33:35,911 INFO [stdout] (default task-3) [16:33:35.911] > TRACE: XQConnection-1.createExpression() invoked. > > 2020-09-03 16:33:35,911 INFO [stdout] (default task-3) [16:33:35.911] > TRACE: XQConnection.getStaticContext() invoked. > > 2020-09-03 16:33:35,926 INFO [stdout] (default task-3) [16:33:35.926] > TRACE: XQExpression-6.executeQuery(), raw XQuery resolved to: > db:replace('testenvSSL1', '6301e16280000114/0601e16280001ada.xml', > 'C:\Program Files (x86)\BaseX\data\0601e16280001ada.xml') > > 2020-09-03 16:33:35,989 INFO [stdout] (default task-3) [16:33:35.989] > DEBUG: XQExpression-6.close() invoked. > > 2020-09-03 16:33:35,989 ERROR [stderr] (default task-3) > javax.xml.xquery.XQException: > 2020-09-03 16:33:35,989 ERROR [stderr] (default task-3) *BXCE001 - > Stopped at ., 1/11*: > > 2020-09-03 16:33:35,989 ERROR [stderr] (default task-3) [FOUP0002] > "C:/Program Files (x86)/BaseX/data/0601e16280001ada.xml" *(Line 1): > Invalid byte 1 of 1-byte UTF-8 sequence.* > > 2020-09-03 16:33:35,989 ERROR [stderr] (default task-3) at > deployment.XMLStoreService.ear.XMLStoreService.war//com.xmlstore.repository.BaseXQueryProcessorClient.storeDocument(BaseXQueryProcessorClient.java:354) > > Could you please help me in resolving this issue, as we are near to our > release. > > Thanks & Regards, > Sharath. > >

> > Please find the attached zip, which contains the sample program for > replicating the encoding issue with the provided datafiles. > Could you please review the provided sample and help me in resolving > this issue. >

On Sun, Sep 6, 2020 at 10:19 AM Sharath Babu Pulumati <spulumat at opentext.com> wrote: > > Hi Christian, > > Below is the error stack trace's stating the error during the file uploading using the both BaseX GUI & Sample program: > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > By Using the BaseX GUI: > Error: > Stopped at ., 1/11: > [FOUP0002] "C:/Sharath/outputxml_06000001800029e2p.xml" (Line 1): Invalid byte 1 of 1-byte UTF-8 sequence. > Optimized Query: > db:replace("DEMO_ENCODING", "6300000180000114/06000001800029e2p_re_store.xml", "C:\Sharath\outputxml_06000001800029e2p.xml") > Query: > db:replace('DEMO_ENCODING', '6300000180000114/06000001800029e2p_re_store.xml','C:\Sharath\outputxml_06000001800029e2p.xml') > Query Plan: > <QueryPlan compiled="true" updating="false"> > <DbReplace name="db:replace" type="empty-sequence()" size="0"> > <Str type="xs:string" size="1">DEMO_ENCODING</Str> > <Str type="xs:string" size="1">6300000180000114/06000001800029e2p_re_store.xml</Str> > <Str type="xs:string" size="1">C:\Sharath\outputxml_06000001800029e2p.xml</Str> > </DbReplace> > </QueryPlan> > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > By Using the Sample Program: > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > createDatabase: started creating database=DEMO_ENCODING > [13:30:33.103] INFO: XQConnection-1 created, the databaseName property was never specified at the XQDataSource level. > xqs.getConnection completed > [13:30:33.103] TRACE: XQConnection.getStaticContext() invoked. > [13:30:33.103] TRACE: XQConnection-1.createExpression() invoked. > [13:30:33.135] TRACE: XQConnection.getStaticContext() invoked. > [13:30:33.432] DEBUG: XQExpression-1.close() invoked. > [13:30:33.432] DEBUG: XQExpression-1.close() invoked. > createDatabase: completed creating database=DEMO_ENCODING > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > xqs.getConnection completed > [13:30:33.432] TRACE: XQConnection.getStaticContext() invoked. > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > StoreDocument: BaseX test store ANSI encoded xml doc > [13:30:33.447] TRACE: XQConnection-2.createExpression() invoked. > [13:30:33.447] TRACE: XQConnection.getStaticContext() invoked. > [13:30:33.510] TRACE: XQExpression-2.executeQuery(), raw XQuery resolved to: db:replace('DEMO_ENCODING', '6300000180000114/06000001800029e2p.xml','C:\Sharath\p.xml') > xqpe.executeQuery completed > [13:30:34.369] DEBUG: XQExpression-2.close() invoked. > StoreDocument: BaseX content push for ANSI encoding is completed > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > RetrieveDocument: BaseX test retrieve xml doc > [13:30:34.385] TRACE: XQConnection-2.createExpression() invoked. > [13:30:34.385] TRACE: XQConnection.getStaticContext() invoked. > [13:30:34.400] TRACE: XQExpression-3.executeQuery(), raw XQuery resolved to: db:open('DEMO_ENCODING', '6300000180000114/06000001800029e2p.xml') > xqpe.executeQuery completed > RetrieveDocument: BaseX test retrieve xml doc completed > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > ReStoreDocument: BaseX test store ANSI encoded xml doc > [13:30:34.416] TRACE: XQConnection-2.createExpression() invoked. > [13:30:34.416] TRACE: XQConnection.getStaticContext() invoked. > [13:30:34.416] TRACE: XQExpression-4.executeQuery(), raw XQuery resolved to: db:replace('DEMO_ENCODING', '6300000180000114/06000001800029e2p_re_store.xml','C:\Sharath\outputxml_06000001800029e2p.xml') > javax.xml.xquery.XQException: > BXCE001 - Stopped at ., 1/11: > [FOUP0002] "C:/Sharath/outputxml_06000001800029e2p.xml" (Line 1): Invalid byte 1 of 1-byte UTF-8 sequence. > at com.test.BasexTest.ReStoreDocument(BasexTest.java:159) > at com.test.BasexTest.main(BasexTest.java:32) > ReStoreDocument: BaseX content push for ANSI encoding is completed > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > FYI, I have attached the new sample program along with the snippet collected from BaseX GUI while reproducing the issue. > My query is regarding, why i'm unable to import the document which is exported from basex? > > Thanks, > Sharath. > > ________________________________ > From: Christian Grün <christian.gruen at gmail.com> > Sent: Saturday, September 5, 2020 10:44 PM > To: Sharath Babu Pulumati <spulumat at opentext.com> > Cc: James Ball <basex-talk at jamesball.co.uk>; BaseX <basex-talk at mailman.uni-konstanz.de>; Rajiv Pandya <rpandya at opentext.com>; Srikumar Choudhury <schoudhu at opentext.com> > Subject: Re: [EXTERNAL] - Re: Re. Encoding support for the BaseX server > > > Please find the attached zip, which contains the sample program for replicating the encoding issue with the provided datafiles. > Could you please review the provided sample and help me in resolving this issue. > > > After having created "testenvSSL1" database, and after having fixed your local paths, I ran your example, and it didn’t trigger any errors. > > Please give me more information on what you believe goes wrong. > >

> 3. Without "<?xml version="1.0" encoding="ISO-8859-1"?>" encoding line in XML file, we are failing store XML file(i.e., ISO-8859-1 encoded file) with below error: > [FOUP0002] "C:/Sharath/outputxml_06000001800029e2p.xml" (Line 1): Invalid byte 1 of 1-byte UTF-8 sequence.

On Mon, Sep 7, 2020 at 9:47 AM Srikumar Choudhury <schoudhu at opentext.com> wrote: > > Hi Christian, > > 1. Currently by default, we are able to store UTF-8 encoding XML files. > 2. Storing of different encoding files are succeeding if we specify the below line: > <?xml version="1.0" encoding="ISO-8859-1"?> > 3. Without "<?xml version="1.0" encoding="ISO-8859-1"?>" encoding line in XML file, we are failing store XML file(i.e., ISO-8859-1 encoded file) with below error: > [FOUP0002] "C:/Sharath/outputxml_06000001800029e2p.xml" (Line 1): Invalid byte 1 of 1-byte UTF-8 sequence. > How do we make basex server to store certain format of encoded file(In above example, it is ISO-8859-1 encoded file)? > Thanks, > Srikumar > > ________________________________ > From: Christian Grün <christian.gruen at gmail.com> > Sent: Sunday, September 6, 2020 2:05 PM > To: Sharath Babu Pulumati <spulumat at opentext.com> > Cc: James Ball <basex-talk at jamesball.co.uk>; BaseX <basex-talk at mailman.uni-konstanz.de>; Rajiv Pandya <rpandya at opentext.com>; Srikumar Choudhury <schoudhu at opentext.com> > Subject: Re: [EXTERNAL] - Re: Re. Encoding support for the BaseX server > > Please note that your code has various flaws that might lead to the > erroneous result. For example, you should avoid calling > String.getBytes, as it will use the default character encoding of your > platform, not necessarily UTF-8 (see [1]). > > I have attached the output that I get for your updated example (it > doesn’t trigger any errors). Did you check the encoding of your > document that’s output by your Java code > (outputxml_06000001800029e2p.xml)? > > [1] https://urldefense.com/v3/__https://docs.oracle.com/javase/7/docs/api/java/lang/String.html*getBytes()__;Iw!!Obbck6kTJA!LxT3Pm8ADcqWi_e1UoDWlVfb8utULC5mRavjj4IBSPTUn0TMhXpH09_wHn0GifRdPg$ > > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > createDatabase: started creating database=DEMO_ENCODING > [10:29:20.124] INFO: XQConnection-1 created, the databaseName > property was never specified at the XQDataSource level. > xqs.getConnection completed > [10:29:20.124] TRACE: XQConnection.getStaticContext() invoked. > [10:29:20.125] TRACE: XQConnection-1.createExpression() invoked. > [10:29:20.183] TRACE: XQConnection.getStaticContext() invoked. > [10:29:20.270] DEBUG: XQExpression-1.close() invoked. > [10:29:20.270] DEBUG: XQExpression-1.close() invoked. > createDatabase: completed creating database=DEMO_ENCODING > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > xqs.getConnection completed > [10:29:20.278] TRACE: XQConnection.getStaticContext() invoked. > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > StoreDocument: BaseX test store ANSI encoded xml doc > [10:29:20.279] TRACE: XQConnection-2.createExpression() invoked. > [10:29:20.279] TRACE: XQConnection.getStaticContext() invoked. > [10:29:20.308] TRACE: XQExpression-2.executeQuery(), raw XQuery > resolved to: db:replace('DEMO_ENCODING', > '6300000180000114/06000001800029e2p.xml','c:\Users\user\Desktop\p.xml') > xqpe.executeQuery completed > [10:29:20.844] DEBUG: XQExpression-2.close() invoked. > StoreDocument: BaseX content push for ANSI encoding is completed > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > RetrieveDocument: BaseX test retrieve xml doc > [10:29:20.844] TRACE: XQConnection-2.createExpression() invoked. > [10:29:20.844] TRACE: XQConnection.getStaticContext() invoked. > [10:29:20.844] TRACE: XQExpression-3.executeQuery(), raw XQuery > resolved to: db:open('DEMO_ENCODING', > '6300000180000114/06000001800029e2p.xml') > xqpe.executeQuery completed > RetrieveDocument: BaseX test retrieve xml doc completed > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > ReStoreDocument: BaseX test store ANSI encoded xml doc > [10:29:20.851] TRACE: XQConnection-2.createExpression() invoked. > [10:29:20.851] TRACE: XQConnection.getStaticContext() invoked. > [10:29:20.851] TRACE: XQExpression-4.executeQuery(), raw XQuery > resolved to: db:replace('DEMO_ENCODING', > '6300000180000114/06000001800029e2p_re_store.xml','c:\Users\user\Desktop\outputxml_06000001800029e2p.xml') > xqpe.executeQuery completed > [10:29:20.858] DEBUG: XQExpression-4.close() invoked. > ReStoreDocument: BaseX content push for ANSI encoding is completed > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > > On Sun, Sep 6, 2020 at 10:19 AM Sharath Babu Pulumati > <spulumat at opentext.com> wrote: > > > > Hi Christian, > > > > Below is the error stack trace's stating the error during the file uploading using the both BaseX GUI & Sample program: > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > By Using the BaseX GUI: > > Error: > > Stopped at ., 1/11: > > [FOUP0002] "C:/Sharath/outputxml_06000001800029e2p.xml" (Line 1): Invalid byte 1 of 1-byte UTF-8 sequence. > > Optimized Query: > > db:replace("DEMO_ENCODING", "6300000180000114/06000001800029e2p_re_store.xml", "C:\Sharath\outputxml_06000001800029e2p.xml") > > Query: > > db:replace('DEMO_ENCODING', '6300000180000114/06000001800029e2p_re_store.xml','C:\Sharath\outputxml_06000001800029e2p.xml') > > Query Plan: > > <QueryPlan compiled="true" updating="false"> > > <DbReplace name="db:replace" type="empty-sequence()" size="0"> > > <Str type="xs:string" size="1">DEMO_ENCODING</Str> > > <Str type="xs:string" size="1">6300000180000114/06000001800029e2p_re_store.xml</Str> > > <Str type="xs:string" size="1">C:\Sharath\outputxml_06000001800029e2p.xml</Str> > > </DbReplace> > > </QueryPlan> > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > By Using the Sample Program: > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > createDatabase: started creating database=DEMO_ENCODING > > [13:30:33.103] INFO: XQConnection-1 created, the databaseName property was never specified at the XQDataSource level. > > xqs.getConnection completed > > [13:30:33.103] TRACE: XQConnection.getStaticContext() invoked. > > [13:30:33.103] TRACE: XQConnection-1.createExpression() invoked. > > [13:30:33.135] TRACE: XQConnection.getStaticContext() invoked. > > [13:30:33.432] DEBUG: XQExpression-1.close() invoked. > > [13:30:33.432] DEBUG: XQExpression-1.close() invoked. > > createDatabase: completed creating database=DEMO_ENCODING > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > xqs.getConnection completed > > [13:30:33.432] TRACE: XQConnection.getStaticContext() invoked. > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > StoreDocument: BaseX test store ANSI encoded xml doc > > [13:30:33.447] TRACE: XQConnection-2.createExpression() invoked. > > [13:30:33.447] TRACE: XQConnection.getStaticContext() invoked. > > [13:30:33.510] TRACE: XQExpression-2.executeQuery(), raw XQuery resolved to: db:replace('DEMO_ENCODING', '6300000180000114/06000001800029e2p.xml','C:\Sharath\p.xml') > > xqpe.executeQuery completed > > [13:30:34.369] DEBUG: XQExpression-2.close() invoked. > > StoreDocument: BaseX content push for ANSI encoding is completed > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > RetrieveDocument: BaseX test retrieve xml doc > > [13:30:34.385] TRACE: XQConnection-2.createExpression() invoked. > > [13:30:34.385] TRACE: XQConnection.getStaticContext() invoked. > > [13:30:34.400] TRACE: XQExpression-3.executeQuery(), raw XQuery resolved to: db:open('DEMO_ENCODING', '6300000180000114/06000001800029e2p.xml') > > xqpe.executeQuery completed > > RetrieveDocument: BaseX test retrieve xml doc completed > > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > ReStoreDocument: BaseX test store ANSI encoded xml doc > > [13:30:34.416] TRACE: XQConnection-2.createExpression() invoked. > > [13:30:34.416] TRACE: XQConnection.getStaticContext() invoked. > > [13:30:34.416] TRACE: XQExpression-4.executeQuery(), raw XQuery resolved to: db:replace('DEMO_ENCODING', '6300000180000114/06000001800029e2p_re_store.xml','C:\Sharath\outputxml_06000001800029e2p.xml') > > javax.xml.xquery.XQException: > > BXCE001 - Stopped at ., 1/11: > > [FOUP0002] "C:/Sharath/outputxml_06000001800029e2p.xml" (Line 1): Invalid byte 1 of 1-byte UTF-8 sequence. > > at com.test.BasexTest.ReStoreDocument(BasexTest.java:159) > > at com.test.BasexTest.main(BasexTest.java:32) > > ReStoreDocument: BaseX content push for ANSI encoding is completed > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > FYI, I have attached the new sample program along with the snippet collected from BaseX GUI while reproducing the issue. > > My query is regarding, why i'm unable to import the document which is exported from basex? > > > > Thanks, > > Sharath. > > > > ________________________________ > > From: Christian Grün <christian.gruen at gmail.com> > > Sent: Saturday, September 5, 2020 10:44 PM > > To: Sharath Babu Pulumati <spulumat at opentext.com> > > Cc: James Ball <basex-talk at jamesball.co.uk>; BaseX <basex-talk at mailman.uni-konstanz.de>; Rajiv Pandya <rpandya at opentext.com>; Srikumar Choudhury <schoudhu at opentext.com> > > Subject: Re: [EXTERNAL] - Re: Re. Encoding support for the BaseX server > > > > > > Please find the attached zip, which contains the sample program for replicating the encoding issue with the provided datafiles. > > Could you please review the provided sample and help me in resolving this issue. > > > > > > After having created "testenvSSL1" database, and after having fixed your local paths, I ran your example, and it didn’t trigger any errors. > > > > Please give me more information on what you believe goes wrong. > > > >