Main Page » Mailing List » April 2018

Validation Module: validate:xsd-report( ) improvement

Thursday, 19 April 2018 · 5 messages
YK
Yitzhak Khabinsky19 Apr 2018 19:20 #
Hello, I am successfully using BaseX Validation Module. Along the following lines:
let $xml := 'd:\Temp\CDW\HOME\id4879_BO201801_HomeSubscriberMovementFact.xml'
let $xsd := 'd:\Temp\CDW\HOME\HomeSubscriberMovementFact.xsd'
return validate:xsd-report($xml, $xsd, '1.1')
My XML files have multi-megabyte size and lots of validation errors. In tens or hundreds of thousands of errors. Behind the scenes, Saxon validator 9.8.0.11 is running. Unfortunately, the output structure contains a repeating url attribute. The BaseX output pane cannot present all the errors. It says: "(Chopped) Results".
<report>
    <status>invalid</status>
    <message level="Error" line="10" column="26" url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The content "N/A" of element &lt;CommercialServiceCode&gt; does not match the required simple type. Value "N/A" contravenes the enumeration facet "R60080-X00162, R60080-X00163, ..." of the type Q{http://www.millicom.com}CommercialServiceCodeType</message>
    <message level="Error" line="19" column="23" url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The content "TBD" of element &lt;MovementTechnology&gt; does not match the required simple type. Value "TBD" contravenes the enumeration facet "N/A, HFC, GPON, MMDS, FIBER, C..." of the type Q{http://www.millicom.com}MovementTechnologyType</message>
    <message level="Error" line="24" column="18" url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The content "-1.0000" of element &lt;DownloadSpeed&gt; does not match the required simple type. Value "-1" contravenes the minExclusive facet "0" of the type Q{http://www.millicom.com}DownloadSpeedType</message>
    <message level="Error" line="26" column="6" url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The 7th field in constraint {PK} has no value</message>
    ...
</report>
My proposal is to eliminate the repeated url attribute from the each message and elevate it to its own element just once under the root report tag. Along the following output structure:
<report>
    <status>invalid</status>
    <url>file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml</url>
    <message level="Error" line="10" column="26">The content "N/A" of element &lt;CommercialServiceCode&gt; does not match the required simple type. Value "N/A" contravenes the enumeration facet "R60080-X00162, R60080-X00163, ..." of the type Q{http://www.millicom.com}CommercialServiceCodeType</message>
    <message level="Error" line="19" column="23">The content "TBD" of element &lt;MovementTechnology&gt; does not match the required simple type. Value "TBD" contravenes the enumeration facet "N/A, HFC, GPON, MMDS, FIBER, C..." of the type Q{http://www.millicom.com}MovementTechnologyType</message>
    <message level="Error" line="24" column="18">The content "-1.0000" of element &lt;DownloadSpeed&gt; does not match the required simple type. Value "-1" contravenes the minExclusive facet "0" of the type Q{http://www.millicom.com}DownloadSpeedType</message>
    <message level="Error" line="26" column="6">The 7th field in constraint {PK} has no value</message>
    ...
</report>
This way the output of the validation is much more readable and hopefully will fit in its entirety to the output pane. Regards, Yitzhak Khabinsky Technical Services Lead Millicom International Services LLC 396 Alhambra Circle, Suite 1100 Coral Gables, FL 33134 Skype4B: +1 (305) 445-4172 Tel: (954) 684-8673 yitzhak.khabinsky at millicom.com<mailto:.khabinsky at millicom.com> www.millicom.com<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.millicom.com%2F&data=02%7C01%7CMalcolm.Stewart%40microsoft.com%7C9338023699c2494d08be08d4ad12ce55%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636323743686702678&sdata=DxQ9dfL259rpdVZ9seOAcR6bvgxRobyIerOgvTgPc90%3D&reserved=0>
ML
Marco Lettere19 Apr 2018 19:45 #
Hi Yitzhak, maybe, by slightly rewriting your code, you could remove the unwnated attribute and serialize your output to a file and view it from a text editor? let $xml := 'd:\Temp\CDW\HOME\id4879_BO201801_HomeSubscriberMovementFact.xml'
let $xsd := 'd:\Temp\CDW\HOME\HomeSubscriberMovementFact.xsd'

let $validate := validate:xsd-report($xml, $xsd, '1.1')

return file:write("output.xml",
   copy $newvalidate := $validate
   modify (delete node $newvalidate//@url)
   return $newvalidate
)
Regards, Marco. return validate:xsd-report($xml, $xsd, '1.1')
Quoted text · Yitzhak Khabinsky · 104 lines
On 19/04/2018 19:20, Yitzhak Khabinsky wrote: > > Hello, > > I am successfully using BaseX Validation Module. > > Along the following lines: > > let $xml := > 'd:\Temp\CDW\HOME\id4879_BO201801_HomeSubscriberMovementFact.xml' > > let $xsd := 'd:\Temp\CDW\HOME\HomeSubscriberMovementFact.xsd' > > return validate:xsd-report($xml, $xsd, '1.1') > > My XML files have multi-megabyte size and lots of validation errors. > In tens or hundreds of thousands of errors. > > Behind the scenes, Saxon validator 9.8.0.11 is running. > > Unfortunately, the output structure contains a repeating *url* attribute. > > The BaseX output pane cannot present all the errors. > > It says: “(Chopped) Results”. > > <report> > <status>invalid</status> > <message > level="Error"line="10"column="26"url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The > content "N/A" of element &lt;CommercialServiceCode&gt; does not match > the required simple type. Value "N/A" contravenes the enumeration > facet "R60080-X00162, R60080-X00163, ..." of the type > Q{http://www.millicom.com}CommercialServiceCodeType</message> > <message > level="Error"line="19"column="23"url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The > content "TBD" of element &lt;MovementTechnology&gt; does not match the > required simple type. Value "TBD" contravenes the enumeration facet > "N/A, HFC, GPON, MMDS, FIBER, C..." of the type > Q{http://www.millicom.com}MovementTechnologyType</message> > <message > level="Error"line="24"column="18"url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The > content "-1.0000" of element &lt;DownloadSpeed&gt; does not match the > required simple type. Value "-1" contravenes the minExclusive facet > "0" of the type Q{http://www.millicom.com}DownloadSpeedType</message> > <message > level="Error"line="26"column="6"url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The > 7th field in constraint {PK} has no value</message> > ... > </report> > > My proposal is to eliminate the repeated *url* attribute from the each > *message* and elevate it to its own element just once under the root > *report* tag. > > Along the following output structure: > > > <report> > <status>invalid</status> > <url>file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml</url> > <message level="Error"line="10"column="26">The content "N/A" of > element &lt;CommercialServiceCode&gt; does not match the required > simple type. Value "N/A" contravenes the enumeration facet > "R60080-X00162, R60080-X00163, ..." of the type > Q{http://www.millicom.com}CommercialServiceCodeType</message> > <message level="Error"line="19"column="23">The content "TBD" of > element &lt;MovementTechnology&gt; does not match the required simple > type. Value "TBD" contravenes the enumeration facet "N/A, HFC, GPON, > MMDS, FIBER, C..." of the type > Q{http://www.millicom.com}MovementTechnologyType</message> > <message level="Error"line="24"column="18">The content "-1.0000" of > element &lt;DownloadSpeed&gt; does not match the required simple type. > Value "-1" contravenes the minExclusive facet "0" of the type > Q{http://www.millicom.com}DownloadSpeedType</message> > <message level="Error"line="26"column="6">The 7th field in constraint > {PK} has no value</message> > ... > </report> > > This way the output of the validation is much more readable and > hopefully will fit in its entirety to the output pane. > > Regards, > > Yitzhak Khabinsky > > Technical Services Lead > > Millicom International Services LLC > > 396 Alhambra Circle, Suite 1100 > > Coral Gables, FL 33134 > > Skype4B: +1 (305) 445-4172 > > Tel: (954) 684-8673 > > _yitzhak.khabinsky at millicom.com <mailto:.khabinsky at millicom.com>_ > > www.millicom.com > <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.millicom.com%2F&data=02%7C01%7CMalcolm.Stewart%40microsoft.com%7C9338023699c2494d08be08d4ad12ce55%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636323743686702678&sdata=DxQ9dfL259rpdVZ9seOAcR6bvgxRobyIerOgvTgPc90%3D&reserved=0> >
YK
Yitzhak Khabinsky19 Apr 2018 22:05 #
Hi Marco, The proposed solution is way too slow.
let $xml := 'd:\Temp\CDW\HOME\id4879_BO201801_HomeSubscriberMovementFact.xml'

let $xsd := 'd:\Temp\CDW\HOME\HomeSubscriberMovementFact.xsd'

let $validate := validate:xsd-report($xml, $xsd, '1.1')
return file:write("output.xml",
  copy $newvalidate := $validate
modify (delete node $newvalidate//@url) return $newvalidate ) I guess delete node... is too heavy. It runs for 143 seconds. Without it just 11 seconds. The input XML file size is about 40MB. The output.xml file has about the same size. That's why I was proposing to change the default output format of the validation. Regards, Yitzhak Khabinsky Technical Services Lead Millicom International Services LLC 396 Alhambra Circle, Suite 1100 Coral Gables, FL 33134 Skype4B: +1 (305) 445-4172 Tel: (954) 684-8673 yitzhak.khabinsky at millicom.com<mailto:.khabinsky at millicom.com> www.millicom.com<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.millicom.com%2F&data=02%7C01%7CMalcolm.Stewart%40microsoft.com%7C9338023699c2494d08be08d4ad12ce55%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636323743686702678&sdata=DxQ9dfL259rpdVZ9seOAcR6bvgxRobyIerOgvTgPc90%3D&reserved=0>
Quoted text · Yitzhak Khabinsky · 19 lines
From: Yitzhak Khabinsky Sent: Thursday, April 19, 2018 2:51 PM To: m.lettere at gmail.com Subject: Re: [basex-talk] Validation Module: validate:xsd-report( ) improvement Hi Marco, Thanks for the proposed solution. It works. But I was referring to the default behavior. The url attribute is redundant for every message element. Regards, Yitzhak Khabinsky Technical Services Lead Millicom International Services LLC 396 Alhambra Circle, Suite 1100 Coral Gables, FL 33134 Skype4B: +1 (305) 445-4172 Tel: (954) 684-8673 yitzhak.khabinsky at millicom.com<mailto:.khabinsky at millicom.com> www.millicom.com<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.millicom.com%2F&data=02%7C01%7CMalcolm.Stewart%40microsoft.com%7C9338023699c2494d08be08d4ad12ce55%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636323743686702678&sdata=DxQ9dfL259rpdVZ9seOAcR6bvgxRobyIerOgvTgPc90%3D&reserved=0>
CG
Christian Grünin reply to Yitzhak Khabinsky, 19 Apr 2018, 19:2019 Apr 2018 22:49 #
Hi Yitzhak, Thanks for your suggestion. There are two reasons why we’ll probably need to stick with the existing output format, though: • Changing the format would introduce incompatibilities with previous versions (this is something we only do when switching to new major versions). • More importantly, an XML file might import other documents (e.g. via XInclude), so it will not be guaranteed that the URL is always identical. I wanted to propose a similar solution as Marco did. I would have expected it to be a bit faster, but it’s true that you can save a lot of time if the number of nodes to be deleted or inserted is that large. The following query creates a report with 1 million message elements. It takes 6,5 seconds on my machine (I think this should be ok):
  let $report :=
    <report>
      <status>invalid</status>
      {
        for $in 1 to 1000000
        return <message blu="BLU" url="URL">blablablabla</message>
      }
    </report>
  let $report := element { node-name($report) } {
    $report/* ! element { node-name() } { @* except @url, text() }
  }
  return file:write(file:base-dir() || 'report.xml', $report)
Hope this helps, Christian
Quoted text · Yitzhak Khabinsky · 117 lines
On Thu, Apr 19, 2018 at 7:20 PM, Yitzhak Khabinsky <Yitzhak.Khabinsky at millicom.com> wrote: > Hello, > > > > I am successfully using BaseX Validation Module. > > Along the following lines: > > let $xml := > 'd:\Temp\CDW\HOME\id4879_BO201801_HomeSubscriberMovementFact.xml' > > let $xsd := 'd:\Temp\CDW\HOME\HomeSubscriberMovementFact.xsd' > > return validate:xsd-report($xml, $xsd, '1.1') > > > > My XML files have multi-megabyte size and lots of validation errors. In tens > or hundreds of thousands of errors. > > Behind the scenes, Saxon validator 9.8.0.11 is running. > > > > Unfortunately, the output structure contains a repeating url attribute. > > The BaseX output pane cannot present all the errors. > > It says: “(Chopped) Results”. > > > > <report> > <status>invalid</status> > <message level="Error" line="10" column="26" > url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The > content "N/A" of element &lt;CommercialServiceCode&gt; does not match the > required simple type. Value "N/A" contravenes the enumeration facet > "R60080-X00162, R60080-X00163, ..." of the type > Q{http://www.millicom.com}CommercialServiceCodeType</message> > <message level="Error" line="19" column="23" > url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The > content "TBD" of element &lt;MovementTechnology&gt; does not match the > required simple type. Value "TBD" contravenes the enumeration facet "N/A, > HFC, GPON, MMDS, FIBER, C..." of the type > Q{http://www.millicom.com}MovementTechnologyType</message> > <message level="Error" line="24" column="18" > url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The > content "-1.0000" of element &lt;DownloadSpeed&gt; does not match the > required simple type. Value "-1" contravenes the minExclusive facet "0" of > the type Q{http://www.millicom.com}DownloadSpeedType</message> > <message level="Error" line="26" column="6" > url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml">The > 7th field in constraint {PK} has no value</message> > ... > </report> > > > > My proposal is to eliminate the repeated url attribute from the each message > and elevate it to its own element just once under the root report tag. > > Along the following output structure: > > > <report> > <status>invalid</status> > > <url>file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml</url> > <message level="Error" line="10" column="26">The content "N/A" of > element &lt;CommercialServiceCode&gt; does not match the required simple > type. Value "N/A" contravenes the enumeration facet "R60080-X00162, > R60080-X00163, ..." of the type > Q{http://www.millicom.com}CommercialServiceCodeType</message> > <message level="Error" line="19" column="23">The content "TBD" of > element &lt;MovementTechnology&gt; does not match the required simple type. > Value "TBD" contravenes the enumeration facet "N/A, HFC, GPON, MMDS, FIBER, > C..." of the type Q{http://www.millicom.com}MovementTechnologyType</message> > <message level="Error" line="24" column="18">The content "-1.0000" of > element &lt;DownloadSpeed&gt; does not match the required simple type. Value > "-1" contravenes the minExclusive facet "0" of the type > Q{http://www.millicom.com}DownloadSpeedType</message> > <message level="Error" line="26" column="6">The 7th field in constraint > {PK} has no value</message> > ... > </report> > > > > This way the output of the validation is much more readable and hopefully > will fit in its entirety to the output pane. > > > > Regards, > > Yitzhak Khabinsky > > Technical Services Lead > > Millicom International Services LLC > > 396 Alhambra Circle, Suite 1100 > > Coral Gables, FL 33134 > > Skype4B: +1 (305) 445-4172 > > Tel: (954) 684-8673 > > yitzhak.khabinsky at millicom.com > > www.millicom.com > >
YK
Yitzhak Khabinsky19 Apr 2018 23:10 #
Hi Christian, The proposed solution works much faster. It took 11 seconds to run on machine and completely ok. Thanks a lot. Regards, Yitzhak Khabinsky Technical Services Lead Millicom International Services LLC 396 Alhambra Circle, Suite 1100 Coral Gables, FL 33134 Skype4B: +1 (305) 445-4172 Tel: (954) 684-8673 yitzhak.khabinsky at millicom.com www.millicom.com
Quoted text · Christian Grün · 143 lines
-----Original Message----- From: Christian Grün [mailto:christian.gruen at gmail.com] Sent: Thursday, April 19, 2018 4:50 PM To: Yitzhak Khabinsky <Yitzhak.Khabinsky at Millicom.com> Cc: basex-talk at mailman.uni-konstanz.de Subject: Re: [basex-talk] Validation Module: validate:xsd-report( ) improvement Hi Yitzhak, Thanks for your suggestion. There are two reasons why we’ll probably need to stick with the existing output format, though: • Changing the format would introduce incompatibilities with previous versions (this is something we only do when switching to new major versions). • More importantly, an XML file might import other documents (e.g. via XInclude), so it will not be guaranteed that the URL is always identical. I wanted to propose a similar solution as Marco did. I would have expected it to be a bit faster, but it’s true that you can save a lot of time if the number of nodes to be deleted or inserted is that large. The following query creates a report with 1 million message elements. It takes 6,5 seconds on my machine (I think this should be ok): let $report := <report> <status>invalid</status> { for $in 1 to 1000000 return <message blu="BLU" url="URL">blablablabla</message> } </report> let $report := element { node-name($report) } { $report/* ! element { node-name() } { @* except @url, text() } } return file:write(file:base-dir() || 'report.xml', $report) Hope this helps, Christian On Thu, Apr 19, 2018 at 7:20 PM, Yitzhak Khabinsky <Yitzhak.Khabinsky at millicom.com> wrote: > Hello, > > > > I am successfully using BaseX Validation Module. > > Along the following lines: > > let $xml := > 'd:\Temp\CDW\HOME\id4879_BO201801_HomeSubscriberMovementFact.xml' > > let $xsd := 'd:\Temp\CDW\HOME\HomeSubscriberMovementFact.xsd' > > return validate:xsd-report($xml, $xsd, '1.1') > > > > My XML files have multi-megabyte size and lots of validation errors. > In tens or hundreds of thousands of errors. > > Behind the scenes, Saxon validator 9.8.0.11 is running. > > > > Unfortunately, the output structure contains a repeating url attribute. > > The BaseX output pane cannot present all the errors. > > It says: “(Chopped) Results”. > > > > <report> > <status>invalid</status> > <message level="Error" line="10" column="26" > url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFa > ct.xml">The content "N/A" of element &lt;CommercialServiceCode&gt; > does not match the required simple type. Value "N/A" contravenes the > enumeration facet "R60080-X00162, R60080-X00163, ..." of the type > Q{http://www.millicom.com}CommercialServiceCodeType</message> > <message level="Error" line="19" column="23" > url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFa > ct.xml">The content "TBD" of element &lt;MovementTechnology&gt; does > not match the required simple type. Value "TBD" contravenes the > enumeration facet "N/A, HFC, GPON, MMDS, FIBER, C..." of the type > Q{http://www.millicom.com}MovementTechnologyType</message> > <message level="Error" line="24" column="18" > url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFa > ct.xml">The content "-1.0000" of element &lt;DownloadSpeed&gt; does > not match the required simple type. Value "-1" contravenes the > minExclusive facet "0" of the type > Q{http://www.millicom.com}DownloadSpeedType</message> > <message level="Error" line="26" column="6" > url="file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFa > ct.xml">The 7th field in constraint {PK} has no value</message> > ... > </report> > > > > My proposal is to eliminate the repeated url attribute from the each > message and elevate it to its own element just once under the root report tag. > > Along the following output structure: > > > <report> > <status>invalid</status> > > <url>file:///D:/Temp/CDW/HOME/id4879_BO201801_HomeSubscriberMovementFact.xml</url> > <message level="Error" line="10" column="26">The content "N/A" of > element &lt;CommercialServiceCode&gt; does not match the required > simple type. Value "N/A" contravenes the enumeration facet > "R60080-X00162, R60080-X00163, ..." of the type > Q{http://www.millicom.com}CommercialServiceCodeType</message> > <message level="Error" line="19" column="23">The content "TBD" of > element &lt;MovementTechnology&gt; does not match the required simple type. > Value "TBD" contravenes the enumeration facet "N/A, HFC, GPON, MMDS, > FIBER, C..." of the type Q{http://www.millicom.com}MovementTechnologyType</message> > <message level="Error" line="24" column="18">The content "-1.0000" > of element &lt;DownloadSpeed&gt; does not match the required simple > type. Value "-1" contravenes the minExclusive facet "0" of the type > Q{http://www.millicom.com}DownloadSpeedType</message> > <message level="Error" line="26" column="6">The 7th field in > constraint {PK} has no value</message> > ... > </report> > > > > This way the output of the validation is much more readable and > hopefully will fit in its entirety to the output pane. > > > > Regards, > > Yitzhak Khabinsky > > Technical Services Lead > > Millicom International Services LLC > > 396 Alhambra Circle, Suite 1100 > > Coral Gables, FL 33134 > > Skype4B: +1 (305) 445-4172 > > Tel: (954) 684-8673 > > yitzhak.khabinsky at millicom.com > > www.millicom.com > >

Written for BaseX 9.0. BaseX may have changed since: please check the current documentation.

« Streaming resultsIncrease/Decrease font size not possible with MacOS »
⚡Generated with XQuery