Rounding Issue
Monday, 4 July 2016 · 3 messages
PG
prasad gavhane4 Jul 2016 09:39 #
Hi,
I am using BaseX 8.4.3.
I found some discriminating results while executing following query in two different ways :
Query : round(((( $V1 * $V2 )+( $V3 * $V4 )+( $V5 * $V6 ))div( $V1 + $V3
+ $V5 ))*10000) div 100001. Type 1 (standalone values)
let $V1 := 1500000000
let $V2 := 0.0926
let $V3 := 1000000000
let $V4 := 0.082
let $V5 := 500000000
let $V6 := 0.0931
let $V7 := 0.0892
return round(((( $V1 * $V2 )+( $V3 * $V4 )+( $V5 * $V6 ))div( $V1 + $V3 +
$V5 ))*10000) div 10000Output : 0.0892
2 : Type 2 (values from input xml attached in this mail - val.xml)
let $V1 := data/V1
let $V2 := data/V2
let $V3 := data/V3
let $V4 := data/V4
let $V5 := data/V5
let $V6 := data/V6
let $V7 := data/V7
return round(((( $V1 * $V2 )+( $V3 * $V4 )+( $V5 * $V6 ))div( $V1 + $V3 +
$V5 ))*10000) div 10000Output : 0.0891
Is this a bug or am I doing something wrong?
Thanks,
Prasad Gavhane.
CG
Christian Grün4 Jul 2016 09:46 #
Hi Prasad,
The difference is here:
Quoted text · prasad gavhane · 1 line
> let $V2 := 0.0926
This value is of type xs:decimal.
<V2>0.0926</V2>This value is untyped, and implicitly converted to a double (not a decimal) value when being included in an arithmetic expression. As the following example shows…
4 - 3.1, 4 - 3.1e0
…It’s often a difference if you work with decimals or doubles.
In your case, it may be best to explicitly convert your values before doing the computations:
let $V1 := xs:decimal(data/V1)
let $V2 := xs:decimal(data/V2)
let $V3 := xs:decimal(data/V3)
let $V4 := xs:decimal(data/V4)
let $V5 := xs:decimal(data/V5)
let $V6 := xs:decimal(data/V6)
let $V7 := xs:decimal(data/V7)
return round(((( $V1 * $V2 )+( $V3 * $V4 )+( $V5 * $V6 ))div( $V1 +
$V3 + $V5 ))*10000)Hope this helps,
Christian
PG
prasad gavhane4 Jul 2016 11:34 #
Hi,
Thanks for the response.
While digging, I found following results,
1. result for (((( $V1 * $V2 )+( $V3 * $V4 )+( $V5 * $V6 ))div( $V1 + $V3 +
$V5 ))) is "0.08915"
2.result for (((( $V1 * $V2 )+( $V3 * $V4 )+( $V5 * $V6 ))div( $V1 + $V3 +
$V5 ))*10000) is "891.4999999999999"Hence while multiplying 1 with 10000 ideally it should give 891.5.
Thanks,
Prasad Gavhane.
Quoted text · Christian Grün · 36 lines
On Mon, Jul 4, 2016 at 1:16 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Prasad, > > The difference is here: > > > let $V2 := 0.0926 > > This value is of type xs:decimal. > > <V2>0.0926</V2> > > This value is untyped, and implicitly converted to a double (not a > decimal) value when being included in an arithmetic expression. As the > following example shows… > > 4 - 3.1, 4 - 3.1e0 > > …It’s often a difference if you work with decimals or doubles. > > In your case, it may be best to explicitly convert your values before > doing the computations: > > let $V1 := xs:decimal(data/V1) > let $V2 := xs:decimal(data/V2) > let $V3 := xs:decimal(data/V3) > let $V4 := xs:decimal(data/V4) > let $V5 := xs:decimal(data/V5) > let $V6 := xs:decimal(data/V6) > let $V7 := xs:decimal(data/V7) > return round(((( $V1 * $V2 )+( $V3 * $V4 )+( $V5 * $V6 ))div( $V1 + > $V3 + $V5 ))*10000) > > Hope this helps, > Christian >
Written for BaseX 8.4. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery