Loading data

> fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json') > => parse-json() (: Convert to XQuery item representation :) > => array:for-each(function($map){ (: For each entry in that array, do :) > element item { (: Construct an XML element named item :) > $map => map:keys() > => for-each(function($key){ (: For each key in the map, do: :) > element { $key } { (: Return an element named $key :) > $map($key) (: …and the value of $map($key) :) > } > }) > } > }) > => array:flatten() (: Converts the array to a sequence :)

> fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json') > => json:parse()

> Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson at gmail.com>: > > I have a JSON file > https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json <https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json> > > I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array. >

> On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms at basex.org> wrote: > > Dear Wray, > > I tried to come up with a sketch of what–I think–might help you: > https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9 > > Basically it boils down to: > 1. Fetch the JSON > 2. Convert that JSON to an XQuery item > 3. Iterate over each array entry and explicitly construct the XML representation you want > > > So in a nutshell, in XQuery 3.1, something like the following: >> fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json') >> => parse-json() (: Convert to XQuery item representation :) >> => array:for-each(function($map){ (: For each entry in that array, do :) >> element item { (: Construct an XML element named item :) >> $map => map:keys() >> => for-each(function($key){ (: For each key in the map, do: :) >> element { $key } { (: Return an element named $key :) >> $map($key) (: …and the value of $map($key) :) >> } >> }) >> } >> }) >> => array:flatten() (: Converts the array to a sequence :) > > > I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away: >> fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json') >> => json:parse() > For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Module > > > As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want. > > > Hope this helps ;-) > > Best from Konstanz > > Michael > >> Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson at gmail.com>: >> >> I have a JSON file >> https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json >> >> I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array. >> >

From: <basex-talk-bounces at mailman.uni-konstanz.de> on behalf of "E. Wray Johnson" <wray.johnson at gmail.com> Date: Sunday, November 19, 2017 at 5:15 AM To: Michael Seiferle <ms at basex.org> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Loading data Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works. Wray Johnson (m) 704-293-9008 On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms at basex.org<mailto:ms at basex.org>> wrote: Dear Wray, I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=> Basically it boils down to: 1. Fetch the JSON 2. Convert that JSON to an XQuery item 3. Iterate over each array entry and explicitly construct the XML representation you want So in a nutshell, in XQuery 3.1, something like the following: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :) I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>') => json:parse() For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Module<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=> As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want. Hope this helps ;-) Best from Konstanz Michael Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson at gmail.com<mailto:wray.johnson at gmail.com>>: I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.

On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw <kendall.shaw at workday.com> wrote: > Hi, > > > > Just about building maps, in general: If you look at map:merge in > http://docs.basex.org/wiki/Map_Module it shows a way to build a map with > a for expression. If the question updating maps in a loop was about this, > then it is probably preferable to have the result of the for expression be > a map, rather than updating a map, iteratively. > > > > Kendall > > > > *From: *<basex-talk-bounces at mailman.uni-konstanz.de> on behalf of "E. > Wray Johnson" <wray.johnson at gmail.com> > *Date: *Sunday, November 19, 2017 at 5:15 AM > *To: *Michael Seiferle <ms at basex.org> > *Cc: *BaseX <basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] Loading data > > > > Thanks. I tried doing #2 inside the iteration and that did not work. I > will try it your way and let you know if it works. > > Wray Johnson > > (m) 704-293-9008 <(704)%20293-9008> > > > On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms at basex.org> wrote: > > Dear Wray, > > > > I tried to come up with a sketch of what–I think–might help you: > > https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=> > > > > Basically it boils down to: > > 1. Fetch the JSON > > 2. Convert that JSON to an XQuery item > > 3. Iterate over each array entry and explicitly construct the XML > representation you want > > > > > > So in a nutshell, in XQuery 3.1, something like the following: > > fetch:text('https://gist.githubusercontent.com/ > Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e8 > 1d25e2c6f6/cities.json > <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> > ') > > => parse-json() (: Convert to XQuery item representation :) > > => array:for-each(function($map){ (: For each entry in that array, do :) > > element item { (: Construct an XML element named item :) > > $map => map:keys() > > => for-each(function($key){ (: For each key in the map, do: :) > > element { $key } { (: Return an element named $key :) > > $map($key) (: …and the value of $map($key) :) > > } > > }) > > } > > }) > > => array:flatten() (: Converts the array to a sequence :) > > > > > > I assume you are using BaseX, so maybe the json:parse() function might be > another option, that is based on our own implementation and will create an > XML representation right away: > > fetch:text('https://gist.githubusercontent.com/ > Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e8 > 1d25e2c6f6/cities.json > <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> > ') > > => json:parse() > > For more info on our BaseX JSON-Module see: http://docs.basex.org/ > wiki/JSON_Module > <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=> > > > > As I can only guess what your XQuery code actually looks like I hope this > comes somewhat close to what you want. > > > > > > Hope this helps ;-) > > > > Best from Konstanz > > > > Michael > > > > Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson at gmail.com>: > > > > I have a JSON file > https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/ > 2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json > <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> > > I want to load it into a database in a different XML format where I can > use gml:Point in place of longitude and latitude as separate elements. > However my for loop does not emit individual XML elements for each json > object in the loaded array. > > > >

From: "E. Wray Johnson" <wray.johnson at gmail.com> Date: Monday, November 20, 2017 at 10:37 AM To: Kendall Shaw <kendall.shaw at workday.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Loading data I may need to load hundreds of thousands of items each with dozens of sub-nodes. I am looking for a solution that will not be at risk of running out of heap memory. On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw <kendall.shaw at workday.com<mailto:kendall.shaw at workday.com>> wrote: Hi, Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Module<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Map-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=Vqrcy5xoTXekYA_60uYsRLfwE8plvHtMAGq8kejU0do&s=NQHz3DOpC6VgtvcpVqQQScfIlaR_5kZ_TbihOEVZATY&e=> it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be a map, rather than updating a map, iteratively. Kendall From: <basex-talk-bounces at mailman.uni-konstanz.de<mailto:basex-talk-bounces at mailman.uni-konstanz.de>> on behalf of "E. Wray Johnson" <wray.johnson at gmail.com<mailto:wray.johnson at gmail.com>> Date: Sunday, November 19, 2017 at 5:15 AM To: Michael Seiferle <ms at basex.org<mailto:ms at basex.org>> Cc: BaseX <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Loading data Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works. Wray Johnson (m) 704-293-9008<tel:(704)%20293-9008> On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms at basex.org<mailto:ms at basex.org>> wrote: Dear Wray, I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=> Basically it boils down to: 1. Fetch the JSON 2. Convert that JSON to an XQuery item 3. Iterate over each array entry and explicitly construct the XML representation you want So in a nutshell, in XQuery 3.1, something like the following: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :) I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>') => json:parse() For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Module<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=> As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want. Hope this helps ;-) Best from Konstanz Michael Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson at gmail.com<mailto:wray.johnson at gmail.com>>: I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.

From: <basex-talk-bounces at mailman.uni-konstanz.de> on behalf of Kendall Shaw <kendall.shaw at workday.com> Date: Monday, November 20, 2017 at 12:13 PM To: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Loading data I hope that the tuple stream of a FLWOR expression does not accumulate heap memory. Is that the case, Christian, or anyone? Kendall From: "E. Wray Johnson" <wray.johnson at gmail.com> Date: Monday, November 20, 2017 at 10:37 AM To: Kendall Shaw <kendall.shaw at workday.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Loading data I may need to load hundreds of thousands of items each with dozens of sub-nodes. I am looking for a solution that will not be at risk of running out of heap memory. On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw <kendall.shaw at workday.com<mailto:kendall.shaw at workday.com>> wrote: Hi, Just about building maps, in general: If you look at map:merge in http://docs.basex.org/wiki/Map_Module<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Map-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=Vqrcy5xoTXekYA_60uYsRLfwE8plvHtMAGq8kejU0do&s=NQHz3DOpC6VgtvcpVqQQScfIlaR_5kZ_TbihOEVZATY&e=> it shows a way to build a map with a for expression. If the question updating maps in a loop was about this, then it is probably preferable to have the result of the for expression be a map, rather than updating a map, iteratively. Kendall From: <basex-talk-bounces at mailman.uni-konstanz.de<mailto:basex-talk-bounces at mailman.uni-konstanz.de>> on behalf of "E. Wray Johnson" <wray.johnson at gmail.com<mailto:wray.johnson at gmail.com>> Date: Sunday, November 19, 2017 at 5:15 AM To: Michael Seiferle <ms at basex.org<mailto:ms at basex.org>> Cc: BaseX <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Loading data Thanks. I tried doing #2 inside the iteration and that did not work. I will try it your way and let you know if it works. Wray Johnson (m) 704-293-9008<tel:(704)%20293-9008> On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms at basex.org<mailto:ms at basex.org>> wrote: Dear Wray, I tried to come up with a sketch of what–I think–might help you: https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=> Basically it boils down to: 1. Fetch the JSON 2. Convert that JSON to an XQuery item 3. Iterate over each array entry and explicitly construct the XML representation you want So in a nutshell, in XQuery 3.1, something like the following: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>') => parse-json() (: Convert to XQuery item representation :) => array:for-each(function($map){ (: For each entry in that array, do :) element item { (: Construct an XML element named item :) $map => map:keys() => for-each(function($key){ (: For each key in the map, do: :) element { $key } { (: Return an element named $key :) $map($key) (: …and the value of $map($key) :) } }) } }) => array:flatten() (: Converts the array to a sequence :) I assume you are using BaseX, so maybe the json:parse() function might be another option, that is based on our own implementation and will create an XML representation right away: fetch:text('https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=>') => json:parse() For more info on our BaseX JSON-Module see: http://docs.basex.org/wiki/JSON_Module<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=> As I can only guess what your XQuery code actually looks like I hope this comes somewhat close to what you want. Hope this helps ;-) Best from Konstanz Michael Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson at gmail.com<mailto:wray.johnson at gmail.com>>: I have a JSON file https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> I want to load it into a database in a different XML format where I can use gml:Point in place of longitude and latitude as separate elements. However my for loop does not emit individual XML elements for each json object in the loaded array.

On Mon, Nov 20, 2017 at 3:43 PM, Kendall Shaw <kendall.shaw at workday.com> wrote: > Also, I would think that map:merge would be the typical FP idea where > conceptually each step is a copy of the entire map, but it’s implemented as > only what has changed being new. If that isn’t the case that would be > important to know. > > > > Kendall > > > > *From: *<basex-talk-bounces at mailman.uni-konstanz.de> on behalf of Kendall > Shaw <kendall.shaw at workday.com> > *Date: *Monday, November 20, 2017 at 12:13 PM > *To: *BaseX <basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] Loading data > > > > I hope that the tuple stream of a FLWOR expression does not accumulate > heap memory. Is that the case, Christian, or anyone? > > > > Kendall > > > > *From: *"E. Wray Johnson" <wray.johnson at gmail.com> > *Date: *Monday, November 20, 2017 at 10:37 AM > *To: *Kendall Shaw <kendall.shaw at workday.com> > *Cc: *BaseX <basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] Loading data > > > > I may need to load hundreds of thousands of items each with dozens of > sub-nodes. > > I am looking for a solution that will not be at risk of running out of > heap memory. > > > > On Mon, Nov 20, 2017 at 12:44 PM, Kendall Shaw <kendall.shaw at workday.com> > wrote: > > Hi, > > > > Just about building maps, in general: If you look at map:merge in > http://docs.basex.org/wiki/Map_Module > <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Map-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=Vqrcy5xoTXekYA_60uYsRLfwE8plvHtMAGq8kejU0do&s=NQHz3DOpC6VgtvcpVqQQScfIlaR_5kZ_TbihOEVZATY&e=> > it shows a way to build a map with a for expression. If the question > updating maps in a loop was about this, then it is probably preferable to > have the result of the for expression be a map, rather than updating a > map, iteratively. > > > > Kendall > > > > *From: *<basex-talk-bounces at mailman.uni-konstanz.de> on behalf of "E. > Wray Johnson" <wray.johnson at gmail.com> > *Date: *Sunday, November 19, 2017 at 5:15 AM > *To: *Michael Seiferle <ms at basex.org> > *Cc: *BaseX <basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] Loading data > > > > Thanks. I tried doing #2 inside the iteration and that did not work. I > will try it your way and let you know if it works. > > Wray Johnson > > (m) 704-293-9008 <(704)%20293-9008> > > > On Nov 19, 2017, at 6:30 AM, Michael Seiferle <ms at basex.org> wrote: > > Dear Wray, > > > > I tried to come up with a sketch of what–I think–might help you: > > https://gist.github.com/micheee/8a8734a1713a7121cab15eb3dfb389d9 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_micheee_8a8734a1713a7121cab15eb3dfb389d9&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Guzfb_5gXuH4sJ2goSUdevQcE_tYJ-A_zXytOn3yTPs&e=> > > > > Basically it boils down to: > > 1. Fetch the JSON > > 2. Convert that JSON to an XQuery item > > 3. Iterate over each array entry and explicitly construct the XML > representation you want > > > > > > So in a nutshell, in XQuery 3.1, something like the following: > > fetch:text('https://gist.githubusercontent.com/ > Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e8 > 1d25e2c6f6/cities.json > <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> > ') > > => parse-json() (: Convert to XQuery item representation :) > > => array:for-each(function($map){ (: For each entry in that array, do :) > > element item { (: Construct an XML element named item :) > > $map => map:keys() > > => for-each(function($key){ (: For each key in the map, do: :) > > element { $key } { (: Return an element named $key :) > > $map($key) (: …and the value of $map($key) :) > > } > > }) > > } > > }) > > => array:flatten() (: Converts the array to a sequence :) > > > > > > I assume you are using BaseX, so maybe the json:parse() function might be > another option, that is based on our own implementation and will create an > XML representation right away: > > fetch:text('https://gist.githubusercontent.com/ > Miserlou/c5cd8364bf9b2420bb29/raw/2bf258763cdddd704f8ffd3ea9a3e8 > 1d25e2c6f6/cities.json > <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> > ') > > => json:parse() > > For more info on our BaseX JSON-Module see: http://docs.basex.org/ > wiki/JSON_Module > <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_JSON-5FModule&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=Uia9oCJta1vQG3jvblV437dpVM5JvI9GsfmVjj2xssY&e=> > > > > As I can only guess what your XQuery code actually looks like I hope this > comes somewhat close to what you want. > > > > > > Hope this helps ;-) > > > > Best from Konstanz > > > > Michael > > > > Am 19.11.2017 um 06:25 schrieb E. Wray Johnson <wray.johnson at gmail.com>: > > > > I have a JSON file > https://gist.githubusercontent.com/Miserlou/c5cd8364bf9b2420bb29/raw/ > 2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6/cities.json > <https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.githubusercontent.com_Miserlou_c5cd8364bf9b2420bb29_raw_2bf258763cdddd704f8ffd3ea9a3e81d25e2c6f6_cities.json&d=DwMFaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=JgwnBEpN1c-DDmq-Up2QMq9rrGyfWK0KtSpT7dxRglA&m=0dY_XpYSlIufV9bPNvpZD4BvkOL2-P3MduC5WpG4FTc&s=oX2YSOYRvXFIiOp87Q6C_ev1UGt0DWVkaCFu0whK6GM&e=> > > I want to load it into a database in a different XML format where I can > use gml:Point in place of longitude and latitude as separate elements. > However my for loop does not emit individual XML elements for each json > object in the loaded array. > > > > >