basex docs - bulk export?

Tuesday, 19 February 2013 · 8 messages · last reply 20 Feb 2013
CG
Christian Grünin reply to Colin McEnearney, 19 Feb 2013, 21:5119 Feb 2013 20:59 #
You may want to have a look at the PDF versions of our Wiki: http://docs.basex.org/wiki/Documentation If anyone out there has some ideas how to make PDF generation easier, we’d be glad for your feedback!
Quoted text · Colin McEnearney · 14 lines
> Is there a way to bulk export the basex docs wiki? Or, if it is backed by a > db, get a dump of it? > > Travelling offline and would love to have access (the built-in pdf creator > seems nice but not for getting *everything*) > > Many thanks, > Colin > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
CG
Christian Grünin reply to Colin McEnearney, 19 Feb 2013, 22:0719 Feb 2013 21:11 #
Quoted text · Colin McEnearney · 2 lines
> I see that now on the homepage. Can I suggest that this link be repeated > under "print/export" in the left-hand menu?
Ok, done.
Quoted text · Colin McEnearney · 2 lines
> And just out of curiousity, is there a way to get as data (markupl, csv, > etc.) ?
If you find a way, I’d be interested as well ;) The MediaWiki homepage might be a good starting point [1]. Christian [1] http://www.mediawiki.org/
CM
Colin McEnearney19 Feb 2013 21:51 #
Hi, Is there a way to bulk export the basex docs wiki? Or, if it is backed by a db, get a dump of it? Travelling offline and would love to have access (the built-in pdf creator seems nice but not for getting *everything*) Many thanks, Colin
CM
Colin McEnearneyin reply to Christian Grün, 19 Feb 2013, 20:5919 Feb 2013 22:07 #
Thanks! I see that now on the homepage. Can I suggest that this link be repeated under "print/export" in the left-hand menu? And just out of curiousity, is there a way to get as data (markupl, csv, etc.) ? On Tue, Feb 19, 2013 at 2:59 PM, Christian Grün <christian.gruen at gmail.com>wrote:
Quoted text · Christian Grün · 25 lines
> You may want to have a look at the PDF versions of our Wiki: > > http://docs.basex.org/wiki/Documentation > > If anyone out there has some ideas how to make PDF generation easier, > we’d be glad for your feedback! > ___________________________ > > > Is there a way to bulk export the basex docs wiki? Or, if it is backed > by a > > db, get a dump of it? > > > > Travelling offline and would love to have access (the built-in pdf > creator > > seems nice but not for getting *everything*) > > > > Many thanks, > > Colin > > > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > >
CG
Christian Grünin reply to Colin McEnearney, 19 Feb 2013, 23:3319 Feb 2013 23:04 #
..thanks for investigating!
Quoted text · Colin McEnearney · 31 lines
On Tue, Feb 19, 2013 at 10:33 PM, Colin McEnearney <colinmcenearney at gmail.com> wrote: > It looks like recent versions of MediaWiki have an api enabled by default - > if you have a login you can use it to query the docs, etc. > > I've just applied for one for docs.basex - if I find anything really cool > I'll post back. > > Thanks again, > Colin > > > On Tue, Feb 19, 2013 at 3:11 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> > I see that now on the homepage. Can I suggest that this link be repeated >> > under "print/export" in the left-hand menu? >> >> Ok, done. >> >> > And just out of curiousity, is there a way to get as data (markupl, csv, >> > etc.) ? >> >> If you find a way, I’d be interested as well ;) The MediaWiki homepage >> might be a good starting point [1]. >> >> Christian >> >> [1] http://www.mediawiki.org/ > >
CM
Colin McEnearneyin reply to Christian Grün, 19 Feb 2013, 21:1119 Feb 2013 23:33 #
It looks like recent versions of MediaWiki have an api enabled by default - if you have a login you can use it to query the docs, etc. I've just applied for one for docs.basex - if I find anything really cool I'll post back. Thanks again, Colin On Tue, Feb 19, 2013 at 3:11 PM, Christian Grün <christian.gruen at gmail.com>wrote:
Quoted text · Christian Grün · 15 lines
> > I see that now on the homepage. Can I suggest that this link be repeated > > under "print/export" in the left-hand menu? > > Ok, done. > > > And just out of curiousity, is there a way to get as data (markupl, csv, > > etc.) ? > > If you find a way, I’d be interested as well ;) The MediaWiki homepage > might be a good starting point [1]. > > Christian > > [1] http://www.mediawiki.org/ >
JE
Jens Erat20 Feb 2013 00:40 #
All docs pages in a single XML file probably would be a great source for demonstrating full text search. ;) By the way, you can fetch all pages in export format (is there any reader for that but a local mediawiki instance?) -- by using XQuery, of course:
    declare namespace xhtml = "http://www.w3.org/1999/xhtml";
    declare namespace wiki = "http://www.mediawiki.org/xml/export-0.4/";
    declare option db:parser "html";

    let $pages := fn:serialize(
      doc("http://docs.basex.org/wiki/Special:AllPages")//xhtml:table[@class="mw-allpages-table-chunk"]//xhtml:a,
      map { "method" := "text", "item-separator" := "\r\n" }
    )
    let $request :=
      <http:request href="http://docs.basex.org/index.php?title=Special:Export" method="post">
        <http:body media-type="application/x-www-form-urlencoded" method="text">
          <![CDATA[catname=&curonly=1&wpDownload=1&pages=]]>{ fn:encode-for-uri($pages) }
        </http:body>
      </http:request>
    return http:send-request($request)//wiki:mediawiki
That code interprets the "All Pages" special page and feeds that list into the "Export" page. Regards from Lake Constance, Germany, Jens Erat @Christian: We're definitely missing some HTTP-Post for form data example in the docs, <http:body method="text"/> was really hard to find.
-- Jens Erat [phone]: tel:+49-151-56961126 [mail]: mailto:email at jenserat.de [jabber]: xmpp:jabber at jenserat.de [web]: http://www.jenserat.de PGP: 350E D9B6 9ADC 2DED F5F2 8549 CBC2 613C D745 722B
Quoted text · Colin McEnearney · 28 lines
Am 19.02.2013 um 22:33 schrieb Colin McEnearney <colinmcenearney at gmail.com>: > It looks like recent versions of MediaWiki have an api enabled by default - if you have a login you can use it to query the docs, etc. > > I've just applied for one for docs.basex - if I find anything really cool I'll post back. > > Thanks again, > Colin > > On Tue, Feb 19, 2013 at 3:11 PM, Christian Grün <christian.gruen at gmail.com> wrote: > > I see that now on the homepage. Can I suggest that this link be repeated > > under "print/export" in the left-hand menu? > > Ok, done. > > > And just out of curiousity, is there a way to get as data (markupl, csv, > > etc.) ? > > If you find a way, I’d be interested as well ;) The MediaWiki homepage > might be a good starting point [1]. > > Christian > > [1] http://www.mediawiki.org/ > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
ML
Marco Lettere20 Feb 2013 09:19 #
Quoted text · Jens Erat · 28 lines
On 02/20/2013 12:40 AM, Jens Erat wrote: > All docs pages in a single XML file probably would be a great source for demonstrating full text search. ;) > > By the way, you can fetch all pages in export format (is there any reader for that but a local mediawiki instance?) -- by using XQuery, of course: > > declare namespace xhtml = "http://www.w3.org/1999/xhtml"; > declare namespace wiki = "http://www.mediawiki.org/xml/export-0.4/"; > declare option db:parser "html"; > > let $pages := fn:serialize( > doc("http://docs.basex.org/wiki/Special:AllPages")//xhtml:table[@class="mw-allpages-table-chunk"]//xhtml:a, > map { "method" := "text", "item-separator" := "\r\n" } > ) > let $request := > <http:request href="http://docs.basex.org/index.php?title=Special:Export" method="post"> > <http:body media-type="application/x-www-form-urlencoded" method="text"> > <![CDATA[catname=&curonly=1&wpDownload=1&pages=]]>{ fn:encode-for-uri($pages) } > </http:body> > </http:request> > return http:send-request($request)//wiki:mediawiki > > That code interprets the "All Pages" special page and feeds that list into the "Export" page. > > Regards from Lake Constance, Germany, > Jens Erat > > @Christian: We're definitely missing some HTTP-Post for form data example in the docs, `<http:body method="text"/>` was really hard to find. >
Yeah! I also fought half a day to make it work switching back and forth between BaseX and Expath! :-D M.

⚡Generated with XQuery