Wasn't there a function, that would walk a website?
Tuesday, 31 July 2018 · 7 messages · last reply 1 Aug 2018
AM
Andreas Mixich31 Jul 2018 01:43 #
Hi,
somehow it deems to me, that I once saw a function in BaseX extensive module library, that would recursively read a web-page from remote, following all the links in it. But now I can't find it. Am I dreaming?
--
Goody Bye, Minden jót, Mit freundlichen Grüßen,
Andreas Mixich
CG
Christian Grün31 Jul 2018 08:51 #
I guess you were dreaming ;) But it should definitely be possible to realize this in XQuery without too many lines of code..
Andreas Mixich <mixich.andreas at gmail.com> schrieb am Di., 31. Juli 2018, 01:43:
Quoted text · Andreas Mixich · 10 lines
> Hi, > > somehow it deems to me, that I once saw a function in BaseX extensive > module library, that would recursively read a web-page from remote, > following all the links in it. But now I can't find it. Am I dreaming? > > -- > Goody Bye, Minden jót, Mit freundlichen Grüßen, > Andreas Mixich >
AM
Andreas Mixich1 Aug 2018 08:08 #
Quoted text · Christian Grün · 3 lines
Am 31.07.2018 um 08:51 schrieb Christian Grün: > I guess you were dreaming ;) But it should definitely be possible to > realize this in XQuery without too many lines of code..
Ok, then that's what I am going to do. Thanks for clarification.
--
Goody Bye, Minden jót, Mit freundlichen Grüßen,
Andreas Mixich
CG
Christian Grün1 Aug 2018 09:56 #
Hi Andreas,
Just for fun, I wrote a little crawler in XQuery (see the attached files).
Please note that it’s just a stub; and it should surely be used decently, otherwise the remote server might block further access.
Cheers,
Christian
Quoted text · Andreas Mixich · 11 lines
On Wed, Aug 1, 2018 at 8:08 AM Andreas Mixich <mixich.andreas at gmail.com> wrote: > > Am 31.07.2018 um 08:51 schrieb Christian Grün: > > I guess you were dreaming ;) But it should definitely be possible to > > realize this in XQuery without too many lines of code.. > > Ok, then that's what I am going to do. Thanks for clarification. > > -- > Goody Bye, Minden jót, Mit freundlichen Grüßen, > Andreas Mixich
AM
Andreas Mixich1 Aug 2018 13:27 #
Quoted text · I · 2 lines
Am 01.08.2018 um 09:56 schrieb Christian Grün: > Just for fun, I wrote a little crawler in XQuery (see the attached files).
Very interesting, indeed! Nice to see an example of
lazy:cache and prof:dump. I did not use them, so far, and that is some good news to see them in action.Quoted text · Christian Grün · 2 lines
> it should surely be used decently, otherwise the remote server might > block further access.
Sure! That is one reason I am grinding my teeth on some link analysation right now, so the crawl can be limited to URIs of (a) certain kind(s).
P.S. Resending, since my MUA filled the wrong address (Christian's) instead of the list's into the TO: field and I forgot about it.
--
Goody Bye, Minden jót, Mit freundlichen Grüßen,
Andreas Mixich
LV
Hi Andreas, Christian,
Here attached is a module that I wrote a while ago to limit the rate of requests sent to a web server. This module has been useful in accessing APIs where the SLA does not allow more than a certain number requests per minute, and might be useful for this web crawling scenario. Although Cristian's crawler module already has a sleep built in to it.
Cheers,
Vincent
Quoted text · Christian Grün · 21 lines
-----Original Message----- From: BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> On Behalf Of Christian Grün Sent: Wednesday, August 01, 2018 3:57 AM To: Andreas Mixich <mixich.andreas at gmail.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Wasn't there a function, that would walk a website? Hi Andreas, Just for fun, I wrote a little crawler in XQuery (see the attached files). Please note that it’s just a stub; and it should surely be used decently, otherwise the remote server might block further access. Cheers, Christian On Wed, Aug 1, 2018 at 8:08 AM Andreas Mixich <mixich.andreas at gmail.com> wrote: > > Am 31.07.2018 um 08:51 schrieb Christian Grün: > > I guess you were dreaming ;) But it should definitely be possible to > > realize this in XQuery without too many lines of code.. > > Ok, then that's what I am going to do. Thanks for clarification. > > -- > Goody Bye, Minden jót, Mit freundlichen Grüßen, Andreas Mixich
AM
Andreas Mixich1 Aug 2018 21:58 #
Quoted text · Lizzi, Vincent · 4 lines
Vincent Lizzi wrote: > Hi Andreas, Christian, > > Here attached is a module that I wrote a while ago to limit the rate of requests sent to a web server. This module has been useful in accessing APIs where the SLA does not allow more than a certain number requests per minute, and might be useful for this web crawling scenario. Although Cristian's crawler module already has a sleep built in to it.
Nice! I think I'll just lean back and wait for the code to roll in ;-)
Finally a full example of unit testing in the wild! I never really could wrap my mind around it. Also interesting to see the call to the Java environment. These are advanced techniques (though, they look simple enough, for the start, at least), which I yet have to come by.
--
Goody Bye, Minden jót, Mit freundlichen Grüßen,
Andreas Mixich
Written for BaseX 9.0. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery