33 releases 33 tags

RSS feed

• v0.13.2 d3fc4dbc96

  Compare

  v0.13.2 Stable

  dawid released this 2026-11-04 06:36:04 +00:00 | 81 commits to main since this release

  Signed by dawid

  GPG key ID: 4131A40E8F612CB0

  Added

  • Light browser (website jobs; on for new jobs, off for those saved before): the job's
    browser skips images, fonts and videos while it walks the site, each of which came through the
    browser's checks uncached; pictures' links are still found by their addresses. Ignored for a
    disguised browser and for jobs that save pages or capture the files they load. A picture of a
    failed step shows the pages without their images then.

  • Files linked in an e-mail: a mail job's Also the files linked in the message (off by
    default) downloads the files its messages link to ("Download your invoice"), from the HTML
    part's links and the plain text's addresses, of the Link address or Link text pattern
    (one at least), at most 10 a message; never mailto:, tel:, an unsubscribe link or a private
    address. Through the checked download (public addresses only, at most 10 redirects, the job's
    types and largest file), named by Content-Disposition or the final address, known by the
    message's ID and the link's address. A link that gives a web page is sent to the mail job's
    Pages to the on-demand job (an on-demand website job of the person, whose recipe signs in),
    else logged as a page, not a file; such a page and an expired link (404, 410, 401, 403) are
    logged once and skipped for good.

  • Deleted files downloaded again: Download deleted files again (every job; on, also for the
    jobs saved before) has a run ask Nextcloud, before it skips a file the job saved already,
    whether the file is still there, by its ID: one deleted or in the trash is downloaded again
    (Deleted in Nextcloud, downloaded again: …), one moved or renamed counts as there. Only the
    files the listing finds again are asked for, a listing page's at once (one WebDAV SEARCH per
    100 files at most, each once a run); a file without an ID by the path it was saved at. When
    Nextcloud cannot be asked, nothing is downloaded and the log says so. An unpacked ZIP counts as
    there while its archive or its folder is.

  • More consent managers whose cookie banners Dismiss cookie banners answers by their own
    reject buttons: Sourcepoint (its message's frame, never the page itself), CookieYes and its
    older CookieLawInfo, Complianz, Osano, Iubenda, consentmanager.net, Cookie Script, Termly,
    Axeptio, Cookie Notice and Google's Funding Choices (Do not consent); a recorded click on
    one of their banners is optional then.

  • The recorder adds approvals and handovers: Add step: wait for your approval (once the
    approval went through, until the page's address then) and Add step: hand over to you
    where the recording is; the Steps list changes their label, an approval's wait and their
    until (an address's pattern, this page’s address, an element typed or picked with Pick
    an element).

  • Cookie banners dismissed in the progress log: an info line on the page where it happened,
    naming the consent manager (or a cookie banner) and the button clicked, once a manager in a
    run.

  Changed

  • The app asks Nextcloud and its database less: the admin's settings are read each minute with
    one client (a new one each minute before, never closed), a job's account in one request (one
    per field before, twice a run), the weekly tests' times in one query a minute (one per job),
    and the progress log's poll and the jobs' list read the waiting addresses once (twice).
  • The jobs page asks for less: the list comes without the jobs' options (GET /api/jobs?brief=1
    gives each job's own address instead; a recipe may be 100 KB), a job's form asks for its own
    (the new GET /api/jobs/<id>), and the progress log keeps its lines when the list is drawn
    again (a run that starts or ends), asking only for newer ones, and the queue's status from its
    poll. GET /api/jobs without brief still gives every job's options (Talk AI reads them).
  • The recorder's Network panel keeps at most 8 MB of its answers' bodies together (60 MB a
    recording before: 30 answers of up to 2 MB each), dropping the oldest first; their entries
    stay, and the Preview of one dropped says it is no longer kept.
  • A website job with the browser downloads its files through one checked HTTP client for the
    run (its connections kept), instead of a new one for each file (some 30 ms, a new TLS
    handshake, and a copy of the browser's whole session each): each request still gets the
    browser's cookies for its address as they are when it is sent, and every address and redirect
    is checked as before.
  • Listing pages in the browser are read once their level's links are there (a next page once
    they changed), then after two seconds at most for what a script still loads, instead of half a
    second and up to 15 seconds more for the network to go quiet, which a page with analytics or
    polling never does: three such pages took some 48 seconds, now some 7. On such a page a scroll
    or a Load more round waits up to three seconds for the page to grow instead (a feed of
    three rounds: 79 seconds, now 10). A page whose links do not show within five seconds still
    gets the network's whole wait.
  • The browser's screen comes as binary WebSocket messages, each frame the JPEG itself, instead
    of base64 in JSON (a third larger, and parsed): the page shows each by an object URL and lets
    the one before go.
  • The browser's screen in the recorder (and in Continue in the browser and rclone's sign-in)
    takes less of the server's CPU: every second frame of Chromium's (some 30 frames a second)
    instead of every frame, and none while the jobs page is hidden (another tab, a minimized
    window), again at once when it shows. A view left for another one of the app closes its
    browser at once, instead of after 15 minutes without input.
  • An address sent to an on-demand job again brings back its files deleted in Nextcloud since,
    whatever the option, and the log says where each file fetched before is now (Fetched before,
    in your files at …).
  • A deleted file's record of its content (and the duplicates' that named it) goes when the file
    is found gone, so that its download anew is no duplicate of itself; a known file due for its
    check of changes that is gone is downloaded anew instead of staying away.
  • Saving a file asks Nextcloud less: a file under 5 MB is one PUT (larger ones still go in
    chunks), its ID comes from the answer instead of a lookup, and a job's folders are made once a
    run, not for each file (one made again if it is deleted meanwhile): some 2 requests a file
    instead of about 9.
  • The pause between two downloads counts from the end of the site's last answer (a file, an
    error, a check for changes, a sign-in anew), so that saving and tagging the file are part of
    it: the site still sees the whole pause between two downloads, and a run waits about a second
    less a file.
  • The jobs page asks for the progress log every 30 seconds while none of your jobs runs or
    waits (10 before), and not at all while the page is hidden (another tab, a minimized window):
    at once when it shows again.
  • A page's JavaScript in the job's browser and the recorder's has at most 1 GB of memory (some
    3.7 GB before), so that a page that runs away fails by itself rather than taking the memory of
    the other browsers and the app.
  • The image is some 270 MB smaller: without Playwright's headless shell, which nothing starts
    (the browsers are the full Chromium); an empty HARVESTER_BROWSER_CHANNEL means it too.
  • The app's modules are compiled in the image, not again at each start (some 0.25 s).
  • The container's health check runs every 15 seconds instead of every 2 (a shell and pgrep
    each time, for the container's whole life): a deploy waits some 15 seconds for healthy
    (HaRP waits up to 90); failures in the first five minutes do not count, then four in a row
    make the container unhealthy.
  • The jobs' page on a phone (narrower than 700 pixels): each job a card, its columns one below
    the other with their names and its buttons on a line of their own, instead of a table wider
    than the screen; the header's parts one below the other, the progress log's text below its
    time, and a line of an earlier day with a short date and no seconds. The table's last column
    is named Actions for screen readers.
  • The recorder: Test the recipe and Test as the job runs in its toolbar beside Done
    (below the steps and picks before); a test's result starts with its outcome in one line,
    coloured by it, and is scrolled into view. On a narrow window the screen is above the steps at
    the full width (a sliver beside them before), and the toolbar wraps, Done and Cancel together.
  • The job's form shows a source's essentials (a website's recipe, file name, file types and
    files per run; another source's address, file names and what it takes) and folds the other
    options into Browser and connection, Files and sizes, Repeated runs and a mail
    job's Linked files, each opened by itself when one of its options differs from its
    default, saying how many do ((2 changed)); a website job's form was 21 options in a row.
    GET /api/sources gives each option its group (empty: an essential). Save and the form's
    other buttons stay at the window's foot while the form scrolls.
  • The recorder's step buttons are Add step: scroll to load more, Add step: wait for your
    approval and Add step: hand over to you, on a line of their own below the modes; outside
    Record steps they are off and say Choose Record steps to add steps (only a tooltip before).
    The modes are a group named Mode for screen readers, and the Steps list's ↑, ↓ and
    ✕ are buttons of 34 pixels (a few pixels before), its optional kept with its checkbox.
  • The signs of the recorder's step marks (▶, ⏸, ↻, ✓, –, ✗, ↷, ■), of a pattern's or an
    address's check (✓, ✗) and of ✔ will be used are no part of their translated texts: shown
    beside them, not read out by screen readers.
  • Delete, a template's Remove, Back to the jobs with an account typed and the job's browser's
    Into the account ask in the page, in a dialog of Nextcloud's look (the browser's own
    confirm() before); Escape is No, and a deletion's dialog has the focus on Cancel. The
    recorder's Clear picks offers Undo for 15 seconds, and its Cancel asks before it
    drops changes to the recipe.
  • Buttons alike: the actions in a row or a line of text (a waiting address's Cancel, Show
    what the browser saw, a cookie's Show, the Steps list's this page’s address, Pick an
    element, make optional?) are borderless buttons as Nextcloud's tertiary ones (underlined
    links before), and red is kept for what cannot be undone: a pick's, a level's and a value's
    Remove in the recorder are plain buttons (Undo pick takes them back).

  Fixed

  • Gallery jobs (again): urllib3-future, which nc_py_api's niquests brings, is built without
    installing itself as urllib3 (URLLIB3_NO_OVERRIDE), so gallery-dl gets upstream urllib3 again.
    0.13.1's urllib3 2.7.0 did not help: the fork stood in its place, and lacks
    urllib3.connection.create_urllib3_context.
  • Test the recipe in the recorder failed at a recorded click on a cookie banner that Dismiss
    cookie banners had answered already; it skips the click as optional, as a run does.
  • The jobs' states OK, Failed, Sign-in refused, Changed, Needs you and Paused were
    their text colour on their full colour, hard to read in both themes: a light tint of the colour
    behind the text now.
  • The browser views' screen kept the keyboard: Tab, Shift+Tab and Escape went to the site, with
    no way out but the mouse. Escape twice within a second leaves it for the page's next control,
    which a hint says while the screen has the keyboard; the screen's picture is no longer a stop
    of its own for Tab (its hidden keyboard field is).
  • Screen readers: the progress log was a live region read out at every poll, and the queue's
    load line a status; the log's errors and outcomes are announced through one region now, as is
    the recorder's test (its start and outcome), whose result was a new live region each time.
    Errors are alerts, and a refused Save marks the field it names invalid, described by the
    error, its group opened and the keyboard in it.
  • The page was blank until the app gave its sources and templates: it says Loading…. Save says
    Saving… while the app answers, and a mail job's Pages to the on-demand job says when your
    jobs could not be read (an empty list before). A required field's name (a job's Name and
    Folder) is marked, and a field's help is no longer part of its label but its description.

  Downloads

  • harvester-0.13.2.tar.gz
    0 downloads · 2026-11-04 06:42:13 +00:00 · 1.1 MiB
  • signature.txt
    0 downloads · 2026-11-04 06:42:14 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads