33 releases 33 tags

RSS feed

• v0.14.2 0d5107dc80

  Compare

  v0.14.2 Stable

  dawid released this 2026-11-05 04:15:09 +00:00 | 7 commits to main since this release

  Signed by dawid

  GPG key ID: 4131A40E8F612CB0

  Added

  • Settings at the top of Harvester's page takes an admin to Harvester's settings
    (Administration settings > Additional settings); nobody else sees it.
  • Reset the library… puts a person's library back to how it was before its setup (every
    file, folder and label deleted with its keys), also from the unlock page for someone who lost
    the password and the recovery key; only the admin's occ command could before.
  • A listing that loads more as it is scrolled can be read as it is scrolled
    ("next": {"kind": "scroll"}, the recorder's Read it as it is scrolled): a batch at a time,
    each a page, so the job stops scrolling at known files, Pages or Files per run instead of
    loading the whole listing first, as the step scroll to load more does.
  • Import from Chrome's Recorder and Export for Chrome's Recorder in a website job's
    form: a recording of Chrome DevTools' Recorder made the recipe's steps (its selectors as
    Playwright's, a password typed in it never kept, what a recipe has no step for said), and the
    recipe's steps as such a recording, Harvester's own steps as custom steps; a round trip gives
    the same steps.
  • Steps of the kinds other recorders have: double and right clicks (recorded too), the answer to
    a dialog a click or a key opens (OK, a prompt's text, or Cancel; the recorder answers OK and
    records it), a fill typed key by key, a key pressed on the page, and then wait for an
    element…: a step that waits until an element shows (with a text) or is gone.

  Fixed

  • YouTube's formats are all there again: yt-dlp solves YouTube's challenges ("n challenge
    solving failed: Some formats may be missing") with yt-dlp-ejs's scripts, now in the image, in
    its Node.js, which runs without the app's environment and with Node's permission model.

  Downloads

  • harvester-0.14.2.tar.gz
    1 download · 2026-11-06 10:43:47 +00:00 · 1.7 MiB
  • signature.txt
    0 downloads · 2026-11-06 10:43:47 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.14.1 bc6f453dd5

  Compare

  v0.14.1 Stable

  dawid released this 2026-11-04 22:19:47 +00:00 | 13 commits to main since this release

  Signed by dawid

  GPG key ID: 4131A40E8F612CB0

  Added

  • Save to Immich Into the locked folder: photos and videos uploaded into Immich's locked
    folder (one Immich has already moved there), without an album, which Immich allows no locked
    photo; with Immich's tags each is tagged right after its upload and locked at once (a moment
    in the timeline), its tags kept as they are later; the job knows its files by their content's
    SHA-1, as an API key never sees the locked folder.
  • The Library browses as Files does: a grid or a list, sorted by name, date or size (kept in
    the browser), Recent (the newest files of the whole library), files chosen with a box
    each (Shift for a range, all at once) for Download as one ZIP file streamed as it is made,
    Move (the folders offered), Labels, Rename, Details and Delete, and
    dragged onto a folder or a part of the path to move them.
  • The Library's viewer goes on to the next file and the one before (‹ ›, the arrow keys, a
    swipe); a file's Details name the job that saved it and the link it came from, with its
    values and labels.
  • Move to Files takes files and folders out of the library into a folder of your Files,
    each checked (read back from Files and compared) before the library's copy goes.
  • Changing where a job's files go (another Save to, vault or folder) offers Download the
    files saved before again, into the new place, on by default: the job forgets that it fetched
    them and its next runs download them again as they find them. Off, they stay where they are,
    as before. The log says which, with how many files and both places.

  Fixed

  • The admin's settings are saved: Nextcloud's settings page took "harvester_" off the forms' IDs
    as it saved a value, so AppAPI found no such form and dropped every change (while answering
    OK). The forms have IDs of their own now; the form says that each change saves itself (there
    is no Save button).
  • The admin's numbers show without quotes (0, not "0"): AppAPI showed a number given as text
    in JSON's quotes, and a form saved as shown kept them, which Harvester read as no value. Such
    a value is now read as the number inside the quotes.
  • The Library's viewer and the page's questions open in the middle of the window, at any width
    (a phone's too), not at its top left; the viewer is as wide as the window allows.
  • A move's states from Files are readable in light and dark: tinted pills with the theme's
    success and error text colours, as the jobs' states, not light text on a light green.
  • The progress log names where a file went: Uploaded to Immich: a.jpg (2.1 MB), album “Trip”,
    Already in Immich, Tagged in Immich, Saved in your library, Saved in the vault …,
    rather than Saved /folder/a.jpg as if it were in Files. The size is the file's.
  • A file name from a value ending in a dot before the extension's (a title "We tried." in
    {title}.{ext}, yt-dlp's {filename}) has one dot, not We tried..mp3.
  • A vault's inbox follows the job's Folder in the vault: a change after the job's first file
    reaches the inbox's marker with the next file, rather than the vault app moving the files on
    into the folder the marker named first (the vault's top). A marker made in the vault app keeps
    its folder while the job's is /.
  • Immich's tags show: Immich hides every tag until the person turns Tags on (Account Settings >
    Features, off for a new account), so the tags Harvester gave were there but nowhere to be seen.
    A run that tags in Immich turns them on, once, and says so; a key without
    userPreference.update gets the log's hint where to turn them on. A tag named with a / at its
    ends or doubled is the one Immich makes, no longer left out quietly; a tag Immich does not make
    fails its file.
  • Move to the private library in Files opens the Library with the move it proposed although
    AppAPI's redirect comes without ?fileIds=, and so does a reload in the next 15 minutes, until
    the move starts.

  Downloads

  • harvester-0.14.1.tar.gz
    0 downloads · 2026-11-04 22:35:10 +00:00 · 1.7 MiB
  • signature.txt
    0 downloads · 2026-11-04 22:35:10 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.14.0 ff26738f73

  Compare

  v0.14.0 Stable

  dawid released this 2026-11-04 16:47:13 +00:00 | 27 commits to main since this release

  Signed by dawid

  GPG key ID: 4131A40E8F612CB0

  Added

  • Tags from lists: a job's tag that is a list value alone ({tags}) gives a tag per
    entry (cleaned, without repeats, 20 at most per file; the log says how many more were left
    out); elsewhere a list's entries are joined with a comma. gallery-dl's lists of texts are kept
    ({tags}) and its dicts of them become one each ({tags_general}, {tags_artist}), yt-dlp's
    {tags} and {categories} too; the form offers each source's under Tags.
  • Values of the pages a website job opens: {page_title}, {site_name}, {page_date}
    (with {page_year}, {page_month}, {page_day}) and {page_lang}, from <meta>, JSON-LD
    and <time>, read once per page (one script in the browser, the parsed page over HTTP) and
    passed down to the next levels and the file links. A level's page_values read more on the
    page its links open, and a value with "many": true is a list (tag chips, cleaned of # and
    counts).
  • Recorder: Pick tags beside a level's links (a chip in a row or card), Pick a value on
    its pages and Pick tags on its pages on a page a level's links open (the chip and those
    like it outlined); each level lists the pages it opened with their values, and Test the
    recipe shows each file link's values.
  • Save to in a job's form: the person's files (as before) or Harvester's private
    library, in the app's own storage per person, which Nextcloud's Files, Photos, search,
    WebDAV and sync clients never see, with the admin's limits per person (10 GB by default) and
    in all. A file deleted in the library is fetched again as one deleted in Files (Download
    deleted files again).
  • The private library is encrypted, a hidden space for any file: an X25519 key pair per
    person, its private key wrapped with the library password (scrypt) and a recovery key shown
    once at setup (typed back to confirm); each file's content in 64 KiB chunks of AES-256-GCM
    (ranges decrypt their chunks alone; a chunk changed, swapped or cut off is found), its key
    sealed to the public key, its name, folder, size, type, labels and thumbnail encrypted too.
    Jobs, uploads and moves write while it is locked; reading needs it unlocked, the private key
    in the server's memory only, dropped at lock, after the person's minutes and at a restart.
    Library on the page: setup, unlock, recovery, folders, thumbnails (Pillow), a viewer,
    search by name and label, uploads in pieces (drag and drop), Rename, Move, Delete, and a
    cookie (HttpOnly, SameSite=Strict) or per-file tickets instead of a token in addresses.
  • Move to the private library in Files' actions menu, for files and folders: each streamed
    into the library, checked, the original deleted (its versions with it) and purged from the
    trash bin; a file that fails stays, versions and all, and the page says why. Nothing shared
    with the person and nothing in a team folder is moved. occ harvester:library-reset <user> --confirm: the admin deletes a person's library.
  • Save to Immich: a job's photos and videos uploaded into the person's Immich account
    (Immich's API of version 3), streamed, after Immich was asked for the content's SHA-1 (one
    there already is adopted, and still filed and tagged); an optional Album by a template
    (Magazines {year}), made once and reused; Immich's tags as Tag as (a/b nested); a
    photo deleted in Immich uploaded again; other files skipped for good with why; an unpacked
    ZIP's photos and videos alone. Your Immich account in the form: the server and the
    person's API key (a sensitive preference, tested as it is saved, never sent back;
    GET/PUT/DELETE /api/immich, POST /api/immich/test). The admin's Immich server
    (HARVESTER_IMMICH_URL) is preset for everyone and reached directly; a server a person types,
    through the guard's check; https:// unless Immich over plain http
    (HARVESTER_IMMICH_HTTP).
  • Save to a Cryptomator vault: its inbox (the folder gets the marker of the app Vaults
    for Cryptomator, which moves the files into the vault when the person unlocks it; files moved
    out are never fetched again) or the vault directly (each file encrypted on the server and
    written over WebDAV, harvester/vault.py; the password typed once, only the vault's keys kept,
    as a sensitive preference; a prominent warning that the server then holds the vault's keys).
    PUT /api/jobs/<id>/vault (the password), GET (whether the keys are kept), DELETE.
  • Tag as in a job's form: Nextcloud's tags (as before), Harvester's labels (in the app's
    database, shown and filtered in the job's Files and the library, never a Nextcloud tag)
    or none. GET /api/jobs/<id>/files: the files a job saved, with their tags or labels.
  • Browse beside the address of a gallery, video, feed or release job: the recorder's browser
    alone (nothing recorded), at the field's address, and Use this address takes the page it
    shows into the field.
  • Try the address in the form of every job but a website's (Try the mailbox, Try the
    remote): the listing once as the form holds it, before it is saved, nothing downloaded
    (POST /api/try): what a run would fetch (Files per run), what the job's history has, what
    its file types, sizes and patterns leave out, the first 20 files with the names and tags they
    would get, a mailbox's messages with what each gives, or why it failed with a hint. In a
    runner's place, through the guard, the proxy and the DNS servers, 60 seconds at most, one try
    of a person's at a time.
  • DNS servers of the admin's choice: DNS servers (a form of its own in Harvester's admin
    settings, or HARVESTER_DNS) sends every name the container looks up (the app and its guard,
    the browsers, the downloads, yt-dlp, gallery-dl, rclone, the mail servers) to plain addresses,
    DNS over TLS (tls://host[:port]) or DNS over HTTPS (https://host[:port]/path), a ClientID in
    the name or path, e.g. a filtering AdGuard Home. AdGuard's dnsproxy (in the image, pinned
    and checked) runs on 127.0.0.1:53 and /etc/resolv.conf names it; Nextcloud's, HaRP's and
    the proxy's names stay Docker's. Servers that do not answer fail the lookups, unless Docker's
    DNS when the servers fail (HARVESTER_DNS_FALLBACK, off by default) is on. The form's
    description is the status: whether the proxy runs and a test lookup works.
  • A name the DNS servers block (they answer 0.0.0.0 or ::) is told as such in the job's log,
    by the browser's guard and the downloads' checks (blocked by the DNS server), and a name they
    do not know as possibly blocked.
  • Fetch first in a mail job's form: Attachments, Linked files or Both. The form shows
    only the chosen part's options (an option's shown_when; a group whose options are all hidden
    goes with them) and keeps the others' values; a Linked files job asks for a link address or
    link text and nothing of attachment names. A job saved before keeps what it fetched (its
    attachments, and its linked files when Also the files linked in the message was on), read so
    (Source.upgraded, also for GET /api/jobs/<id> and a recipe file of before), not
    rewritten. A choice's words in the form (an option's labels).
  • A mail job's folder the mailbox does not have: the run's error names the folders it has, the
    closest first, ten of them (The mailbox has no folder “Rechnungen”. It has:
    INBOX/Rechnungen, Archive, INBOX); Try the mailbox checks the folder first and offers the
    mailbox's folders to take into the field (folders in POST /api/try's answer).
  • A mail job's run whose new messages are none of the sender and subject says so as a warning
    naming the patterns (Folder INBOX: none of the 12 new messages is from “…” with a subject
    like “…”).
  • Under a mail job's Pages to the on-demand job, while you have no on-demand website job:
    how to make one (record one for the portal, Website, On demand).

  Changed

  • The recorder's patterns in plain words: Address must match, Text must match (a pick
    by its text) and File's address must match (optional) (a JSON listing), each with a line
    saying that a part written (?P<name>…) becomes {name} for file names and tags, with an
    example.

  • A mail job's Pages to the on-demand job says what it is for in one sentence.

  • The progress log names the address a run looks at (Looking for new files at https://…) and
    says how many of the listed files the job has already (37 listed files are known already: not
    downloaded again), and the run's outcome counts them (0 downloaded, 37 known).

  • A website job's progress log names each page of file links with what it matched (page 2
    (https://…): 12 of 40 links match, 3 new, 20 pages a run) and says why a listing ended: a page
    of known files only (Stop at known files), or the Pages limit with a next page still there
    (Read 10 pages (Pages): older pages are not read this run). A level's waiting for … to show
    is said once a run, not for each of its listings.

  • Feed, video, file store and release jobs say in the log what their listing found: The feed
    “Reports”: 20 entries, 18 file links, 30 entries listed (Entries per run: 30) and the live
    or upcoming ones skipped, rclone listed 12 files in share:Reports (or that Files took
    none), 5 releases looked at (2 drafts or prereleases left out), 7 files match.

  • A download's waits and retries are lines in the log: … asks to wait 60 s (429, 503, a
    wait pattern) and …: the connection failed, trying again in 10 s, not a silent pause.

  • A video job's log has yt-dlp's warnings (ten a run, each once), and yt-dlp's advice for its
    command line is said as the job's options: Use --cookies-from-browser or --cookies reads
    Set the job's account: Cookies, likewise its user name and password, its formats and
    impersonation.

  • A watched page's run outcome says what changed: The page changed: -Tea: 2 EUR +Tea: 2.50 EUR
    (one line, 100 characters at most; The page changed alone before).

  • A mail job's progress log says what the mailbox search found (Folder INBOX: 12 new messages,
    3 from the sender and with the subject asked for, or no new messages and how many were done
    with before) and names each message of the sender and subject, with what the job takes from it
    (1 attachment, 2 links) or why nothing (nothing to fetch: no attachment of the names, types
    and sizes asked for, no link the patterns take): up to 50 a run, the rest counted.

  • The app's routes in groups: Try the address, recipe files and templates and the rclone wizard
    in harvester/routes/ (FastAPI's APIRouter), the jobs and the recorder in main.py; the API
    is the same.

  Fixed

  • A long run (a first one of hundreds of files) no longer pushes its own start out of the
    progress log: its first 30 lines stay beside the job's last 500.
  • A video larger than the job's largest file that yt-dlp finds so only as it downloads is skipped
    for good as one known to be too large (yt-dlp saved no file for …: larger than the job's
    largest file), not failed as yt-dlp left 0 files in every run.
  • A release job whose forge answers with an error or cannot be reached fails with its host,
    answer and what was asked (github.com answered 502 for the releases of o/r), or the address
    it could not reach, not with httpx's text; the log says how many releases were looked at and
    how many files match.
  • A gallery job whose gallery-dl lists part of the gallery and then fails (a page of many) says
    so in a warning with gallery-dl's error, and the run no longer counts as a complete pass; the
    log says how many files gallery-dl listed.
  • A run whose listing breaks after files came (a next page that fails, a feed cut short) still
    fails, but its reason keeps what it did: 3 downloaded; then the listing failed: … (only the
    listing's error before).
  • A linked file whose site cannot be reached (a name that does not resolve, a failed connection,
    an error 5xx, a redirect to a private address) no longer fails a mail job's run when nothing
    else came: it is skipped this time with a line in the log (skipped this time in the run's
    outcome), tried again in the next runs and given up after the third.
  • A file's failure in the progress log (file: why) and a selector's repair (level 1: selector
    repaired (… → …)) are translated (German, Polish), as their neighbours were.
  • The usual reasons a file fails or is skipped (… answered 404, … is not of the job's file
    types, … is larger than the job's largest file, … gave only 12 bytes), a run's (and 3
    more) and a file store's configuration errors are translated (German, Polish).
  • The run's reasons the start page has none of the recipe's links and … asks for a person,
    and the log's line of a page that loads none of the types to capture, are translated (German,
    Polish), as their neighbours were.
  • A recorded click that closes its own window (a sign-in popup's Allow) no longer fails its
    step now and then: the window may close before the browser reports the click as done.

  Downloads

  • harvester-0.14.0.tar.gz
    0 downloads · 2026-11-04 17:02:23 +00:00 · 1.7 MiB
  • signature.txt
    0 downloads · 2026-11-04 17:02:23 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.13.2 d3fc4dbc96

  Compare

  v0.13.2 Stable

  dawid released this 2026-11-04 06:36:04 +00:00 | 81 commits to main since this release

  Signed by dawid

  GPG key ID: 4131A40E8F612CB0

  Added

  • Light browser (website jobs; on for new jobs, off for those saved before): the job's
    browser skips images, fonts and videos while it walks the site, each of which came through the
    browser's checks uncached; pictures' links are still found by their addresses. Ignored for a
    disguised browser and for jobs that save pages or capture the files they load. A picture of a
    failed step shows the pages without their images then.

  • Files linked in an e-mail: a mail job's Also the files linked in the message (off by
    default) downloads the files its messages link to ("Download your invoice"), from the HTML
    part's links and the plain text's addresses, of the Link address or Link text pattern
    (one at least), at most 10 a message; never mailto:, tel:, an unsubscribe link or a private
    address. Through the checked download (public addresses only, at most 10 redirects, the job's
    types and largest file), named by Content-Disposition or the final address, known by the
    message's ID and the link's address. A link that gives a web page is sent to the mail job's
    Pages to the on-demand job (an on-demand website job of the person, whose recipe signs in),
    else logged as a page, not a file; such a page and an expired link (404, 410, 401, 403) are
    logged once and skipped for good.

  • Deleted files downloaded again: Download deleted files again (every job; on, also for the
    jobs saved before) has a run ask Nextcloud, before it skips a file the job saved already,
    whether the file is still there, by its ID: one deleted or in the trash is downloaded again
    (Deleted in Nextcloud, downloaded again: …), one moved or renamed counts as there. Only the
    files the listing finds again are asked for, a listing page's at once (one WebDAV SEARCH per
    100 files at most, each once a run); a file without an ID by the path it was saved at. When
    Nextcloud cannot be asked, nothing is downloaded and the log says so. An unpacked ZIP counts as
    there while its archive or its folder is.

  • More consent managers whose cookie banners Dismiss cookie banners answers by their own
    reject buttons: Sourcepoint (its message's frame, never the page itself), CookieYes and its
    older CookieLawInfo, Complianz, Osano, Iubenda, consentmanager.net, Cookie Script, Termly,
    Axeptio, Cookie Notice and Google's Funding Choices (Do not consent); a recorded click on
    one of their banners is optional then.

  • The recorder adds approvals and handovers: Add step: wait for your approval (once the
    approval went through, until the page's address then) and Add step: hand over to you
    where the recording is; the Steps list changes their label, an approval's wait and their
    until (an address's pattern, this page’s address, an element typed or picked with Pick
    an element).

  • Cookie banners dismissed in the progress log: an info line on the page where it happened,
    naming the consent manager (or a cookie banner) and the button clicked, once a manager in a
    run.

  Changed

  • The app asks Nextcloud and its database less: the admin's settings are read each minute with
    one client (a new one each minute before, never closed), a job's account in one request (one
    per field before, twice a run), the weekly tests' times in one query a minute (one per job),
    and the progress log's poll and the jobs' list read the waiting addresses once (twice).
  • The jobs page asks for less: the list comes without the jobs' options (GET /api/jobs?brief=1
    gives each job's own address instead; a recipe may be 100 KB), a job's form asks for its own
    (the new GET /api/jobs/<id>), and the progress log keeps its lines when the list is drawn
    again (a run that starts or ends), asking only for newer ones, and the queue's status from its
    poll. GET /api/jobs without brief still gives every job's options (Talk AI reads them).
  • The recorder's Network panel keeps at most 8 MB of its answers' bodies together (60 MB a
    recording before: 30 answers of up to 2 MB each), dropping the oldest first; their entries
    stay, and the Preview of one dropped says it is no longer kept.
  • A website job with the browser downloads its files through one checked HTTP client for the
    run (its connections kept), instead of a new one for each file (some 30 ms, a new TLS
    handshake, and a copy of the browser's whole session each): each request still gets the
    browser's cookies for its address as they are when it is sent, and every address and redirect
    is checked as before.
  • Listing pages in the browser are read once their level's links are there (a next page once
    they changed), then after two seconds at most for what a script still loads, instead of half a
    second and up to 15 seconds more for the network to go quiet, which a page with analytics or
    polling never does: three such pages took some 48 seconds, now some 7. On such a page a scroll
    or a Load more round waits up to three seconds for the page to grow instead (a feed of
    three rounds: 79 seconds, now 10). A page whose links do not show within five seconds still
    gets the network's whole wait.
  • The browser's screen comes as binary WebSocket messages, each frame the JPEG itself, instead
    of base64 in JSON (a third larger, and parsed): the page shows each by an object URL and lets
    the one before go.
  • The browser's screen in the recorder (and in Continue in the browser and rclone's sign-in)
    takes less of the server's CPU: every second frame of Chromium's (some 30 frames a second)
    instead of every frame, and none while the jobs page is hidden (another tab, a minimized
    window), again at once when it shows. A view left for another one of the app closes its
    browser at once, instead of after 15 minutes without input.
  • An address sent to an on-demand job again brings back its files deleted in Nextcloud since,
    whatever the option, and the log says where each file fetched before is now (Fetched before,
    in your files at …).
  • A deleted file's record of its content (and the duplicates' that named it) goes when the file
    is found gone, so that its download anew is no duplicate of itself; a known file due for its
    check of changes that is gone is downloaded anew instead of staying away.
  • Saving a file asks Nextcloud less: a file under 5 MB is one PUT (larger ones still go in
    chunks), its ID comes from the answer instead of a lookup, and a job's folders are made once a
    run, not for each file (one made again if it is deleted meanwhile): some 2 requests a file
    instead of about 9.
  • The pause between two downloads counts from the end of the site's last answer (a file, an
    error, a check for changes, a sign-in anew), so that saving and tagging the file are part of
    it: the site still sees the whole pause between two downloads, and a run waits about a second
    less a file.
  • The jobs page asks for the progress log every 30 seconds while none of your jobs runs or
    waits (10 before), and not at all while the page is hidden (another tab, a minimized window):
    at once when it shows again.
  • A page's JavaScript in the job's browser and the recorder's has at most 1 GB of memory (some
    3.7 GB before), so that a page that runs away fails by itself rather than taking the memory of
    the other browsers and the app.
  • The image is some 270 MB smaller: without Playwright's headless shell, which nothing starts
    (the browsers are the full Chromium); an empty HARVESTER_BROWSER_CHANNEL means it too.
  • The app's modules are compiled in the image, not again at each start (some 0.25 s).
  • The container's health check runs every 15 seconds instead of every 2 (a shell and pgrep
    each time, for the container's whole life): a deploy waits some 15 seconds for healthy
    (HaRP waits up to 90); failures in the first five minutes do not count, then four in a row
    make the container unhealthy.
  • The jobs' page on a phone (narrower than 700 pixels): each job a card, its columns one below
    the other with their names and its buttons on a line of their own, instead of a table wider
    than the screen; the header's parts one below the other, the progress log's text below its
    time, and a line of an earlier day with a short date and no seconds. The table's last column
    is named Actions for screen readers.
  • The recorder: Test the recipe and Test as the job runs in its toolbar beside Done
    (below the steps and picks before); a test's result starts with its outcome in one line,
    coloured by it, and is scrolled into view. On a narrow window the screen is above the steps at
    the full width (a sliver beside them before), and the toolbar wraps, Done and Cancel together.
  • The job's form shows a source's essentials (a website's recipe, file name, file types and
    files per run; another source's address, file names and what it takes) and folds the other
    options into Browser and connection, Files and sizes, Repeated runs and a mail
    job's Linked files, each opened by itself when one of its options differs from its
    default, saying how many do ((2 changed)); a website job's form was 21 options in a row.
    GET /api/sources gives each option its group (empty: an essential). Save and the form's
    other buttons stay at the window's foot while the form scrolls.
  • The recorder's step buttons are Add step: scroll to load more, Add step: wait for your
    approval and Add step: hand over to you, on a line of their own below the modes; outside
    Record steps they are off and say Choose Record steps to add steps (only a tooltip before).
    The modes are a group named Mode for screen readers, and the Steps list's ↑, ↓ and
    ✕ are buttons of 34 pixels (a few pixels before), its optional kept with its checkbox.
  • The signs of the recorder's step marks (▶, ⏸, ↻, ✓, –, ✗, ↷, ■), of a pattern's or an
    address's check (✓, ✗) and of ✔ will be used are no part of their translated texts: shown
    beside them, not read out by screen readers.
  • Delete, a template's Remove, Back to the jobs with an account typed and the job's browser's
    Into the account ask in the page, in a dialog of Nextcloud's look (the browser's own
    confirm() before); Escape is No, and a deletion's dialog has the focus on Cancel. The
    recorder's Clear picks offers Undo for 15 seconds, and its Cancel asks before it
    drops changes to the recipe.
  • Buttons alike: the actions in a row or a line of text (a waiting address's Cancel, Show
    what the browser saw, a cookie's Show, the Steps list's this page’s address, Pick an
    element, make optional?) are borderless buttons as Nextcloud's tertiary ones (underlined
    links before), and red is kept for what cannot be undone: a pick's, a level's and a value's
    Remove in the recorder are plain buttons (Undo pick takes them back).

  Fixed

  • Gallery jobs (again): urllib3-future, which nc_py_api's niquests brings, is built without
    installing itself as urllib3 (URLLIB3_NO_OVERRIDE), so gallery-dl gets upstream urllib3 again.
    0.13.1's urllib3 2.7.0 did not help: the fork stood in its place, and lacks
    urllib3.connection.create_urllib3_context.
  • Test the recipe in the recorder failed at a recorded click on a cookie banner that Dismiss
    cookie banners had answered already; it skips the click as optional, as a run does.
  • The jobs' states OK, Failed, Sign-in refused, Changed, Needs you and Paused were
    their text colour on their full colour, hard to read in both themes: a light tint of the colour
    behind the text now.
  • The browser views' screen kept the keyboard: Tab, Shift+Tab and Escape went to the site, with
    no way out but the mouse. Escape twice within a second leaves it for the page's next control,
    which a hint says while the screen has the keyboard; the screen's picture is no longer a stop
    of its own for Tab (its hidden keyboard field is).
  • Screen readers: the progress log was a live region read out at every poll, and the queue's
    load line a status; the log's errors and outcomes are announced through one region now, as is
    the recorder's test (its start and outcome), whose result was a new live region each time.
    Errors are alerts, and a refused Save marks the field it names invalid, described by the
    error, its group opened and the keyboard in it.
  • The page was blank until the app gave its sources and templates: it says Loading…. Save says
    Saving… while the app answers, and a mail job's Pages to the on-demand job says when your
    jobs could not be read (an empty list before). A required field's name (a job's Name and
    Folder) is marked, and a field's help is no longer part of its label but its description.

  Downloads

  • harvester-0.13.2.tar.gz
    0 downloads · 2026-11-04 06:42:13 +00:00 · 1.1 MiB
  • signature.txt
    0 downloads · 2026-11-04 06:42:14 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.13.1 4f25da8900

  Compare

  v0.13.1 Stable

  dawid released this 2026-11-03 20:34:36 +00:00 | 123 commits to main since this release

  Signed by dawid

  GPG key ID: 4131A40E8F612CB0

  Added

  • Watching a page: a website recipe's "watch": {} (or {"selector": "…"}) saves no file but
    compares the start page's text (or the selector's) with the last run's, whitespace aside; a
    change is a line in the log and a notification with an excerpt of the lines gone and come (at
    most 300 characters), and the run's outcome says whether the page changed.
  • ZIP files unpacked: Unpack ZIP files (every job, off by default) puts a downloaded .zip
    file's files into a folder of its name beside it, tagged; Keep the ZIP files (on by default)
    keeps the archive too. A suspect archive is saved as it is, the log saying why: a path out of
    its folder or an absolute one, a symbolic link, an encrypted file, two files of one name, more
    than 1000 files, more than the job's largest file in all, more than 100 times its packed size
    (counted as it unpacks). No nested unpacking; .epub, .docx and other ZIPs of another name
    stay files.
  • Duplicates by content: a job keeps the SHA-256 of each file it saves; a file whose content one
    of its files from another address has is not saved again, and the log and the run's outcome
    say so (A duplicate of …, 1 duplicate (not saved again)).
  • Versions: Check known files for changes (website and feed jobs; on for new jobs, off for
    those saved before) asks the addresses of the files a job saved again, each at most once in 20
    hours: over HTTP with If-None-Match and If-Modified-Since (304: no download), else by the
    downloaded file's hash. A changed file is saved over its own, so that Nextcloud keeps the one
    before among its versions; a deleted file is not brought back, a failed check fails nothing.

  Fixed

  • Gallery jobs: urllib3 is held at 2.7.0, since 2.8.0 no longer has the function gallery-dl
    calls (every gallery job failed with AttributeError: module 'urllib3.connection' has no attribute 'create_urllib3_context').

  Downloads

  • harvester-0.13.1.tar.gz
    0 downloads · 2026-11-03 20:47:28 +00:00 · 1.1 MiB
  • signature.txt
    0 downloads · 2026-11-03 20:47:29 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.13.0 fa479de12a

  Compare

  v0.13.0 Stable

  dawid released this 2026-11-03 19:59:06 +00:00 | 130 commits to main since this release

  Signed by dawid

  GPG key ID: 4131A40E8F612CB0

  Added

  • Cookie banners dismissed on their own: a website job's Dismiss cookie banners (on for new
    jobs, off for those saved before) has its browser answer OneTrust, Cookiebot, Usercentrics,
    Didomi, Quantcast (TCF), TrustArc, Borlabs and Klaro with "Reject all", else "Only necessary",
    and a button of exactly such a text (German, English, Polish) in a dialog or a floating banner
    that speaks of cookies; in the steps and the walk, in frames and shadow roots, and in the
    recorder's browser, which records no step of it. Nothing else is clicked; a recorded click on a
    banner is optional then.
  • Approvals in an app: the step {"action": "approve", "until": {"url", "selector"}, "label", "wait"} shows Needs you with its label and notifies the job's person, then waits until the
    page shows until (an address's pattern, an element, or both) that the approval went through,
    or fails after its wait; Stop ends it, and a page that shows until already asks nothing.
  • Handing over to the person: the step {"action": "handover", "label"} stops the run as a page
    asking for a person does, keeping the browser's session as it was there: Continue in the
    browser opens the page, the person does what only they can (a passkey), and Done runs the
    job, whose kept session skips the sign-in. Its optional until skips it when the page shows
    it already; a recipe with a handover needs its start. Approvals and handovers end the sign-in
    a kept session skips.
  • Codes from the phone: the step {"action": "ask-code", "selector", "label", "wait"} pauses
    the run at a code Harvester cannot read itself (an SMS, a call, an authenticator it has no
    secret of): the job shows Needs you with a field Enter the code from … on the jobs page
    and notifies its person, types the code sent from there into the field and goes on; after the
    wait (10 minutes by default, 30 s to 30 minutes) the run fails cleanly, and Stop ends the wait.
    The code goes only to the job's own run while it asks (POST /api/jobs/<id>/code) and is never
    logged or kept. The recorder's Steps list offers code from the phone? on a typed code, and
    Test the recipe asks for the code below the steps.
  • Exports a site prepares first: a level's or the file links' "wait_for": {"selector", "timeout", "reload"} looks for the element of the ready file on each page (reloading it every
    reload seconds if asked) for up to 30 minutes, then reads and downloads its links as before,
    clicked ones too.
  • Choices: the step each goes through every option of a <select> or a custom dropdown, or a
    fixed list of values (every account, year, child), doing the steps after it and the walk for
    each, the choice's text {choice} in the file names, tags, key and addresses. The recorder's
    Steps list offers Repeat for each option on a recorded choice, and Test the recipe lists
    the choices it goes through.
  • JSON listings asked for with POST: a form's or a JSON body (GraphQL with its variables), whose
    pages are counted in the body or chained by a cursor the answer gives (endCursor with
    hasNextPage, nextCursor, an answer's next page's address); the headers the page sent that
    matter, a token that changes read when the job runs from a cookie, a <meta> or the page's
    storage, never written into the recipe. The Network panel lists POST answers too, with a
    GraphQL operation's name, and proposes all of it from one.
  • The recorder's Network panel shows an answer's content: Preview beside each one, its JSON
    indented (the first 20 000 characters).
  • File links that are clicked ("click": true) give the file also when the click opens a tab
    or popup that shows it (a PDF in the viewer, a picture) or starts its download, or takes its
    own tab to it; checked as any download, never a web page, and the tab goes back to the listing.
  • Levels to follow that are clicked ("click": true): rows, cards or buttons without an address
    that open an entry. A run clicks each, reads the next level where it lands (another address, a
    new tab or popup, a single-page app's route, a dialog) and returns to the listing (Back,
    Escape, else the listing opened again with its loading of more); a click that fails is a gap.
  • The recorder picks elements without an address (buttons, links to javascript:, rows and cards
    a script opens) as file links or links to follow that are clicked: their pattern proposed on
    their texts, the panel saying the job clicks them, Click in the browser to switch it; Test
    the recipe walks such levels by clicking and lists clicked file links by their text.
  • Pages saved as files: the file links' "save": "pdf", "html" or "markdown" opens each page
    they lead to in the job's browser (also what an element clicked shows), settles and scrolls it,
    and saves it: Chromium's print (A4), one HTML file with its stylesheets, pictures and frames
    inside (Chromium's MHTML snapshot, its resources as data: addresses), or its main content as
    Markdown (Readability's way, on BeautifulSoup). Named by {title}, with the extension; checked
    as a download (the job's file types must take it, markdown a new one). A recipe's own
    "save" saves its start page; the template Save a page does it for the pages sent to an
    on-demand job (Fetch a URL…, Send to Harvester). The recorder's file links pick has Save
    the pages as, and Test the recipe lists the pages it would save.
  • Files a page only shows: the file links' "capture": ["application/pdf", "image/*"] keeps the
    files of those types each page loads (a viewer's PDF, pictures), embeds (<embed>, <object>,
    frames: fetched with the browser's cookies) or holds as blob: addresses (read in the page); at
    most 20 a page and 64 MB each, checked as downloads; a page whose files were all placed is not
    opened again.
  • Pictures picked as file links (img, picture) are taken at their full size: the largest
    candidate of srcset and <source>, an attribute naming the full size, the lazy-loading ones
    (data-src, data-original, …) over a placeholder src; in the browser, over HTTP and in the
    recorder.

  Changed

  • A level's pick tells a link's address apart from # and javascript: (no address).
  • Releases through the shared action nextcloud-app-release of dawid/ci-actions: the release's
    text is the version's section of this changelog, and a tag without one fails.

  Downloads

  • harvester-0.13.0.tar.gz
    0 downloads · 2026-11-03 20:11:59 +00:00 · 1 MiB
  • signature.txt
    0 downloads · 2026-11-03 20:11:59 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.12.0 dddae0f133

  Compare

  v0.12.0 Stable

  dawid released this 2026-11-03 10:19:55 +00:00 | 149 commits to main since this release

  Signed by dawid

  GPG key ID: 4131A40E8F612CB0

  Downloads

  • harvester-0.12.0.tar.gz
    0 downloads · 2026-11-03 13:14:47 +00:00 · 983 KiB
  • signature.txt
    0 downloads · 2026-11-03 13:14:48 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.11.0 63a3e251ec

  Compare

  v0.11.0 Stable

  dawid released this 2026-11-03 04:44:52 +00:00 | 162 commits to main since this release

  Downloads

  • harvester-0.11.0.tar.gz
    0 downloads · 2026-11-03 10:21:21 +00:00 · 287 KiB
  • signature.txt
    0 downloads · 2026-11-03 10:21:23 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.10.0 6652ee9753

  Compare

  v0.10.0 Stable

  dawid released this 2026-11-02 17:57:17 +00:00 | 200 commits to main since this release

  Downloads

  • harvester-0.10.0.tar.gz
    0 downloads · 2026-11-03 10:15:01 +00:00 · 181 KiB
  • signature.txt
    0 downloads · 2026-11-03 10:15:02 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads
• v0.9.0 89ba28a27d

  Compare

  v0.9.0 Stable

  dawid released this 2026-11-02 14:10:57 +00:00 | 213 commits to main since this release

  Downloads

  • harvester-0.9.0.tar.gz
    0 downloads · 2026-11-03 11:20:44 +00:00 · 170 KiB
  • signature.txt
    0 downloads · 2026-11-03 11:20:46 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads