Adding a New Site

vi scraper-config.json

{
  "magnetDL": {
    "name": "magnetDL",
    "url": "https://www.magnetdl.org/search/?q={{query}}&m=1&x=0&y=0",
    "list": "table[class=download] tbody tr",
    "result": {
      "name": [ "td[class=n]", "a[href]", "@title" ],
      "url": [ "td[class=n]", "a[href]", "@href" ],
      "magnet": [ "td[class=m]", "a[href]", "@href" ],
      "size": "td:nth-child(6)",
      "seeds": "td[class=s]"
    }
  },
...
}


Push to Github


Updating Docker Container Scraping Config

Update docker container config file located in /

vi cloud-torrent.yaml


Revise scraperurl

...
scraperurl: https://raw.githubusercontent.com/xxx/simple-torrent/master/scraper-config.json
...


Other Torrent Sites

SiteURLNotes
idope.sehttps://idope.se/torrent-list/{{query}}/?p={{page:1}}Uses Cloudflare
eztv https://eztv.re/search/{{query}}Uses Cloudflare
torrentgalaxy https://tgx.rs/torrents.php?search={{query}}&lang=0&nox=2&nowildcard=1"Uses Cloudflare



Testing a Scraping Configuration

Using jpillora/scraper https://github.com/jpillora/scraper/tree/master


Install scraper binary

curl https://i.jpillora.com/scraper | bash
sudo mv ./scraper /usr/local/bin


Create a Scraping Configuration

vi config.json

{
  "/search": {
    "url": "https://www.magnetdl.org/search/?q={{query}}&m=1&x=0&y=0",
    "list": "table[class=download] tbody tr",
    "result": { 
      "name": [ "td[class=n]", "a[href]", "@title" ],
      "magnet": [ "td[class=m]", "a[href]", "@href" ],
      "url": [ "td[class=n]", "a[href]", "@href" ],
      "seeders": "td[class=s]",
      "size": "td:nth-of-type(6)"
    }
  }
}


{
  "/search": {
    "url": "https://bitlordsearch.com/search?q={{query}}",
    "list": "table.table-torrents tbody tr",
    "result": {
      "name": "td.name span.title",
      "magnet": ["a.magnet-button","@href"],
      "url": ["td.name span.title a.btn","@href"],
      "size": "td.size",
      "seeds": "td.seeds",
      "peers": "td.peers"
    }
  }
}



Start Scrape Server

scraper config.json
2024/02/14 12:36:47 [scraper] Loaded endpoint: /search
2024/02/14 12:36:47 [scraper] Listening on 3000...


Attempt Scrape

curl "localhost:3000/search?query=halo"


Validate Output

[
  {
    "magnet": "magnet:?xt=urn:btih:7161c3a3f7068b9784fba7f722b09bb8272e9996&dn=Halo.S02E02.XviD-AFG+%5BTGx%5D&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451&tr=udp://tracker.internetwarriors.net:1337&tr=udp%3A%2F%2Fexodus.desync.com%3A6969",
    "name": "Halo.S02E02.XviD-AFG [TGx]",
    "seeders": "30",
    "size": "276.24 MB",
    "url": "/file/5926991/halo.s02e02.xvid-afg-tgx/"
  },
...
]


Copyright JMEHAN.COM