Adding a New Site
vi scraper-config.json
{
"magnetDL": {
"name": "magnetDL",
"url": "https://www.magnetdl.org/search/?q={{query}}&m=1&x=0&y=0",
"list": "table[class=download] tbody tr",
"result": {
"name": [ "td[class=n]", "a[href]", "@title" ],
"url": [ "td[class=n]", "a[href]", "@href" ],
"magnet": [ "td[class=m]", "a[href]", "@href" ],
"size": "td:nth-child(6)",
"seeds": "td[class=s]"
}
},
...
}
Push to Github
Updating Docker Container Scraping Config
Update docker container config file located in /
vi cloud-torrent.yaml
Revise scraperurl
... scraperurl: https://raw.githubusercontent.com/xxx/simple-torrent/master/scraper-config.json ...
Other Torrent Sites
| Site | URL | Notes |
|---|---|---|
| idope.se | https://idope.se/torrent-list/{{query}}/?p={{page:1}} | Uses Cloudflare |
| eztv | https://eztv.re/search/{{query}} | Uses Cloudflare |
| torrentgalaxy | https://tgx.rs/torrents.php?search={{query}}&lang=0&nox=2&nowildcard=1" | Uses Cloudflare |
Testing a Scraping Configuration
Using jpillora/scraper https://github.com/jpillora/scraper/tree/master
Install scraper binary
curl https://i.jpillora.com/scraper | bash sudo mv ./scraper /usr/local/bin
Create a Scraping Configuration
vi config.json
{
"/search": {
"url": "https://www.magnetdl.org/search/?q={{query}}&m=1&x=0&y=0",
"list": "table[class=download] tbody tr",
"result": {
"name": [ "td[class=n]", "a[href]", "@title" ],
"magnet": [ "td[class=m]", "a[href]", "@href" ],
"url": [ "td[class=n]", "a[href]", "@href" ],
"seeders": "td[class=s]",
"size": "td:nth-of-type(6)"
}
}
}
Start Scrape Server
scraper config.json 2024/02/14 12:36:47 [scraper] Loaded endpoint: /search 2024/02/14 12:36:47 [scraper] Listening on 3000...
Attempt Scrape
curl "localhost:3000/search?query=halo"
Validate Output
[
{
"magnet": "magnet:?xt=urn:btih:7161c3a3f7068b9784fba7f722b09bb8272e9996&dn=Halo.S02E02.XviD-AFG+%5BTGx%5D&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451&tr=udp://tracker.internetwarriors.net:1337&tr=udp%3A%2F%2Fexodus.desync.com%3A6969",
"name": "Halo.S02E02.XviD-AFG [TGx]",
"seeders": "30",
"size": "276.24 MB",
"url": "/file/5926991/halo.s02e02.xvid-afg-tgx/"
},
...
]