Getting Started
Overview Installation Quick Start
Features
Command Line Flags Media Types Browser Integration Job Management Scheduling Event Hooks Content Extraction Tagging System
Interfaces
Web Interface API Reference
Configuration
Configuration File Data Sources Storage Settings
Misc
Test Links Interesting Sites Related Tools

API Reference

GatherHub provides a comprehensive RESTful API for programmatic control over the application. This enables automation, integration with other systems, and building custom interfaces.

Table of Contents

API Configuration

The API server is configured in the config.toml file: 

[api]
enabled = true
host = '127.0.0.1'
port = 5000
debug = false

[api.auth]
enabled = true
api_secret = 'your-api-key'
apiKey_expiry_hours = 24

To disable authentication for development or trusted environments, set api.auth.enabled to false.

Warning: It's recommended to keep authentication enabled and to use a strong, unique secret.

Starting the API Server

To start the API server, use the --api flag: 

./gatherhub --api

You can also run the API server alongside other components: 

./gatherhub --api --web --daemon

Authentication

When authentication is enabled, all API requests require a API Key in the Authorization header: 

GET /api/jobs
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...

The authentication implementation uses simple API Key validation based on the configured secret key. The token is required for all endpoints when authentication is enabled.

Core API Endpoints

The following endpoints are defined in the API server (internal/api/server.go). These endpoints provide the essential functionality of GatherHub's API.

Job Management

Endpoint Method Description
/api/jobs GET Get all jobs (up to 1000)
/api/jobs POST Add a new job
/api/jobs/{id} GET Get job by ID
/api/jobs/{id} PUT Update job status and error message
/api/jobs/{id} DELETE Delete job

Get All Jobs

GET /api/jobs

Response (array of job objects):

[
  {
    "id": 1,
    "source_id": "bookmark123",
    "source_name": "firefox-bookmarks",
    "url": "https://example.com",
    "media_type": "html",
    "status": "completed",
    "file_path": {"String": "./downloads/html/example_com_1.html", "Valid": true},
    "created_at": "2023-05-10T14:00:00Z",
    "updated_at": "2023-05-10T14:05:20Z",
    "attempts": 1,
    "error": {"String": "", "Valid": false}
  },
  ...
]

Add New Job

POST /api/jobs
Content-Type: application/json

{
  "url": "https://github.com/username/repository",
  "source_name": "manual",
  "source_id": "user_123",
  "media_type": "git"
}

Response (created job object):

{
  "id": 42,
  "source_id": "user_123",
  "source_name": "manual",
  "url": "https://github.com/username/repository",
  "media_type": "git",
  "status": "pending",
  "created_at": "2023-05-10T15:00:00Z",
  "updated_at": "2023-05-10T15:00:00Z",
  "attempts": 0,
  "error": {"String": "", "Valid": false}
}

Get Job by ID

GET /api/jobs/42

Response (job object):

{
  "id": 42,
  "source_id": "user_123",
  "source_name": "manual",
  "url": "https://github.com/username/repository",
  "media_type": "git",
  "status": "completed",
  "file_path": {"String": "./downloads/git/repository_42", "Valid": true},
  "created_at": "2023-05-10T15:00:00Z",
  "updated_at": "2023-05-10T15:05:30Z",
  "attempts": 1,
  "error": {"String": "", "Valid": false}
}

Update Job Status

PUT /api/jobs/42
Content-Type: application/json

{
  "status": "failed",
  "error": "Connection timeout"
}

Response (updated job object):

{
  "id": 42,
  "source_id": "user_123",
  "source_name": "manual",
  "url": "https://github.com/username/repository",
  "media_type": "git",
  "status": "failed",
  "file_path": {"String": "", "Valid": false},
  "created_at": "2023-05-10T15:00:00Z",
  "updated_at": "2023-05-10T15:10:00Z",
  "attempts": 1,
  "error": {"String": "Connection timeout", "Valid": true}
}

Delete Job

DELETE /api/jobs/42

Response: HTTP 204 No Content

Bulk Operations

Endpoint Method Description
/api/bulk/jobs POST Perform operations on multiple jobs

Bulk Job Operations

POST /api/bulk/jobs
Content-Type: application/json

{
  "action": "delete",
  "ids": [1, 2, 3, 4]
}

Response:

{
  "success": true,
  "deleted": 3,
  "failed": 1,
  "message": "Deleted 3 jobs, 1 failed"
}

The action field can be one of:

  • delete - Delete multiple jobs
  • retry - Retry multiple jobs (set status to "pending")

System Operations

Endpoint Method Description
/api/scan POST Scan sources for new content
/api/download POST Process pending downloads
/api/retry POST Retry failed downloads
/api/clean POST Clean failed and old jobs
/api/stats GET Get system statistics

Scan Sources

POST /api/scan

Response: HTTP 200 OK

Scan completed

Process Downloads

POST /api/download

Response: HTTP 200 OK

Download processing completed

Retry Failed Jobs

POST /api/retry

Response: HTTP 200 OK

Retry completed

Clean Jobs

POST /api/clean

Response: HTTP 200 OK

Cleaned 5 failed jobs and 10 old jobs

Get System Statistics

GET /api/stats

Response:

{
  "status_counts": {
    "pending": 5,
    "downloading": 2,
    "completed": 120,
    "failed": 8
  },
  "media_type_counts": {
    "html": 80,
    "videos": 30,
    "git": 15,
    "images": 10
  },
  "total_count": 135
}

File Operations

Endpoint Method Description
/api/upload POST Upload files directly

Upload Files

POST /api/upload
Content-Type: multipart/form-data

file: [file data]
media_type: pdf (optional)

Response:

{
  "success": true,
  "message": "2 of 2 files uploaded successfully",
  "jobs": [
    {
      "id": 157,
      "url": "file:///path/to/uploaded/file1.pdf",
      "source_name": "Manual Upload",
      "source_id": "upload",
      "media_type": "pdf",
      "status": "completed"
    },
    {
      "id": 158,
      "url": "file:///path/to/uploaded/file2.jpg",
      "source_name": "Manual Upload",
      "source_id": "upload",
      "media_type": "jpg",
      "status": "completed"
    }
  ],
  "job": {
    "id": 157,
    "url": "file:///path/to/uploaded/file1.pdf",
    "source_name": "Manual Upload",
    "source_id": "upload",
    "media_type": "pdf",
    "status": "completed"
  }
}

Error Handling

The API returns appropriate HTTP status codes:

  • 200 OK - Successful request
  • 204 No Content - Successful deletion
  • 400 Bad Request - Invalid request parameters
  • 401 Unauthorized - Authentication required
  • 404 Not Found - Resource not found
  • 405 Method Not Allowed - Invalid HTTP method
  • 500 Internal Server Error - Server-side error

Error responses include a message: 

Failed to decode job: unexpected EOF

API Usage Examples

Scanning and Downloading

To scan for new content and then download it: 

# Scan sources
curl -X POST http://localhost:5000/api/scan

# Process pending downloads
curl -X POST http://localhost:5000/api/download

Adding and Managing Jobs

# Add a new job
curl -X POST http://localhost:5000/api/jobs \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "media_type": "html"}'

# Get all jobs
curl -X GET http://localhost:5000/api/jobs

# Delete a job
curl -X DELETE http://localhost:5000/api/jobs/42

Bulk Operations

# Delete multiple jobs
curl -X POST http://localhost:5000/api/bulk/jobs \
  -H "Content-Type: application/json" \
  -d '{"action": "delete", "ids": [1, 2, 3, 4]}'

# Retry multiple jobs
curl -X POST http://localhost:5000/api/bulk/jobs \
  -H "Content-Type: application/json" \
  -d '{"action": "retry", "ids": [5, 6, 7, 8]}'

File Upload

# Upload a file
curl -X POST http://localhost:5000/api/upload \
  -F "file=@/path/to/document.pdf" \
  -F "media_type=pdf"
Note: When using the API with authentication enabled, add the Authorization: Bearer YOUR_TOKEN header to all requests.
Additional Information: The above endpoints are implemented in internal/api/server.go. There are additional API endpoints available in internal/web/server.go that provide extended functionality.
Search Results

Type to search documentation...