Export

Overview

The export API lets you extract large volumes of search results from Profundis. Unlike the standard search endpoints that return paginated results, exports run asynchronously and produce a downloadable file once completed.

The workflow is straightforward:

1. (Optional) Estimate the export to preview the quota impact ;
2. Create an export job ;
3. Poll its status until it reaches completed ;
4. Download the resulting file.

You can export from any search type: hosts, dns, whois or vhosts. The output format can be json, csv or txt.

warning

Exports require a Premium subscription or above. Each exported result is deducted from your monthly results quota. If your remaining quota is insufficient to cover the full export, the results will be truncated.

tip

Before using the example commands below, define your API key in a variable. Replace xxx by the correct value. See how to retrieve your API key here.

API_KEY=xxx

Estimate an export

Before creating an export, you can check how many results match your query and how it will impact your quota. This is useful to avoid wasting your quota on a large export you didn't anticipate.

curl "https://api.profundis.io/api/v2/common/exports/hosts/estimate" \
  -H "X-API-KEY: $API_KEY" \
  -H "content-type: application/json" \
  -X POST \
  -d '{
    "raw_query": "host:*.example.com",
    "time_frame": "all_time"
  }'

The response tells you the estimated result count, your current quota usage, and what the quota will look like after the export. If the export would exceed your remaining quota, will_be_partial is true and partial_limit indicates the maximum number of results you will actually get.

{
  "estimated_count": 5000,
  "quota_current": {
    "results_used": 15000,
    "results_max": 100000
  },
  "quota_after_export": {
    "results_used": 20000,
    "results_max": 100000
  },
  "will_be_partial": false,
  "estimated_file_size_bytes": 1000000
}

Replace hosts in the URL by dns, whois or vhosts depending on the search type you want to estimate.

Create an export

The following command creates an asynchronous export job. The export runs in the background; you will need to poll its status to know when it's done.

curl "https://api.profundis.io/api/v2/common/exports/hosts" \
  -H "X-API-KEY: $API_KEY" \
  -H "content-type: application/json" \
  -X POST \
  -d '{
    "raw_query": "host:*.example.com",
    "time_frame": "all_time",
    "format": "json"
  }'

The format parameter accepts json, csv or txt. The response contains the export ID that you will use in all subsequent calls.

{
  "export_id": "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4"
}

Poll export status

Use the export ID to check the current status. Poll this endpoint periodically until the status is completed (or failed).

curl "https://api.profundis.io/api/v2/common/exports/a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4/poll" \
  -H "X-API-KEY: $API_KEY"

The response includes all the details about the export job.

{
  "id": "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4",
  "status": "completed",
  "search_type": "hosts",
  "raw_query": "host:*.example.com",
  "format": "json",
  "results_count": 1500,
  "file_size_bytes": 102400,
  "original_size_bytes": 512000,
  "estimated_count": 2000,
  "was_truncated": false,
  "created_at": "2024-01-15T10:30:00Z",
  "expires_at": "2024-01-22T10:30:00Z"
}

The possible values for status are: pending, processing, completed, failed, cancelled.

If the export was truncated because of insufficient quota or a timeout, was_truncated is true and truncation_reason indicates the cause (quota_exhausted or timeout).

Download the file

Once the status is completed, you can download the file.

curl "https://api.profundis.io/api/v2/common/exports/a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4/download" \
  -H "X-API-KEY: $API_KEY" \
  -o export.json

The server stores files in gzip format. If your client sends Accept-Encoding: gzip, the compressed file is streamed directly. Otherwise the server decompresses it on the fly before sending it.

List all exports

You can retrieve all your exports at once. They are ordered by creation date, newest first.

curl "https://api.profundis.io/api/v2/common/exports" \
  -H "X-API-KEY: $API_KEY"

Delete an export

To delete a single export, use its ID. If the export is still pending or processing, it will be cancelled if not already running. If it is already completed, failed or cancelled, it will be deleted along with its associated file.

curl "https://api.profundis.io/api/v2/common/exports/a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4" \
  -H "X-API-KEY: $API_KEY" \
  -X DELETE

To delete all your exports at once:

curl "https://api.profundis.io/api/v2/common/exports" \
  -H "X-API-KEY: $API_KEY" \
  -X DELETE

This returns the number of deleted exports.

{
  "deleted_count": 5
}

Frequent questions

What happens when my quota runs out during an export?

The export continues but is truncated. The results you already got are kept in the file, and the response will show was_truncated: true with truncation_reason: "quota_exhausted". You will still be able to download the partial file.

How long are export files available?

Each export has an expires_at timestamp visible in the poll response. After that date, the file is automatically cleaned up from our servers.

Can I run multiple exports at the same time?

Yes. Each export runs independently. You can create several export jobs and poll them individually.

What search types are supported?

The export API supports the same search types as the standard search endpoints: hosts, dns, whois and vhosts. Replace the {searchType} segment in the URL accordingly.