Supabase Backend as a Service · Lesson

Task Queues with Supabase & Workers

Explore strategies for implementing background task processing and message queues using Supabase and external worker services.

Task Queues with Supabase & Workers is a free Supabase Backend as a Service lesson on CoddyKit — lesson 2 of 3. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Supabase Backend as a Service learning path, one of 3 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Tasks: Background vs. Foreground

Imagine your app needs to send a welcome email, process a large image, or generate a report. If your user has to wait for these long tasks to finish, their experience suffers.

These are background tasks: operations that don't need immediate user interaction and can run independently without blocking the user interface.

Introducing Task Queues

A task queue (or message queue) is a system that allows different parts of your application to communicate asynchronously. It acts like a temporary holding area for tasks.

Producers: Add tasks to the queue.
Consumers (Workers): Pick up tasks from the queue and process them.

This decouples the task creation from its execution.

Why Use a Task Queue?

Task queues bring several key benefits to your application architecture:

Improved Responsiveness: Users don't wait for long operations.
Scalability: You can add more workers to handle increased load.
Reliability: Tasks can be retried if they fail.
Decoupling: Separates task submission from task execution.

Supabase as a Simple Queue

While Supabase isn't a dedicated message queue, its powerful PostgreSQL database can serve as a simple task queue for many use cases. We can create a dedicated table to store tasks.

A typical tasks table might have columns like:

id (Primary Key)
payload (JSONB, for task data)
status (e.g., 'pending', 'processing', 'completed', 'failed')
created_at (Timestamp)
processed_at (Timestamp, nullable)

Enqueueing Tasks (Producer)

To enqueue a task, your client-side application or API endpoint simply inserts a new row into the tasks table with a 'pending' status. The payload column holds all the necessary data for the worker to process.

For example, to send a welcome email after user signup:

import { createClient } from '@supabase/supabase-js';

const supabaseUrl = 'YOUR_SUPABASE_URL';
const supabaseKey = 'YOUR_SUPABASE_ANON_KEY';
const supabase = createClient(supabaseUrl, supabaseKey);

async function enqueueWelcomeEmail(userId, email) {
  const { data, error } = await supabase
    .from('tasks')
    .insert({
      type: 'send_welcome_email',
      payload: { userId, email },
      status: 'pending'
    });

  if (error) {
    console.error('Error enqueueing task:', error.message);
  } else {
    console.log('Welcome email task enqueued!');
  }
}

// Example usage (not runnable directly without Supabase setup)
// enqueueWelcomeEmail('user123', 'test@example.com');

Introducing External Workers

An external worker is a separate application or service that continuously monitors the task queue for new jobs. When it finds a 'pending' task, it picks it up, processes it, and updates its status.

Workers can be written in any language (Node.js, Python, Go) and run on various platforms (servers, serverless functions, containers). They connect to your Supabase database to read and update task records.

Dequeueing Tasks (Consumer/Worker)

A worker needs to:

Fetch 'pending' tasks.
Mark a task as 'processing' to prevent other workers from picking it up.
Execute the task logic using the payload.
Update the task status to 'completed' or 'failed'.

Workers can either poll (periodically check) or use Supabase's Realtime feature to listen for new task insertions.

Code: Simple Worker Logic (Polling)

Here's a simplified Node.js worker logic that polls the Supabase tasks table every few seconds. Remember to replace placeholder values.

// worker.js
const { createClient } = require('@supabase/supabase-js');

const SUPABASE_URL = 'YOUR_SUPABASE_URL';
const SUPABASE_KEY = 'YOUR_SUPABASE_SERVICE_ROLE_KEY'; // Use service key for workers
const supabase = createClient(SUPABASE_URL, SUPABASE_KEY);

async function processTask(task) {
  console.log(`Processing task ${task.id}: ${task.type}`);
  // Simulate work (e.g., sending email, processing image)
  await new Promise(resolve => setTimeout(resolve, 2000));
  console.log(`Task ${task.id} processed.`);
  // In a real app, this would involve calling an email API etc.
}

async function runWorker() {
  console.log('Worker started, looking for tasks...');
  setInterval(async () => {
    try {
      // Fetch one pending task and lock it by updating status
      const { data: tasks, error } = await supabase
        .from('tasks')
        .select('*')
        .eq('status', 'pending')
        .order('created_at', { ascending: true })
        .limit(1);

      if (error) throw error;

      if (tasks.length > 0) {
        const task = tasks[0];
        // Atomically update status to 'processing'
        const { error: updateError } = await supabase
          .from('tasks')
          .update({ status: 'processing', processed_at: new Date().toISOString() })
          .eq('id', task.id)
          .eq('status', 'pending'); // Ensure no other worker picked it up

        if (updateError) {
          if (updateError.code === '40600') { // Row already updated by another process
            console.log(`Task ${task.id} already picked up.`);
            return; 
          }
          throw updateError;
        }

        await processTask(task);

        // Update status to 'completed'
        await supabase
          .from('tasks')
          .update({ status: 'completed' })
          .eq('id', task.id);

      } else {
        // console.log('No pending tasks.');
      }
    } catch (err) {
      console.error('Worker error:', err.message);
      // Implement error handling, e.g., update task status to 'failed'
    }
  }, 5000); // Poll every 5 seconds
}

// To run this: node worker.js
// (Requires 'npm install @supabase/supabase-js' and a Supabase project)
runWorker();

Task States & Reliability

Properly managing task states is crucial for reliability:

Pending: Task is waiting to be processed.
Processing: Worker has picked up the task.
Completed: Task finished successfully.
Failed: Task encountered an error.

For failed tasks, you might implement retries (e.g., after a delay) and ensure tasks are idempotent (running them multiple times has the same effect as running once) to prevent unintended side effects.

Queueing Question

You've learned about using Supabase and external workers for task queues. Consider a scenario where a user uploads a large video file that needs transcoding.

Recap: Task Queues & Workers

We've explored how task queues enable efficient background processing in your applications. By using Supabase as a simple queue and external workers, you can:

Offload long-running operations.
Improve user experience by keeping your app responsive.
Build more scalable and robust systems.

This pattern is fundamental for building complex, high-performance web and mobile applications.

Frequently asked questions

Is the “Task Queues with Supabase & Workers” lesson free?

Yes — the full text of “Task Queues with Supabase & Workers” is free to read here on the web, and the Supabase Backend as a Service course includes 3 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Supabase Backend as a Service course, upgrade to CoddyKit PRO.

What will I learn in “Task Queues with Supabase & Workers”?

Explore strategies for implementing background task processing and message queues using Supabase and external worker services. You practise Supabase Backend as a Service with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Supabase Backend as a Service?

No prior experience is required. Supabase Backend as a Service on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 3, so you can start here or from the beginning and move at your own pace.

How long does the “Task Queues with Supabase & Workers” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Supabase Backend as a Service lesson?

Yes. Every Supabase Backend as a Service lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Supabase Backend as a Service