Write Something!

Blog


Request consultation in Tela!

codomari · couchdb · couchbeam · poolboy · pool · worker

Codomari #6 (database/couchdb)

I found this task to be interesting and enjoyably challenging.

Reason for this is that plenty of CouchDB libraries in Hex.pm are outdated or not documented or available as Erlang packages - needs a bit understanding of how it works.

After checking packages in Hex.pm sorted by last update I’ve stopped between CouchX and Couchbeam.

Both of them are regularly updated, but:

  1. CouchX as written “Limited CouchDb Adapter for Ecto” which means may not have some features which CouchDB provides + I don’t want to stick my logic around Ecto capabilities.
  2. Couchbeam is “Erlang CouchDB client” which means is available to use since it’s in Hex.pm but docs are in maintainer’s hosting

So I decided to go with (2) to be able to extend raw features with my own ones.

So let’s dive in!

Couchbeam (installation)

Installation of Couchbeam by adding to dependency in mix.exs:

def deps do 
  ...
  {:couchbeam, "~> 1.5.3"},
  ...
end

I’ll cut it here about Couchbeam for Poolboy

Poolboy

It’s library to create pool of workers, it’s kinda supervisor of workers which makes sure there always be active worker when required.

Install it like this:

def deps do
  ...
  {:poolboy, "~> 1.5.1"},
  ...
end

Based on documentation which is not by maintainer :) (cause it’s also from Erlang world) simple worker looks like this:

defmodule CodomariBackend.GreetingsWorker do
  use GenServer

  def start_link(_) do
    GenServer.start_link(__MODULE__, nil)
  end

  def init(_) do
    {:ok, nil}
  end

  def handle_call({:greet, name}, _from, state) do
    IO.puts("process #{inspect(self())} handled greeting method call")
    Process.sleep(1000)
    {:reply, ~s(Hello #{name}), state}
  end
end

now let’s register it in supervisor tree using Poolboy:

defmodule CodomariBackend.Application do
  @moduledoc false

  use Application

  alias :poolboy, as: Poolboy

  @impl true
  def start(_type, _args) do
    ...

    children = [
      ...
      Poolboy.child_spec(:worker, [
        name: {:local, :worker},
        worker_module: CodomariBackend.GreetingsWorker,
        size: 5,
        max_overflow: 2
      ]),
      ...
    ]

    opts = [strategy: :one_for_one, name: CodomariBackend.Supervisor]
    Supervisor.start_link(children, opts)
  end

  ...
end

Why Poolboy and Couchbeam?

Cause we need connection reliability, making sure that there will be many threads serving connections.

This gives ability to balance between active workers and if one worker crashes = new one is created cause of GenServer and supervision.

Implementing Couchbeam database worker for Poolboy

Unfortunately I could not find any information about connection pool in Couchbeam manual so I came to idea of implementing it myself using worker pattern for Poolboy.

So I wrote CodomariBackend.Database.PoolWorker:

defmodule CodomariBackend.Database.PoolWorker do
  @moduledoc """
  Worker process for managing a CouchDB connection.
  """

  use GenServer

  alias :couchbeam, as: Couchbeam

  def start_link(_) do
    GenServer.start_link(__MODULE__, nil)
  end

  def init(_) do
    %{conn: conn, db: db} = connect()

    {:ok, %{conn: conn, db: db}}
  end

  def connect do
    config = Application.get_env(:codomari_backend, :couchdb)

    host = Keyword.fetch!(config, :host)
    port = Keyword.fetch!(config, :port)
    username = Keyword.get(config, :username, nil)
    password = Keyword.get(config, :password, nil)

    conn = open_connection(host, port, username, password)
    {:ok, db} = open_database(conn)

    %{conn: conn, db: db}
  end

  def open_connection(host, port, nil, nil) do
    Couchbeam.server_connection(
      ~s(http://#{host}:#{port}),
      []
    )
  end

  def open_connection(host, port, username, password) do
    Couchbeam.server_connection(
      ~s(http://#{host}:#{port}),
      basic_auth: {username, password}
    )
  end

  def open_database(conn) do
    config = Application.get_env(:codomari_backend, :couchdb)

    database = Keyword.fetch!(config, :database)

    Couchbeam.open_db(conn, database)
  end

  def handle_call(:get_connection, _from, state) do
    {:reply, state[:conn], state}
  end

  def handle_call(:get_db, _from, state) do
    {:reply, state[:db], state}
  end

  def handle_call(:server_info, _from, state) do
    {:ok, server_info} = Couchbeam.server_info(state[:conn])
    {:reply, server_info, state}
  end
end

and registered in supervision tree:

defmodule CodomariBackend.Application do
  @moduledoc false

  use Application

  alias :poolboy, as: Poolboy

  @impl true
  def start(_type, _args) do
    database_pool_config = Application.get_env(:codomari_backend, :couchdb_pool)

    children = [
      ...
      # DB Pool
      Poolboy.child_spec(database_pool_config[:name], database_pool_config)
    ]

    opts = [strategy: :one_for_one, name: CodomariBackend.Supervisor]
    Supervisor.start_link(children, opts)
  end

  ...
end

config files are:

config/couchdb.exs

import Config

config :codomari_backend, :couchdb,
  host: "127.0.0.1",
  port: 5984,
  username: "codomari",
  password: "codomari",
  database: "codomari"

config :codomari_backend, :couchdb_pool,
  name: {:local, :couchdb_pool},
  worker_module: CodomariBackend.Database.PoolWorker,
  size: 10,
  max_overflow: 5

config/config.exs

import Config

...

# DB
import_config "couchdb.exs"

Usage

I wrote CodomariBackend.Database.Methods module to keep methods separately from connection worker.

defmodule CodomariBackend.Database.Methods do
  @moduledoc """
  Module to provide CouchDB methods.
  """

  alias :poolboy, as: Poolboy

  def server_info do
    call_on_worker(:server_info)
  end

  def call_on_worker(method) do
    Poolboy.transaction(:couchdb_pool, fn worker_pid ->
      GenServer.call(worker_pid, method)
    end)
  end
end

So idea is to acquire worker pid from pool of workers based on worker name :couchdb_pool and using GenServer.call to send message to worker which will be handled on worker side using handle_call.

Let’s say we want to have find_by_id method, need to:

  1. create find_by_id(id) in Methods module by example of server_info
  2. add handle_call({:find_by_id, id}, _from, state) which will use open_doc/2 method provided by Couchbeam

Calling inside handler is simple as:

defmodule CodomariBackend.Handlers.Api.DbInfoHandler do
  @moduledoc """
  Handler serves information about database.
  """

  use CodomariBackend, :controller
  alias CodomariBackend.Database.Methods, as: DB

  @doc """
  Returns information about the database
  """
  @spec handle(Plug.Conn.t(), map) :: Plug.Conn.t()
  def handle(conn, _params) do
    DB.server_info()    # <= CALLING DB METHOD
    |> tuple_to_map()
    |> then(&json(conn, &1))
  end

  defp tuple_to_map({list}) when is_list(list) do
    list
    |> Enum.into(%{}, fn
      {key, value} when is_tuple(value) -> {key, tuple_to_map(value)}
      {key, value} -> {key, value}
    end)
  end

  defp tuple_to_map(other), do: other
end

output will be:

┌[num8er☮g8way1.local]-(~)
└> curl http://127.0.0.1:4000/api/db/info | jq
{
  "couchdb": "Welcome",
  "features": [
    "access-ready",
    "partitioned",
    "pluggable-storage-engines",
    "reshard",
    "scheduler"
  ],
  "git_sha": "6e5ad2a5c",
  "uuid": "d513c89477207a2a8e68cffdaad9c028",
  "vendor": {
    "name": "The Apache Software Foundation"
  },
  "version": "3.4.2"
}

fin

Don’t forget to check repo: https://github.com/num8er/codomari-backend

Talk Soon!