Breadcrumbs (#235)

* RingBuffer
* Basic working version
* Added Collector
* Adding error breadcrumb in notice
* Fixing some specs, ignore DateTime in to_encodable
* Added Utils.sanitize
* Moved notice breadcrumb creation out of notice.new
* Now sanitizing breadcrumb metadata
* Filter breadcrumbs
* Convert structs to Map in sanitizer
* Dropped elixir 1.7 and added 1.9 to the matrix
* Storing error breadcrumb
* Basic setup for telemetry events
* Added specs for telemetry
* Updated docs

Author: rabidpraxis

.travis.yml

@@ -1,8 +1,8 @@
 language: elixir
 sudo: false
 elixir:
-  - 1.7
   - 1.8
+  - 1.9
 otp_release:
   - 21.2
   - 22.0

README.md

@@ -119,6 +119,53 @@ rescue
 end
 ```
 
+## Breadcrumbs
+
+Breadcrumbs allow you to record events along a processes execution path. If
+an error is thrown, the set of breadcrumb events will be sent along with the
+notice. These breadcrumbs can contain useful hints while debugging.
+
+Breadcrumbs are stored in the logger context, referenced by the calling
+process. If you are sending messages between processes, breadcrumbs will not
+transfer automatically. Since a typical system might have many processes, it
+is advised that you be conservative when storing breadcrumbs as each
+breadcrumb consumes memory.
+
+### Enabling Breadcrumbs
+
+As of version `0.13.0`, Breadcrumbs are _available_ yet _disabled_. You must
+explicitly enable them if you want breadcrumbs to be reported. We plan on
+enabling this by default in a future release.
+
+Toggle `breadcrumbs_enabled` in the config to start sending Breadcrumbs with
+notices:
+
+```elixir
+config :honeybadger,
+  breadcrumbs_enabled: true
+```
+
+### Automatic Breadcrumbs
+
+We leverage the `telemetry` library to automatically create breadcrumbs from
+specific events.
+
+__Phoenix__
+
+If you are using `phoenix` (>= v1.4.7) we add a breadcrumb from the router
+start event.
+
+__Ecto__
+
+We can create breadcrumbs from Ecto SQL calls if you are using `ecto_sql` (>=
+v3.1.0). You also must specify in the config which ecto adapters you want to
+be instrumented:
+
+```elixir
+config :honeybadger,
+  ecto_repos: [MyApp.Repo]
+```
+
 ## Sample Application
 
 If you'd like to see the module in action before you integrate it with your apps, check out our [sample Phoenix application](https://github.com/honeybadger-io/crywolf-elixir).
@@ -187,6 +234,8 @@ Here are all of the options you can pass in the keyword list:
 | `filter_disable_params`  | If true, will remove the request params                                                       | `false`                                  |
 | `notice_filter`          | Module implementing `Honeybadger.NoticeFilter`. If `nil`, no filtering is done.               | `Honeybadger.NoticeFilter.Default`       |
 | `use_logger`             | Enable the Honeybadger Logger for handling errors outside of web requests                     | `true`                                   |
+| `breadcrumbs_enabled`    | Enable breadcrumb event tracking                                                              | `false`                                  |
+| `ecto_repos`             | Modules with implemented Ecto.Repo behaviour for tracking SQL breadcrumb events               | `[]`                                    |
 
 ## Public Interface
 
@@ -248,6 +297,25 @@ end)
 
 ---
 
+### `Honeybadger.add_breadcrumb/2`: Store breadcrumb within process
+
+Appends a breadcrumb to the notice. Use this when you want to add some custom
+data to your breadcrumb trace in effort to help debugging. If a notice is
+reported to Honeybadger, all breadcrumbs within the execution path will be
+appended to the notice. You will be able to view the breadcrumb trace in the
+Honeybadger interface to see what events led up to the notice.
+
+#### Examples:
+
+```elixir
+Honeybadger.add_breadcrumb("Email sent", metadata: %{
+  user: user.id,
+  message: message
+})
+```
+
+---
+
 ## Proxy configuration
 
 If your server needs a proxy to access honeybadger, add the following to your config

dummy/mixapp/mix.lock

@@ -9,5 +9,6 @@
   "plug": {:hex, :plug, "1.4.3", "236d77ce7bf3e3a2668dc0d32a9b6f1f9b1f05361019946aae49874904be4aed", [], [{:cowboy, "~> 1.0.1 or ~> 1.1", [hex: :cowboy, repo: "hexpm", optional: true]}, {:mime, "~> 1.0", [hex: :mime, repo: "hexpm", optional: false]}], "hexpm"},
   "poison": {:hex, :poison, "3.1.0", "d9eb636610e096f86f25d9a46f35a9facac35609a7591b3be3326e99a0484665", [:mix], [], "hexpm"},
   "ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.1", "28a4d65b7f59893bc2c7de786dec1e1555bd742d336043fe644ae956c3497fbe", [:make, :rebar], [], "hexpm"},
+  "telemetry": {:hex, :telemetry, "0.4.0", "8339bee3fa8b91cb84d14c2935f8ecf399ccd87301ad6da6b71c09553834b2ab", [:rebar3], [], "hexpm"},
   "unicode_util_compat": {:hex, :unicode_util_compat, "0.3.1", "a1f612a7b512638634a603c8f401892afbf99b8ce93a45041f8aaca99cadb85e", [:rebar3], [], "hexpm"},
 }

lib/honeybadger.ex

@@ -16,6 +16,8 @@ defmodule Honeybadger do
         environment_name: :prod,
         app: :my_app_name,
         exclude_envs: [:dev, :test],
+        breadcrumbs_enabled: false,
+        ecto_repos: [MyAppName.Ecto.Repo],
         hostname: "myserver.domain.com",
         origin: "https://api.honeybadger.io",
         proxy: "http://proxy.net:PORT",
@@ -121,11 +123,51 @@ defmodule Honeybadger do
       end
 
   See `Honeybadger.Filter` for details on implementing your own filter.
+
+  ### Breadcrumbs
+
+  Breadcrumbs allow you to record events along a processes execution path. If
+  an error is thrown, the set of breadcrumb events will be sent along with the
+  notice. These breadcrumbs can contain useful hints while debugging.
+
+  Breadcrumbs are stored in the logger context, referenced by the calling
+  process. If you are sending messages between processes, breadcrumbs will not
+  transfer automatically. Since a typical system might have many processes, it
+  is advised that you be conservative when storing breadcrumbs as each
+  breadcrumb consumes memory.
+
+  Ensure that you enable breadcrumbs in the config (as it is disabled by
+  default):
+
+      config :honeybadger,
+        breadcrumbs_enabled: true
+
+  See `Honeybadger.add_breadcrumb` for info on how to add custom breadcrumbs.
+
+  ### Automatic Breadcrumbs
+
+  We leverage the `telemetry` library to automatically create breadcrumbs from
+  specific events.
+
+  #### Phoenix
+
+  If you are using `phoenix` (>= v1.4.7) we add a breadcrumb from the router
+  start event.
+
+  #### Ecto
+
+  We can create breadcrumbs from Ecto SQL calls if you are using `ecto_sql` (>=
+  v3.1.0). You also must specify in the config which ecto adapters you want to
+  be instrumented:
+
+      config :honeybadger,
+        ecto_repos: [MyApp.Repo]
   """
 
   use Application
 
   alias Honeybadger.{Client, Notice}
+  alias Honeybadger.Breadcrumbs.{Collector, Breadcrumb}
 
   defmodule MissingEnvironmentNameError do
     defexception message: """
@@ -152,6 +194,10 @@ defmodule Honeybadger do
       _ = Logger.add_backend(Honeybadger.Logger)
     end
 
+    if config[:breadcrumbs_enabled] do
+      Honeybadger.Breadcrumbs.Telemetry.attach()
+    end
+
     children = [
       worker(Client, [config])
     ]
@@ -203,11 +249,53 @@ defmodule Honeybadger do
   """
   @spec notify(Notice.noticeable(), map(), list()) :: :ok
   def notify(exception, metadata \\ %{}, stacktrace \\ []) do
+    # Grab process local breadcrumbs if not passed with call and add notice breadcrumb
+    breadcrumbs =
+      metadata
+      |> Map.get(:breadcrumbs, Collector.breadcrumbs())
+      |> Collector.put(notice_breadcrumb(exception))
+      |> Collector.output()
+
+    metadata_with_breadcrumbs =
+      metadata
+      |> Map.delete(:breadcrumbs)
+      |> contextual_metadata()
+      |> Map.put(:breadcrumbs, breadcrumbs)
+
     exception
-    |> Notice.new(contextual_metadata(metadata), stacktrace)
+    |> Notice.new(metadata_with_breadcrumbs, stacktrace)
     |> Client.send_notice()
   end
 
+  @doc """
+  Stores a breadcrumb item.
+
+  Appends a breadcrumb to the notice. Use this when you want to add some custom
+  data to your breadcrumb trace in effort to help debugging. If a notice is
+  reported to Honeybadger, all breadcrumbs within the execution path will be
+  appended to the notice. You will be able to view the breadcrumb trace in the
+  Honeybadger interface to see what events led up to the notice.
+
+  ## Breadcrumb with metadata
+
+      Honeybadger.add_breadcrumb("email sent", metadata: %{
+        user: user.id, message: message
+      })
+      => :ok
+
+  ## Breadcrumb with specified category. This will display a query icon in the interface
+
+      Honeybadger.add_breadcrumb("ETS Lookup", category: "query", metadata: %{
+        key: key,
+        value: value
+      })
+      => :ok
+  """
+  @spec add_breadcrumb(String.t(), Breadcrumb.opts()) :: :ok
+  def add_breadcrumb(message, opts \\ []) when is_binary(message) and is_list(opts) do
+    Collector.add(Breadcrumb.new(message, opts))
+  end
+
   @doc """
   Retrieves the context that will be sent to the Honeybadger API when an exception occurs in the
   current process.
@@ -216,7 +304,7 @@ defmodule Honeybadger do
   """
   @spec context() :: map()
   def context do
-    Logger.metadata() |> Map.new()
+    Logger.metadata() |> Map.new() |> Map.delete(Collector.metadata_key())
   end
 
   @doc """
@@ -291,6 +379,30 @@ defmodule Honeybadger do
 
   # Helpers
 
+  # Allows for Notice breadcrumb to have custom text as message if an error is
+  # not passed to the notice function. We can assume if it was passed an error
+  # then there will be an error breadcrumb right before this one.
+  defp notice_breadcrumb(exception) do
+    reason =
+      case exception do
+        title when is_binary(title) ->
+          title
+
+        error when is_atom(error) and not is_nil(error) ->
+          :error
+          |> Exception.normalize(error)
+          |> Map.get(:message, to_string(error))
+
+        _ ->
+          nil
+      end
+
+    ["Honeybadger Notice", reason]
+    |> Enum.reject(&is_nil/1)
+    |> Enum.join(": ")
+    |> Breadcrumb.new(category: "notice")
+  end
+
   defp put_dynamic_env(config) do
     hostname = fn ->
       :inet.gethostname()

lib/honeybadger/breadcrumbs/breadcrumb.ex

@@ -0,0 +1,43 @@
+defmodule Honeybadger.Breadcrumbs.Breadcrumb do
+  @moduledoc false
+
+  @derive Jason.Encoder
+
+  @type t :: %__MODULE__{
+          message: String.t(),
+          category: String.t(),
+          timestamp: DateTime.t(),
+          metadata: map()
+        }
+
+  @type opts :: [{:metadata, map()} | {:category, String.t()}]
+  @enforce_keys [:message, :category, :timestamp, :metadata]
+
+  @default_category "custom"
+  @default_metadata %{}
+
+  defstruct [:message, :category, :timestamp, :metadata]
+
+  @spec new(String.t(), opts()) :: t()
+  def new(message, opts) do
+    %__MODULE__{
+      message: message,
+      category: opts[:category] || @default_category,
+      timestamp: DateTime.utc_now(),
+      metadata: opts[:metadata] || @default_metadata
+    }
+  end
+
+  @spec from_error(any()) :: t()
+  def from_error(error) do
+    error = Exception.normalize(:error, error, [])
+
+    %{__struct__: error_mod} = error
+
+    new(
+      Honeybadger.Utils.module_to_string(error_mod),
+      metadata: %{message: error_mod.message(error)},
+      category: "error"
+    )
+  end
+end

lib/honeybadger/breadcrumbs/collector.ex

@@ -0,0 +1,59 @@
+defmodule Honeybadger.Breadcrumbs.Collector do
+  @moduledoc false
+
+  @doc """
+  The Collector provides an interface for accessing and affecting the current
+  set of breadcrumbs. Most operations are delegated to the supplied Buffer
+  implementation. This is mainly for internal use.
+  """
+
+  alias Honeybadger.Breadcrumbs.{RingBuffer, Breadcrumb}
+  alias Honeybadger.Utils
+
+  @buffer_impl RingBuffer
+  @buffer_size 40
+  @metadata_key :hb_breadcrumbs
+
+  @type t :: %{enabled: boolean(), trail: [Breadcrumb.t()]}
+
+  @spec output() :: t()
+  def output(), do: output(breadcrumbs())
+
+  @spec output(@buffer_impl.t()) :: t()
+  def output(breadcrumbs) do
+    %{
+      enabled: Honeybadger.get_env(:breadcrumbs_enabled),
+      trail: @buffer_impl.to_list(breadcrumbs)
+    }
+  end
+
+  @spec put(@buffer_impl.t(), Breadcrumb.t()) :: @buffer_impl.t()
+  def put(breadcrumbs, breadcrumb) do
+    @buffer_impl.add(
+      breadcrumbs,
+      Map.update(breadcrumb, :metadata, %{}, &Utils.sanitize(&1, max_depth: 1))
+    )
+  end
+
+  @spec add(Breadcrumb.t()) :: :ok
+  def add(breadcrumb) do
+    if Honeybadger.get_env(:breadcrumbs_enabled) do
+      Logger.metadata([{@metadata_key, put(breadcrumbs(), breadcrumb)}])
+    end
+
+    :ok
+  end
+
+  @spec clear() :: :ok
+  def clear() do
+    Logger.metadata([{@metadata_key, @buffer_impl.new(@buffer_size)}])
+  end
+
+  def metadata_key(), do: @metadata_key
+
+  @spec breadcrumbs() :: @buffer_impl.t()
+  def breadcrumbs() do
+    Logger.metadata()
+    |> Keyword.get(@metadata_key, @buffer_impl.new(@buffer_size))
+  end
+end