[Awesome Ruby Gem] Use httplog gem to log outgoing HTTP requests in ruby

Posted on Edited on In , , Views: Word count in article: 6.2k Reading time ≈ 6 mins.

httplog

Log outgoing HTTP requests made from your application. Helps with debugging pesky API error responses, or just generally understanding what’s going on under the hood.

Requires ruby >= 2.5

This gem works with the following ruby modules and libraries:

  • Net::HTTP v4+

  • Ethon

  • Excon

  • OpenURI

  • Patron

  • HTTPClient

  • HTTParty

  • HTTP

These libraries are at least partially supported, where they use one of the above as adapters, but not explicitly tested - YMMV:

  • Faraday

  • Typhoeus

In theory, it should also work with any library built on top of these. But the difference between theory and practice is bigger in practice than in theory.

This is very much a development and debugging tool; it is not recommended to use this in a production environment as it is monkey-patching the respective HTTP implementations. You have been warned - use at your own risk.

Installation

You can install it as a gem:

1
$ gem install httplog

or add it into a Gemfile (Bundler):

1
2
3
4
5
6
7
8
# Gemfile

# Put gems used only for development or testing in the appropriate group in the Gemfile
group :development do
  # trusche/httplog: Log outgoing HTTP requests in ruby
  # https://github.com/trusche/httplog
  gem 'httplog', '1.5.0'
end

Then, run bundle install.

1
$ bundle install

Usage

1
require 'httplog' # require this *after* your HTTP gem of choice

By default, this will log all outgoing HTTP requests and their responses to $stdout on DEBUG level.

Notes on content types

  • Binary data from response bodies (as indicated by the Content-Type header)is not logged.

  • Text data (text/* and most application/* types) is encoded as UTF-8, with invalid characters replaced. If you need to inspect raw non-UTF data exactly as sent over the wire, this tool is probably not for you.

Configuration

If you want to use this in a Rails app, I’d suggest configuring this specifically for each environment. A global initializer is not a good idea since HttpLog will be undefined in production. Because you’re not using this in production, right? :)

You can override the following default options:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
HttpLog.configure do |config|

  # Enable or disable all logging
  config.enabled = true

  # You can assign a different logger or method to call on that logger
  config.logger = Rails.logger
  # config.logger = Logger.new($stdout)
  config.logger_method = :log

  # I really wouldn't change this...
  config.severity = Logger::Severity::DEBUG

  # Tweak which parts of the HTTP cycle to log...
  config.log_connect   = true
  config.log_request   = true
  config.log_headers   = false
  config.log_data      = true
  config.log_status    = true
  config.log_response  = true
  config.log_benchmark = true

  # ...or log all request as a single line by setting this to `true`
  config.compact_log = false

  # You can also log in JSON format
  config.json_log = false

  # Prettify the output - see below
  config.color = false

  # Limit logging based on URL patterns
  config.url_whitelist_pattern = nil
  config.url_blacklist_pattern = nil

  # Mask sensitive information in request and response JSON data.
  # Enable global JSON masking by setting the parameter to `/.*/`
  config.url_masked_body_pattern = nil

  # You can specify any custom JSON serializer that implements `load` and `dump` class methods
  # to parse JSON responses
  config.json_parser = JSON

  # When using graylog, you can supply a formatter here - see below for details
  config.graylog_formatter = nil

  # Mask the values of sensitive request parameters
  config.filter_parameters = %w[password]
  
  # Customize the prefix with a proc or lambda
  config.prefix = ->{ "[httplog] #{Time.now} " }
end

You can colorize the output to make it stand out in your logfile, either with a single color for the text:

1
2
3
HttpLog.configure do |config|
  config.color = :red
end

Or with a color hash for text and background:

1
2
3
HttpLog.configure do |config|
  config.color = {color: :black, background: :yellow}
end

For more color options please refer to the rainbow documentation

See Configuration - https://github.com/trusche/httplog#configuration to learn more.

Examples

With the default configuration, the log output might look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
[httplog] Connecting: localhost:80
[httplog] Sending: GET http://localhost:9292/index.html
[httplog] Status: 200
[httplog] Benchmark: 0.00057 seconds
[httplog] Response:
<html>
  <head>
    <title>Test Page</title>
  </head>
  <body>
    <h1>This is the test page.</h1>
  </body>
</html>

With log_headers = true and a parameter ‘password’ in the request query and headers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[httplog] Connecting: localhost:80
[httplog] Sending: GET http://localhost:9292/index.html?password=[FILTERED]
[httplog] Header: accept: *.*
[httplog] Header: password=[FILTERED]
[httplog] Status: 200
[httplog] Benchmark: 0.00057 seconds
[httplog] Response:
<html>
  <head>
    <title>Test Page</title>
  </head>
  <body>
    <h1>This is the test page.</h1>
  </body>
</html>

Compact logging

If the log is too noisy for you, but you don’t want to completely disable it either, set the compact_log option to true. This will log each request in a single line with method, request URI, response status and time, but no data or headers. No need to disable any other options individually.

1
[httplog] GET http://localhost:9292/index.html completed with status code 200 in 0.00057 seconds

JSON logging

If you want to log HTTP requests in a JSON format, set the json_log option to true. You can combine this with compact_log to only log the basic request metrics without headers and bodies.

1
2
3
4
5
# With json_log enabled:
[httplog] {"method":"GET","url":"localhost:80","request_body":null, "request_headers":{"foo":"bar"}, "response_code":200,"response_body":"<html>\n      <head>\n        <title>Test Page</title>\n      </head>\n      <body>\n        <h1>This is the test page.</h1>\n      </body>\n    </html>","response_headers":{"foo":"bar"},"benchmark":0.00057}

# And with json_log and compact_log enabled:
[httplog] {"method":"GET","url":"localhost:80","response_code":200,"benchmark":0.00057}

Parameter filtering

Just like in Rails, you can filter the values of sensitive parameters by setting the filter_parameters to an array of (lower case) keys. The value for “password” is filtered by default.

Please note that this will only filter the request data with well-formed parameters (in the URL, the headers, and the request data) but not the response. It does not currently filter JSON request data either, just standard “key=value” pairs in the request body.

References

[1] trusche/httplog: Log outgoing HTTP requests in ruby - https://github.com/trusche/httplog

[2] httplog | RubyGems.org | your community gem host - https://rubygems.org/gems/httplog/