Cross-cluster Associations

Most recently, Github Ruby team extracted internal functionality to disable joining queries when an association crossed multiple databases. Prior to our work in this area, Rails had no support for handling associations that spanned across clusters; teams had to write SQL to achieve this.

The overall implementation is relatively small. To accomplish disabling joins, we added an option to has_many :through associations called disable_joins. When set to true for an association, Rails will generate separate queries for each database rather than a join query.

Terraform LDAP Examples

Examples about using Terraform LDAP Provider to manage ldap objects.

Github: https://github.com/CloudoLife/terraform-ldap-examples

Prerequsites

Your must have a LDAP server with bind account and password, and Terraform CLI have been installed.

LDAP Server with Bind Account and Password

Recommand use osixia/docker-openldap: A docker image to run OpenLDAP 🐳 - https://github.com/osixia/docker-openldap to run OpenLDAP server within container if you want to install the LDAP Server.

Thne, remember the admin account and bind password.

Terrraform and Terrafrom LDAP Provider

• Terraform by HashiCorp

  Use Infrastructure as Code to provision and manage any cloud, infrastructure, or service

• CloudoLife/terraform-provider-ldap: LDAP provider for Terraform

  The LDAP provider gives the ability to deploy and configure resources in a LDAP server.

Installation

First, install and config Terraform LDAP provider.

1

./install.sh

It will go get github.com/CloudoLife/terraform-provider-ldap and create or append these content to ~/.terraformrc file.

1
2
3
4

# ~/.terraformrc
providers {
    ldap = "${GOPATH}/bin/terraform-provider-ldap"
}

Configuration

LDAP Provider Variables

First, Specify values for LDAP Provider variables within terraform.tfvars.

1
2
3
4
5
6
7
8
9

ldap_host = "localhost"

ldap_port = 389

use_tls = false 

bind_user = "cn=admin,dc=<Your Domain>,dc=com"

bind_password = "password"

LDAP Object

Config LDAP objects within ./objects

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

resource "ldap_object" "foo" {
    # DN must be complete (no RDN!)
    dn = "uid=foo,dc=example,dc=com"

    # classes are specified as an array
    object_classes = [
        "inetOrgPerson",
        "posixAccount",
    ]

    # attributes are specified as a set of 1-element maps
    attributes = [
        { sn              = "10" },
        { cn              = "bar" },
        { uidNumber       = "1234" },
        { gidNumber       = "1234" },
        { homeDirectory   = "/home/billy" },
        { loginShell      = "/bin/bash" },
        # when an attribute has multiple values, it must be specified multiple times
        { mail            = "[email protected]" },
        { mail            = "[email protected]" },
    ]
}

Run

Terraform Init

1

terraform init

It will download the Terraform LDAP Provider binary.

Terraform apply

1

terraform apply

It will dispaly the execute plan and wait for your make yes or no.
Then yes to create ldap objects.

Import

Import and create .tf file by run ./import.sh command.

1

./import.sh uid=foo,dc=example,dc=com

It will import LDAP object(uid=foo,dc=example,dc=com) to Terraform State(terraform.tfstate) and auto create the ldap_object.foo.tf file.

You can continue to maintain that file and run terraform apply command to manage the exist LDAP objects without copying & pasting from Terraform State(terraform.tfstate).

Inspire

Inspired by Pryz/terraform-provider-ldap: LDAP provider for Terraform.

Reference

[2] CloudoLife/terraform-provider-ldap: LDAP provider for Terraform

[2] Terraform by HashiCorp - https://www.terraform.io/

[3] Pryz/terraform-provider-ldap: LDAP provider for Terraform

[4] osixia/docker-openldap: A docker image to run OpenLDAP 🐳 - https://github.com/osixia/docker-openldap

Annotate (aka AnnotateModels)

The Annotate (aka AnnotateModels) gem annotating Rails classes with schema and routes info by adding a comment summarizing the current schema to the top or bottom of each of your…

bullet

The Bullet gem is designed to help you increase your application’s performance by reducing the number of queries it makes. It will watch your queries while you develop your application and notify you when you should add eager loading (N+1 queries), when you’re using eager loading that isn’t necessary and when you should use counter cache.

marginalia

The marginalia gem attaching comments to your ActiveRecord queries. By default, it adds the application, controller, and action names as a comment at the end of each query.

This helps when searching log files for queries, and seeing where slow queries came from.

Database Configuration in Rails

Since there are two ways to configure your connection (using config/database.yml or using an environment variable ENV['DATABASE_URL']) it is important to understand how they can interact.

If you look at the options of the application generator, you will see that one of the options is named --database. This option allows you to choose an adapter from a list of the most used relational databases. You can even run the generator repeatedly:

1

$ cd .. && rails new blog --database=mysql

When you confirm the overwriting of the config/database.yml file, your application will be configured for MySQL instead of SQLite. Detailed examples of the common database connections are below.

Print Configuration

1
2
3
4
5
6
7
8
9
10
11

# Current database configuration
> ActiveRecord::Base.connection_db_config

# DEPRECATION WARNING: connection_config is deprecated and will be removed from Rails 6.2 (Use connection_db_config instead)
# > ActiveRecord::Base.connection_config

# Current database configuration
> Rails.configuration.database_configuration[Rails.env]

# All database configuration
> Rails.configuration.database_configuration

Connection Preference

• If you have an empty config/database.yml file but your ENV['DATABASE_URL'] is present, then Rails will connect to the database via your environment variable ENV['DATABASE_URL'].

• If you have a config/database.yml but no ENV['DATABASE_URL'] then this file will be used to connect to your database:

• If you have both config/database.yml and ENV['DATABASE_URL'] set then Rails will merge the configuration together. To better understand this we must see some examples.

  • When duplicate connection information is provided the environment variable will take precedence

  • If non-duplicate information is provided you will get all unique values, environment variable still takes precedence in cases of any conflicts.

The only way to explicitly not use the connection information in ENV['DATABASE_URL'] is to specify an explicit URL connection using the url sub key:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

$ cat config/database.yml
development:
  url: sqlite3:NOT_my_database

$ echo $DATABASE_URL
postgresql://localhost/my_database

$ bin/rails runner 'puts ActiveRecord::Base.configurations'
#<ActiveRecord::DatabaseConfigurations:0x00007fd50e209a28>

$ bin/rails runner 'puts ActiveRecord::Base.configurations.inspect'
#<ActiveRecord::DatabaseConfigurations:0x00007fc8eab02880 @configurations=[
  #<ActiveRecord::DatabaseConfigurations::UrlConfig:0x00007fc8eab020b0
    @env_name="development", @spec_name="primary",
    @config={"adapter"=>"sqlite3", "database"=>"NOT_my_database"}
    @url="sqlite3:NOT_my_database">
  ]

Here the connection information in ENV['DATABASE_URL'] is ignored, note the different adapter and database name.

(Recommended) Embed ENV[‘DATABASE_URL’] in config/database.yml

Since it is possible to embed ERB in your config/database.yml it is best practice to explicitly show you are using the ENV['DATABASE_URL'] to connect to your database. This is especially useful in production since you should not commit secrets like your database password into your source control (such as Git).

1
2
3

$ cat config/database.yml
production:
  url: <%= ENV['DATABASE_URL'] %>

Now the behavior is clear, that we are only using the connection information in ENV['DATABASE_URL'].

See 3.19 Connection Preference - https://guides.rubyonrails.org/configuring.html#connection-preference to learen more.

Connection Options

MySQL

If you choose to use MySQL or MariaDB instead of the shipped SQLite3 database, your config/database.yml will look a little different. Here’s the development section:

1
2
3
4
5
6
7
8

development:
  adapter: mysql2
  encoding: utf8mb4
  database: blog_development
  pool: 5
  username: root
  password:
  socket: /tmp/mysql.sock

If your development database has a root user with an empty password, this configuration should work for you. Otherwise, change the username and password in the development section as appropriate.

If your MySQL version is 5.5 or 5.6 and want to use the utf8mb4 character set by default, please configure your MySQL server to support the longer key prefix by enabling innodb_large_prefix system variable.

Advisory Locks are enabled by default on MySQL and are used to make database migrations concurrent safe. You can disable advisory locks by setting advisory_locks to false:

1
2
3

production:
  adapter: mysql2
  advisory_locks: false

You can see MySQL connection options from brianmario/mysql2: A modern, simple and very fast Mysql library for Ruby - binding to libmysql - https://github.com/brianmario/mysql2#connection-options.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

Mysql2::Client.new(
  :host,
  :username,
  :password,
  :port,
  :database,
  :socket = '/path/to/mysql.sock',
  :flags = REMEMBER_OPTIONS | LONG_PASSWORD | LONG_FLAG | TRANSACTIONS | PROTOCOL_41 | SECURE_CONNECTION | MULTI_STATEMENTS,
  :encoding = 'utf8',
  :read_timeout = seconds,
  :write_timeout = seconds,
  :connect_timeout = seconds,
  :connect_attrs = {:program_name => $PROGRAM_NAME, ...},
  :reconnect = true/false,
  :local_infile = true/false,
  :secure_auth = true/false,
  :ssl_mode = :disabled / :preferred / :required / :verify_ca / :verify_identity,
  :default_file = '/path/to/my.cfg',
  :default_group = 'my.cfg section',
  :default_auth = 'authentication_windows_client'
  :init_command => sql
  )

PostgreSQL

If you choose to use PostgreSQL, your config/database.yml will be customized to use PostgreSQL databases:

1
2
3
4
5

development:
  adapter: postgresql
  encoding: unicode
  database: blog_development
  pool: 5

By default Active Record uses database features like prepared statements and advisory locks. You might need to disable those features if you’re using an external connection pooler like PgBouncer:

1
2
3
4
5
6
7
8
9
10
11
12

production:
  adapter: postgresql
  prepared_statements: false
  advisory_locks: false
```  

If enabled, Active Record will create up to 1000 prepared statements per database connection by default. To modify this behavior you can set `statement_limit` to a different value:

```yaml
production:
  adapter: postgresql
  statement_limit: 200

The more prepared statements in use: the more memory your database will require. If your PostgreSQL database is hitting memory limits, try lowering statement_limit or disabling prepared statements.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# The PostgreSQL adapter works with the native C (https://github.com/ged/ruby-pg) driver.
#
# Options:
#
# * <tt>:host</tt> - Defaults to a Unix-domain socket in /tmp. On machines without Unix-domain sockets,
#   the default is to connect to localhost.
# * <tt>:port</tt> - Defaults to 5432.
# * <tt>:username</tt> - Defaults to be the same as the operating system name of the user running the application.
# * <tt>:password</tt> - Password to be used if the server demands password authentication.
# * <tt>:database</tt> - Defaults to be the same as the username.
# * <tt>:schema_search_path</tt> - An optional schema search path for the connection given
#   as a string of comma-separated schema names. This is backward-compatible with the <tt>:schema_order</tt> option.
# * <tt>:encoding</tt> - An optional client encoding that is used in a <tt>SET client_encoding TO
#   <encoding></tt> call on the connection.
# * <tt>:min_messages</tt> - An optional client min messages that is used in a
#   <tt>SET client_min_messages TO <min_messages></tt> call on the connection.
# * <tt>:variables</tt> - An optional hash of additional parameters that
#   will be used in <tt>SET SESSION key = val</tt> calls on the connection.
# * <tt>:insert_returning</tt> - An optional boolean to control the use of <tt>RETURNING</tt> for <tt>INSERT</tt> statements
#   defaults to true.
#
# Any further options are used as connection parameters to libpq. See
# https://www.postgresql.org/docs/current/static/libpq-connect.html for the
# list of parameters.
#
# In addition, default connection parameters of libpq can be set per environment variables.
# See https://www.postgresql.org/docs/current/static/libpq-envars.html .

Configuring an SQLite3 Database

Rails comes with built-in support for SQLite3, which is a lightweight serverless database application. While a busy production environment may overload SQLite, it works well for development and testing. Rails defaults to using an SQLite database when creating a new project, but you can always change it later.

Here’s the section of the default configuration file (config/database.yml) with connection information for the development environment:

1
2
3
4
5

development:
  adapter: sqlite3
  database: db/development.sqlite3
  pool: 5
  timeout: 5000

Rails uses an SQLite3 database for data storage by default because it is a zero configuration database that just works. Rails also supports MySQL (including MariaDB) and PostgreSQL “out of the box”, and has plugins for many database systems. If you are using a database in a production environment Rails most likely has an adapter for it.

See Sample config/database.yml from Rails. Postgres, MySQL, and SQLite - https://gist.github.com/jwo/4512764 and [Rails Database yml examples for postgres sqlite and mysql - https://gist.github.com/datt/e12fa0da294e7a8f3ac96abee346a098](https://gist.github.com/datt/ e12fa0da294e7a8f3ac96abee346a098) to learn more exmaples.

Multi Database Support

See Multiple Databases with Active Record — Ruby on Rails Guides - https://guides.rubyonrails.org/active_record_multiple_databases.html
to learn more.

Change Database Support

Rails 6 provide a command has been added to change database adapter automatically.

Let’s say our app has started with SQLite and now we have to switch to MySQL.

1
2
3
4
5
6

$ rails db:system:change --to=mysql
    conflict  config/database.yml
Overwrite /Users/cloudolife/example_app/config/database.yml? (enter "h" for help) [Ynaqdhm] Y
       force  config/database.yml
        gsub  Gemfile
        gsub  Gemfile

Our database.yml is now changed to contain the configuration for MySQL database and the Gemfile also gets updated automatically with addition of mysql2 gem in place of sqlite3.

References

[1] Configuring Rails Applications — Ruby on Rails Guides - https://guides.rubyonrails.org/configuring.html#configuring-a-database(https://guides.rubyonrails.org/configuring.html#configuring-a-database)

[2] Getting Started with Rails — Ruby on Rails Guides - https://guides.rubyonrails.org/getting_started.html

[3] Configuring Rails Applications — Ruby on Rails Guides - https://guides.rubyonrails.org/configuring.html#database-pooling

[4] Multiple Databases with Active Record — Ruby on Rails Guides - https://guides.rubyonrails.org/active_record_multiple_databases.html

[5] Configuring and connecting to a database | BigBinary - https://www.bigbinary.com/learn-rubyonrails-book/configure-and-connect-to-database

[6] Sample config/database.yml from Rails. Postgres, MySQL, and SQLite - https://gist.github.com/jwo/4512764

[7] Rails Database yml examples for postgres sqlite and mysql - https://gist.github.com/datt/e12fa0da294e7a8f3ac96abee346a098

[8] Rails 6 has added a way to change the database of the app | BigBinary Blog - https://www.bigbinary.com/blog/rails-6-has-added-a-way-to-change-the-database-of-the-app

[9] Add rails db:system:change command by gmcgibbon · Pull Request #34832 · rails/rails - https://github.com/rails/rails/pull/34832

[10] Part 2: Multi-db improvements, Refactor Active Record configurations by eileencodes · Pull Request #33637 · rails/rails - https://github.com/rails/rails/pull/33637

[11] rails/postgresql_adapter.rb at main · rails/rails - https://github.com/rails/rails/blob/main/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb

[12] ged/ruby-pg: A PostgreSQL client library for Ruby - https://github.com/ged/ruby-pg

[13] brianmario/mysql2: A modern, simple and very fast Mysql library for Ruby - binding to libmysql - https://github.com/brianmario/mysql2#connection-options

[14] brianmario/mysql2: A modern, simple and very fast Mysql library for Ruby - binding to libmysql - https://github.com/brianmario/mysql2

[15] sparklemotion/sqlite3-ruby: Ruby bindings for the SQLite3 embedded database - https://github.com/sparklemotion/sqlite3-ruby

[16] SQLite Home Page - https://www.sqlite.org/index.html

Resize XFS disk partition size

XFS is a high-performance journaling file system created by Silicon Graphics, Inc. XFS is particularly proficient at parallel IO due to its allocation group based design. This enables extreme scalability of IO threads, filesystem bandwidth, file and filesystem size when spanning multiple storage devices.

GNU Parted is a program for creating and manipulating partition tables.

This article describe how to use parted and xfs_growfs to resize XFS disk partition size dynamically.

SSR-Prerender

ssr-prerender is a wrapper to use Prerender to prerender HTML for SEO. It can run by Docker Compose, Docker or Docker Compose or Shell(Traditional).

Prerender

Prerender is a node server that uses Headless Chrome to render HTML, screenshots, PDFs, and HAR files out of any web page. The Prerender server listens for an http request, takes the URL and loads it in Headless Chrome, waits for the page to finish loading by waiting for the network to be idle, and then returns your content.

It have some advantages:

• Loosely coupled
  No code or only a small codes need to modify no matter what technical architecture is used within Front-End or Back-End.

• Out of the box
  Provide cache support, multiple render types(HTML, Image, PDF, HAR), Allowlist, Blacklist, etc.

Deployment Diagram

Powered by PlantUML

Prerequisites

• Docker - https://www.docker.com/

  Developing apps today requires so much more than writing code. Multiple languages, frameworks, architectures, and discontinuous interfaces between tools for each lifecycle stage creates enormous complexity.

  Docker simplifies and accelerates your workflow, while giving developers the freedom to innovate with their choice of tools, application stacks, and deployment environments for each project.

  To learn more about all the features of Docker, see Docker - https://www.docker.com/.

• Docker Compose - https://docs.docker.com/compose/

  Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

  To learn more about all the features of Docker Compose, see Docker Compose - https://docs.docker.com/compose/.

• Google Chrome - Download the Fast, Secure Browser from Google - https://www.google.com/chrome/

  Google Chrome is the browser built by Google.

  To learn more about all the features of Google Chrome, see the Google Chrome - Download the Fast, Secure Browser from Google - https://www.google.com/chrome/.

  It is optional while run within the docker container.

Configuration

You can change configuration within .env file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

# prerender/prerender: Node server that uses Headless Chrome to render a javascript-rendered page as HTML. To be used in conjunction with prerender middleware.
# https://github.com/prerender/prerende
CHROME_FLAGS="--no-sandbox, --headless, --disable-gpu, --remote-debugging-port=9222, --hide-scrollbars"
LOG_REQUESTS=false
PORT=3000

# memory or redis
STORAGE_CACHE=memory

# prerender-redis-cache/prerenderRedisCache.js at master · JonathanBennett/prerender-redis-cache
# https://github.com/JonathanBennett/prerender-redis-cache/blob/master/lib/prerenderRedisCache.js
# REDISTOGO_URL=redis://redis:6379/0
# REDISCLOUD_URL=redis://redis:6379/0
# REDISGREEN_URL=redis://redis:6379/0
REDIS_URL=redis://redis:6379/0

# prerender/prerender-memory-cache: In memory cache for use with Prerender server
# https://github.com/prerender/prerender-memory-cache
CACHE_MAXSIZE=10000
CACHE_TTL=3600

Run

Recommand to run ss-prerender with Docker Compose or Docker.

Docker Compose

1

docker-compose up

Docker

1

docker run -it --rm --name ssr-prerender -p 3000:3000 cloudolife/ssr-prerender:latest

Kubernetes

TBD

Shell

You can run ss-prerender from local shell or within a docker container.

1

node server.js

That’s all.

Test

http://localhost:3000/http://localhost:8080

Upgrade

Google Chrome

This repository has provider a Google Chrome version and will auto install while docker build process. But you can upgrade Google Chrome to the latest version by download and replace google-chrome-stable_current_amd64.deb file with the latest Google Chrome version.

1

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb -O google-chrome-stable_current_amd64.deb

Then, run command docker build . or docker-compose buiid to rebuild it.

References

[1] prerender/prerender: Node server that uses Headless Chrome to render a javascript-rendered page as HTML. To be used in conjunction with prerender middleware. - https://github.com/prerender/prerender

[2] Prerender - Dynamic Rendering for Effective JavaScript SEO | Prerender - https://prerender.io/

[3] Prerender - Dynamic Rendering for Effective JavaScript SEO | Prerender - https://prerender.io/documentation

[4] Prerender.com | Headless Chrome In The Cloud to Render HTML, Screenshots, PDFs, and HAR Files - https://prerender.com/

[5] Prerender.com | Documentation - https://prerender.com/documentation

[6] prerender-redis-cache/prerenderRedisCache.js at master · JonathanBennett/prerender-redis-cache - https://github.com/JonathanBennett/prerender-redis-cache/blob/master/lib/prerenderRedisCache.js

[7] prerender/prerender-memory-cache: In memory cache for use with Prerender server - https://github.com/prerender/prerender-memory-cache

[8] Docker - https://www.docker.com/

[9] Docker Compose - https://docs.docker.com/compose/

For security reasons, Alibaba Cloud blocks outbound traffic on TCP port 25 by default. that You cannot connect to external addresses TCP port 25 from Alibaba Cloud Cloud servers (ECS).

Background Information

Blocking the outbound direction of TCP 25 port may affect your connection to the SMTP server of a third-party mail service provider through TCP 25 port to send external mail. If you need to use the cloud server on Alibaba Cloud to send external emails, it is recommended that you use Alibaba Cloud Mail products or use the 465 port provided by a third-party mail service provider.

Note If there are special scenarios, you must use TCP port 25 on the cloud server for external connections. Please submit a port 25 unblocking application in the security management and control platform. Alibaba Cloud will review the application reason you submitted and notify you of the review result by email.

Configuration

Remember to replace the content within {{ }} with your prefer value.

The from_address and user must be the same.

1
2
3
4
5
6
7
8

smtp:
  from_address: {{ .Values.smtp.from_address }} # [email protected]
  host: smtp.exmail.qq.com
  port: 465 # not 25
  use_tls: {{ .Values.smtp.use_tls }} # true or false
  user: {{ .Values.smtp.user }} # [email protected]
  password: {{ .Values.smtp.password }}
  enable_replies: false # or true

FAQs

smtp or smtps.

1

The $MAIL_URL is 'smtp://...:465'.  You probably want 'smtps://' (The 's' enables TLS/SSL) since '465' is typically a secure port. undefined 

Use smtps protocols and 465 port instead of smtp and 25.

Error code 501

1

Exception in defer callback: Error: Mail command failed: 501 ϵͳÒÑÇ¿ÖÆ¿ªÆôÕʺÅÉý¼¶ÉèÖã¬ÇëµÇ¼exmail.qq.com°ó¶¨Î¢ÐÅ

This is because Tencent Enterprise Mailbox forced to enable the security certification before send email from client.

You can log in Tencent Enterprise Mailbox Administrator Web UI, find the sender’s account, and edit it to disable enforce secure login option.

References

[1] 腾讯企业邮箱 - https://exmail.qq.com/

[2] TCP 25端口解封申请 - 阿里云 - https://help.aliyun.com/document_detail/56130.html

[3] 阿里云服务器不能发邮件禁用25端口的三种解决方法_weixin_30258027的博客-CSDN博客 - https://blog.csdn.net/weixin_30258027/article/details/95448730?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.control&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.control

Terraform PostgreSQL Database Examples

Examples about using Terraform PostgreSQL Provider to manage databases and roles(users).

Github: https://github.com/CloudoLife/terraform-postgresql-examples