Print-friendly Version

Creating a RubyGems Mirror With HTTP Basic Authentication

William D. Pool, September 2008

Would you like to create your own RubyGems remote repository that bypasses the sometimes overloaded RubyForge mirrors? Here's how. In addition, you can see how to create a repository that is user/password protected, which allows you to restrict the repository for remote and internal use, preventing public Internet usage of the server and your company's bandwidth. This article also explains how to set up each user environment to access the remote repository that is created.

Note: The following environment was used for the procedures in this article: Solaris 10 Operating System, Apache version 2.2.6, Ruby version 1.8.6p114, and RubyGems version 1.2.0. The Apache2 package was installed via CSK 1.2, see Optimized Open Source Software Stack (Cool Stack).

Repository Access Method After Completion

After using the procedures in this article, you should have two repositories that have a setup similar to the following (but with a different domain, that is, not blah.com) and a user/password combination:

• http://gems.blah.com/dev: This gets updated from RubyForge every Sunday morning at 3 a.m.
• http://gems.blah.com/stable: This is a stable list of gems used for QA, staging, and production environments. This should be used for environments that standardize on software.

You can then access your repository using the following authentication method:

user: gems
pass: password

Prerequisites

1. First, upgrade your RubyGems to the latest and greatest version, if possible (at the time of this writing, 2.1.0).

2. Then download the prerequisite Cool Stack (CSKruby) package and install it.

3. Use the following command to install the gem builder (XML Builder library) that is used to create the yaml index:

gem install builder

Creating RubyForge Mirror

1. Set up the mirror configuration file by creating a /root/.gemmirrorrc file with the following:

---
- from: http://gems.rubyforge.org/
  to: /gemrepo/dev

This example creates an "everything" repository that's the latest and greatest within the /gemrepo/dev directory. Change this directory location to suit your needs.

2. To download gems, run the gem command to create the mirror. (You may want to run this in screen mode.)

gem mirror

3. Create a yaml index file that has all the gem information. (You may want to run this in screen mode.)

gem generate_index -d /gemrepo/dev

Note: This command should be run in the directory that contains the gems directory.

Sample output:

gem generate_index -d /gemrepo/dev
Loading 15132 gems from /gemrepo/dev
............................................
............................................
............................................
...

Setting Up Apache Permissions

Next, set up HTTP basic authentication against the gem repository, so users must authenticate to access the repository. Requiring authentication enables you to control who can use the server and prevents the repository from being publicly available. This is useful if you want to control what is distributed.

For instance, here is an example based on the domain blah.com. Change the domain to match your environment settings.

<VirtualHost *:80>
        ServerName gems.blah.com
        ServerAdmin webmaster@blah.com
        DocumentRoot /gemrepo
        <Directory />
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
        </Directory>

        <Directory /gemrepo/ >
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
       AuthName "Dev Access"
        AuthType Basic
        AuthBasicProvider file
        AuthUserFile /opt/coolstack/apache2/conf/htpasswd.users
        Require user gems
# user: gems
# pass: password
                Order allow,deny
                allow from all
        </Directory>

        ErrorLog /var/log/httpd/error-gems.log
        LogLevel warn
        CustomLog /var/log/httpd/access-gems.log combined
        ServerSignature Off

</VirtualHost>

Configuring Ruby Client

Since you're using your own repository with a password (so the public doesn't mooch off of you), you need to include a $HOME/.gemrc file by using the following:

# cat $HOME/.gemrc
gem: --source http://gems:password@gems.blah.com/dev/

Note: The /dev/ can be changed to match your environment, for example, /stable/.

Creating Other Repositories

The /dev repository has every gem available and its version. You can then create a /gemrepo/stable/gems directory with all the gems you standardize for, let's say, production, staging, and QA, since these should have a similar repository setup.

Once, you copy the gems, you then just create the index in the parent, for example:

cd /gemrepo/stable
gem generate_index -d /gemrepo/stable

Change your $HOME/.gemrc file to reflect the changes.

Keeping Your Repository Up to Date

Create a script that will update the repositories.

# cat /root/bin/gem-mirror.sh

#!/bin/ksh
# Author: William D. Pool
# Description: This will update the dev mirror with
# The latest gems. It will also run an index on dev / stable
# So, everything is clean, you still have to manually place
# a gem into the /gemrepo/stable/gems directory for it to be
# indexed!
#
# Put into Crontab
#
# 30 3 * * 0 /root/bin/gem-mirror.sh > /dev/null 2>&1
#
gem mirror --no-verbose -q
gem generate_index -d /gemrepo/dev
gem generate_index -d /gemrepo/stable

Throw it into cron!

30 3 * * 0 /root/bin/gem-mirror.sh > /dev/null 2>&1

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under this License.