Posts Tagged Amazon S3

Install s3fs on Amazon Clouds

Posted by smoreau in Linux on 03 Apr 2012

s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files).

The following instructions detail the steps to install the program s3fs on an Amazon EC2 running Debian 5.0.5.

  1. Install libfuse
    First, you need to install the package libfuse manually as the one provided via apt-get is too old (s3fs needs a version greater than or equal to 2.8.4).

    wget http://downloads.sourceforge.net/project/fuse/fuse-2.X/2.8.7/fuse-2.8.7.tar.gz
tar xzf fuse-2.8.7.tar.gz
cd fuse-2.8.7
./configure --prefix=/usr
make install
  2. Install libxml
    You can simply install the package provided by apt-get:

    apt-get install libxml2-dev
  3. Upgrade mount
    Because of a problem between fuse and mount, you need to upgrade the version of mount:

    wget http://www.kernel.org/pub/linux/utils/util-linux/v2.21/util-linux-2.21-rc1.tar.gz
tar xzf util-linux-2.21-rc1.tar.gz
./configure --prefix=/usr --without-ncurses
make install

    For more information about this issue, please go to the following page: http://code.google.com/p/s3fs/issues/detail?id=228

  4. Install s3fs
    You can now install s3fs using the following commands:

    wget http://s3fs.googlecode.com/files/s3fs-1.61.tar.gz
tar xvzf s3fs-1.61.tar.gz
cd s3fs-1.61/
./configure --prefix=/usr
make install

Note that I wanted to use s3fs to create incremental snapshot-style backups with rsync. Unfortunately, as mentioned on the following page, it didn’t work because s3fs doesn’t support hard links: http://code.google.com/p/s3fs/issues/detail?id=46

, , , , , , , , , , ,

No Comments

S3 command failed if the time is not synced

Posted by smoreau in Linux on 06 Dec 2011

This is already the second post about the s3sync ruby program. The first article was focused on monitoring s3sync with Zabbix.

I will talk on this one about an error I got when running the S3 synchronisation:

S3 command failed:
list_bucket prefix /data max-keys 200 delimiter /
With result 403 Forbidden
S3 ERROR: #
s3sync.rb:290:in `+': can't convert nil into Array (TypeError)
	from s3sync.rb:290:in `s3TreeRecurse'
	from s3sync.rb:346:in `main'
	from ./thread_generator.rb:79:in `call'
	from ./thread_generator.rb:79:in `initialize'
	from ./thread_generator.rb:76:in `new'
	from ./thread_generator.rb:76:in `initialize'
	from s3sync.rb:267:in `new'
	from s3sync.rb:267:in `main'
	from s3sync.rb:735

As you can see, this error is not very human-friendly! 😮 The only thing we know is that the S3 command failed because of the error can't convert nil into Array. It looks to me like an internal error within s3sync…

But after some investigation, it appears it is simply because the system date on the server is not correct. I cannot tell you how much time I spent on this one!  😯

Anyway, if you are doing automatic backups as describe on John Eberly’s blog, you need to add the following code at the top of your upload.sh script:

# update the system date
/usr/sbin/ntpdate 3.uk.pool.ntp.org 2.uk.pool.ntp.org 1.uk.pool.ntp.org 0.uk.pool.ntp.org

NB: please find below the command lines I use to install ntpdate on a Debian server:

apt-get install ntpdate
dpkg-reconfigure tzdata

, , , , , , ,

2 Comments

Monitor s3sync with Zabbix

Posted by smoreau in Linux on 24 Nov 2011

s3sync is a ruby program that easily transfers directories between a local directory and an S3 bucket:prefix. It behaves somewhat, but not precisely, like the rsync program.

I am using this tool to automatically backup the important data from Debian servers to Amazon S3. I am not going to explain here how to install s3sync as it is not the purpose of this article. However, you can read this very useful article from John Eberly’s blog: How I automated my backups to Amazon S3 using s3sync.

If you followed the steps from John Eberly’s post, you should have an upload.sh script and a crontab job which executes this script periodically.

From this point, here is what you need to do to monitor the success of the synchronisation with Zabbix:

  1. Add the following code at the end of your upload.sh script: 
    # print the exit code
RETVAL=$?
[ $RETVAL -eq 0 ] && echo "Synchronization succeed"
[ $RETVAL -ne 0 ] && echo "Synchronization failed"
  2. Log the output of the cron script as follow: 
    30 2 * * sun /path/to/upload.sh > /var/log/s3sync.log 2>&1
  3. On Zabbix, create a new item which will check the existence of the sentence “Synchronization failed” in the file /var/log/s3sync.log:

    Item key: vfs.file.regmatch[/var/log/s3sync.log,Synchronization failed]
  4. Still on Zabbix, define a new trigger for the previously created item:

    Trigger expression: {Template_AmazonCloud_Debian:vfs.file.regmatch[/var/log/s3sync.log,Synchronization failed].last(0)}=1

With these few steps, you should now receive Zabbix alerts when a backup on S3 fails. 🙂

, , , , , , , , ,

No Comments