Retweet 0.2 : bump to Python 3

debian, planetpython, sysadmin

Follow me on Identi.ca  or Twitter  or Diaspora*diaspora-banner

Don’t know Retweet? My last post about it introduced this small Twitter bot whichs just retweets (for now) every tweets from a Twitter account to another one.

Retweet

Retweet was created in order to improve the Journal du hacker Twitter account. The Journal du hacker is a Hacker News-like French-speaking website.

logo-journal-du-hacker

Especially useful to broadcast news through a network of Twitter accounts, Retweet was improved to bump Python version to 3.4 and to improve pep8 compliance (work in progress).

The project is also well documented and should be quite simple to install, configure and use.

After my first blog post about Retweet, new users gave me feedback about it and I now have great ideas for future features for the next release.

Twitter_logo_blue

What about you? If you try it, please tell me what you think about it, opening a bug request or ask for new features. Or just write your comment here ;)

Retweet, a simple bot to retweet tweets from a user

debian, planetpython, sysadmin

Follow me on Identi.ca  or Twitter  or Diaspora*diaspora-banner

For a small project related to the french-speaking Hacker News-style website le Journal du hacker, I needed to retweet every tweets from a Twitter account to another one.

So here it is : Retweet

I wanted to keep it simple, stupid, and coded in Python if I needed to extend it myself for the projet Le Journal du hacker.

logo-journal-du-hacker

It was quite easy to reach these goals, but the project is also well documented and should be quite simple to install, configure and use.

What about you? Do you also have this kind of needs? Don’t hesitate to fork the project, report bugs, ask for feature requests and let us know in the comments of this post ;)

 

My Free activities in July 2015

april, debian, debian-fr, planet-sysadmin, planetpython, python, sysadmin

Follow me on Identi.ca  or Twitter  or Diaspora*diaspora-banner

Here are the details of my Free activities in July 2015.

Carl Chenet’s projects:

Patchs:

  • planet.sysadmin-fr.org – pull request to add Carl Chenet’s blog – #5
    • Carl Chenet’s blog is now on planet.sysadmin-fr.org, which is an aggregator for french-speaking sysadmins. You should git it a try!

Debian bug reports:

  • Docker.io – Manual page for the docker-compose executable is missing – #792518
  • Docker.io – New Docker version 1.7.1 available – #793483
  • Backupchecker – now available in Debian Stretch (Testing) – Backupchecker migration log report

Other bug reports:

  • Docker-compose – A manual page for docker-compose – #1727
  • feedDiasp – Failed to login: ‘NoneType’ object has no attribute ‘group#6
  • rss-bot-diasp – feedDiasp.feedDiasp.Diasp.LoginException: ‘NoneType’ object has no attribute ‘group’ – #1

Feature requests:

  • Weboob – SFR mobile phone Invoices in sfr module – #2045

Backup Checker, the ServerSpec for your backups

debian, planet-cullt, planet-sysadmin, planetpython, sysadmin

Follow me on Identi.ca  or Twitter  or Diaspora*diaspora-banner

Sysadmins usually know ServerSpec, a tool allowing to check if your servers are correctly configured. The use of automated deployment tools makes deployments easier and easier, but lot of unexpected situations can come up using this tool, especially when more and more servers are implied. ServerSpec allows to verify that the result of your deployments are what you were expecting for and that no exception or unexpected event goes undetected.

serverspec

Backup Checker (Github stars appreciated :)) offers the same kind of controls, but for your backups. Once you have backups, you don’t know if they contain what you are expecting for. And this is especially important because having broken backups mean losing your data after a major outage! Moreover if you think you are well protected with backups but they are eventually broken (corrupted archives, archives being filled with empty files, saving useless files…), you will perform your server or database upgrades without being really protected, leading to terrible situations.

github-logo

Backup Checker on github

 

Backup Checker offers lots of controls over backups:

  • check if an archive is corrupted
  • control if some files or directories exist inside an archive exist
  • check if a file size inside an archive is not null or has a specific size (operators smaller than, equals, greater than supported)
  • owner, group, uid, gid of files/directories inside an archive
  • and a lot more. Check the official documentation!
Use Backup Checker with Backup-Manager or Rsnapshot

Starting from Backup Checker 1.7,if you use a backup tool like Backup-Manager or Rsnapshot, you’ll be glad to know that Backup Checker perfectly works with them. The documentation is available on the Backup Checker Community page with the howtos to install and configure Backup Checker with Backup-Manager or with Rsnapshot.

weneedyou

What about you? Let us know in the comments what you think of Backup Checker and its features to connect with other backup tools. We would be happy to get your feedbacks about how you use Backup Checker or what you expect from a backup checking solution.

Verify the backups of backup-manager

debian, planetpython, sysadmin

Follow me on Identi.ca  or Twitter  or Diaspora*diaspora-banner

Backup-manager is a tool creating backups and storing them locally. It’s really usefult to keep a regular backup of a quickly-changing trees of files (like a development environment) or for traditional backups if you have a NFS mount on your server. Backup-managers is also able to send backup itself to another server by FTP.

In order to verify the backups created by backup-manager, we will use also Backup Checker (stars appreciated :) ), the automated tool to verify backups. For each newly-created backup we want to control that:

  • the directory wip/data exists
  • the file wip/dump/db.sql exists and has a size greater than 100MB
  • the files wip/config/accounts did not change and has a specific md5 hash sum.
Installing what we need

We install backup-manager and backup checker. If you use Debian Wheezy, just use the following command:

apt-key adv --keyserver pgp.mit.edu --recv-keys 2B24481A \
&& echo "deb http://debian.mytux.fr wheezy main" > /etc/apt/sources.list.d/mytux.list \
&& apt-get update \
&& apt-get install backupchecker backup-manager

Backup Checker is also available for Debian Squeeze, Debian Sid, FreeBSD. Check out the documentation to install it from PyPi or from sources.

Configuring Backup-Manager

Backup-manager will ask what directory you want to store backups, in our case we choose /home/joe/dev/wip

In the configuration file /etc/backup-manager.conf, you need to have the following lines:

export BM_BURNING_METHOD="none"
export BM_UPLOAD_METHOD="none"
export BM_POST_BACKUP_COMMAND="backupchecker -c /etc/backupchecker -l /var/log/backupchecker.log"
Configuring Backup Checker

In order to configure Backup Checker, use the following commands:

# mkdir /etc/backupchecker && touch /var/log/backupchecker.log

Then write the following in /etc/backupchecker/backupmanager.conf:

[main]
name=backupmanager
type=archive
path=/var/archives/laptop-home-joe-dev-wip.%Y%m%d.master.tar.gz
files_list=/etc/backupchecker/backupmanager.list

You can see we’re using placeholders for the path value, in order to match each time the latest archive. More information about Backup Checker placeholders in the official documentation.

Last step, the description of your controls on the backup:

[files]
wip/data| type|d
wip/config/accounts| md5|27c9d75ba5a755288dbbf32f35712338
wip/dump/dump.sql| >100mb
Launch Backup Manager

Just launch the following command:

# backup-manager

After Backup Manager is launched, Backup Checker is automatically launched and verify the new backup of the day where Backup Manager stores the backups.

Possible control failures

Lets say the dump does not have the expected size. It means someone may have messed up with the database! Backup Checker will warn you with the following message in /var/log/backupchecker.log:

$ cat /var/log/backupchecker.log
WARNING:root:1 file smaller than expected while checking /var/archives/laptop-home-joe-dev-wip-20150328.tar.gz: 
WARNING:root:wip/dump/dump.sql size is 18. Should have been bigger than 104857600.

Other possible failures : someone created an account without asking anyone. The hash sum of the file will change. Here is the alert generated by Backup Checker:

$ cat /var/log/backupchecker.log
WARNING:root:1 file with unexpected hash while checking /var/archives/laptop-home-joe-dev-wip-20150328.tar.gz:
WARNING:root:wip/config/accounts hash is 27c9d75ba5a755288dbbf32f35712338. Should have been 27c9d75ba3a755288dbbf32f35712338.

Another possible failure: someone accidentally (or not) removed the data directory! Backup Checker will detect the missing directory and warn you:

$ cat /var/log/backupchecker.log
WARNING:root:1 file missing in /var/archives/laptop-home-joe-dev-wip-20150328.tar.gz: 
WARNING:root:wip/data

Awesome isn’t it? The power of a backup tool combined with an automated backup checker. No more surprise when you need your backups. Moreover you spare the waste of time and efforts to control the backup by yourself.

weneedyou

What about you? Let us know what you think of it. We would be happy to get your feedbacks. The project cares about our users and the outdated feature was a awesome idea in a feature request by one of the Backup Checker user, thanks Laurent!

 

Unverified backups are useless. Automatize the controls!

debian, planetpython, python, sysadmin

Follow me on Identi.ca  or Twitter  or Diaspora*diaspora-banner

Unverified backups are useless, every sysadmins know that. But manually verifying a backup means wasting time and resources. Moreover it’s boring. You should automatize it!

charlot

Charlie Chaplin Modern Times

Backup Checker is a command line software developed in Python 3.4 on GitHub (stars appreciated :) ) allowing users to verify the integrity of archives (tar, gz, bz2, lzma, zip, tree of files) and the state of the files inside an archive in order to find corruptions or intentional of accidental changes of states or removal of files inside an archive.

github-logo

Backup Checker on github

The new feature of the latest version 1.4 is the control of outdated archives with the new outdated parameter. Lots of data are outdated quite fast, because they are dependent of other data, or because they are only useful in a specific context.

Hey, this database dump is 6 months old, it’s useless today!

Backup Checker now controls the expiration duration and triggers a warning if the given duration starting from the last modification of the archive (mtime) is expired. Short examples of the warning:

WARNING:root:/backups/backups-12022015.tar.gz is outdated. Was good until 01/03/15 00:00:00 – now 22/03/15 21:38:20

You won’t be surprized any more by outdated useless data in your backups.

Backup Checker also offers lots of other controls. Check the features list!

Installing Backup Checker

Backup Checker is available from PyPI using the following command:

# pip3.4 install backupchecker

It’s also available for your Debian Squeeze or Debian Wheezy. Check how to get it for your specific distributions.

weneedyou

What about you? How and what for do you use Backup Checker? We would be happy to get your feedbacks. The project cares about our users and the outdated feature was a awesome idea in a feature request by one of the Backup Checker user, thanks Laurent!

 

Verify your backups… or lose them all

debian, planetpython, python, sysadmin

Follow me on Identi.ca  or Twitter  or Diaspora*diaspora-banner

As a good sysadmin, you thought you had backups for your server but you didn’t verify them before the major migration you attempted? When you need them, they’re empty or full of useless files. And now it’s too late…

nobackup

Wen you discover that you don’t have usable backups – lesjoiesdusysadmin.fr

You won’t guess how often this situation occurs.

Backup Checker is a command line software developed in Python 3.4 on GitHub (stars appreciated :) ) allowing users to verify the integrity of archives (tar, gz, bz2, lzma, zip, tree of files) and the state of the files inside an archive in order to find corruptions or intentional of accidental changes of states or removal of files inside an archive.

github-logo

Install Backup Checker from PyPI

The easiest way to install Backup Checker is from PyPi using the following command:

$ pip3.4 install backupchecker
Debian Wheezy and Squeeze packages for Backup Checker

Backup Checker Debian packages are now available for your stable servers Wheezy and Squeeze in the MyTux Debian repositories.

Debian Squeeze

Just copy/paste the following command on your server to add the MyTux Debian Squeeze repository and install Backup Checker:

apt-key adv --keyserver pgp.mit.edu --recv-keys 2B24481A \
&& echo "deb http://debian.mytux.fr squeeze main" > /etc/apt/sources.list.d/mytux.list \
&& apt-get update \
&& apt-get install backupchecker
debian-squeeze-toy
Debian Wheezy

Just copy/paste the following command on your server to add the MyTux Debian Wheezy repository and install Backup Checker:

apt-key adv --keyserver pgp.mit.edu --recv-keys 2B24481A \
 && echo "deb http://debian.mytux.fr wheezy main" > /etc/apt/sources.list.d/mytux.list \
 && apt-get update \
 && apt-get install backupchecker

wheezy

Using Backup Checker

2 steps are needed to secure your backups. First you need to generate the configuration files of your backups, using the following command:

$ backupchecker -G /backups/backup-08032015.tar.gz

This generates 2 files /backups/backup-08032015.conf and /backups/backup-08032015.list you need to store in order to verify this archive later (the -O option lets you define a custom location to store the files).

Second step is to verify your backup. It is as simple as:

$ backupchecker -c /etc/backupchecker/backup-08032015.conf -l /var/log/backupchecker.log

Then check if warnings have been sent to /var/log/backupchecker.log. Really simple isn’t it? Scripting this command, your backups are now verified and secured. If any modification occurs, it will be detected and pinpointed.

What do you thing about it?

As usual, any feedback is welcome, through bug reports, emails of the author or comments on this blog.

Several companies now use Backup Checker to secure their backups. Let us know if we can help you.

 

Backup Checker 1.2 : verify remote backups

debian, planetpython, sysadmin

Follow me on Identi.ca  or Twitter  or Diaspora*diaspora-banner

Backup Checker is a command line software developed in Python 3.4, allowing users to verify the integrity of archives (tar,gz,bz2,lzma,zip,tree of files) and the state of the files inside an archive in order to find corruptions or intentional of accidental changes of states or removal of files inside an archive.

The major feature of this new version is the ability of Backup Checker to use Unix streams. Using classic Unix tools like OpenSSH or wget, Backup Check is able to verify a remote tar.{gz,bz2,xz} archive. The following example verifies a tar.gz archive located on remote server through SSH:

$ ssh -q server "cat /tmp/backup.tar.gz" | ./backupchecker.py -c . -

Another short example with the FTP protocol, to verify a tar.bz2 archive located on a remote server through FTP:

$ wget --quiet -O - ftp://user:pass@server/backup.tar.gz | ./backupchecker.py -c . -

Moreover in this release, a new option –configuration-name allows the user to define a custome name for the files generated by Backup Checker (default is defined from the name of the archive using the -g or -G options).

It is a major step for Backup Checker. It is indeed easier and easier to use Backup Checker in your own scripts, allowing to fully automate your backup controls.

Several companies now use Backup Checker to secure their backups. Let us know if we can help you.

As usual, any feedback is welcome, through bug reports, emails of the author or comments on this blog.

Backup Checker 1.0, the fully automated backup checker

debian, python, sysadmin

Follow me on Identi.ca  or Twitter  or Diaspora*diaspora-banner

Backup Checker is the new name of the Brebis project.

Backup Checker is a CLI software developed in Python 3.4, allowing users to verify the integrity of archives (tar,gz,bz2,lzma,zip,tree of files) and the state of the files inside an archive in order to find corruptions or intentional of accidental changes of states or removal of files inside an archive.

Brebis version 0.9 was downloaded 1092 times. In order to keep the project growing, several steps were adopted recently:

  • Brebis was renamed Backup Checker, the last one being more explicit.
  • Mercurial ,the distributed version control system of the project, was replaced by Git.
  • The project switched from a self hosted old Redmine to GitHub. Here is the GitHub project page.

This new version 1.0 does not only provide project changes. Starting from 1.0, Backup Checker now verifies the owner name and the owner group name of a file inside an archive, enforcing the possible checks for both an archive and a tree of files.

Moreover, the recent version 0.10 of Brebis published 9 days ago provided the following features

  • The default behaviour calculated the hash sums of every files in the archive or the tree of files, this was discontinued because of poor performances while using Backup Checker on archives of large size.
  • You can force the old behaviour by using the new –hashes option.
  • The new –exceptions-file option allows the user to provide a list of files inside the archive in order to compute their hash sums.
  • The documentation of the project is now available on Readthedocs.

As usual, any feedback is welcome, through bug reports, emails of the author or comments on this blog.

Send the same short message on Twitter, Pump.io, Diaspora*… and a lot more

debian, planet-cullt, sysadmin

Follow me on Identi.ca  or Twitter  or Diaspora*diaspora-banner

This is a feedback about installing a self hosted instance of Friendica on a Debian server (Jessie). If you’re not interested in why I use Friendica, just go to « Prerequisite for Friendica » section below.

Frustration about social networks

Being a huge user of short messages, I was quite frustated to spend so much time on my Twitter account. To be quite honest, there is no much I like about this social network, except the huge population of people being potentially interested in what I write.

twitter-logo

I also have been using Identi.ca (now powered by Pump.io) for a while. But I tried for a while to manage both networks Pump.io and Twitter by hand and it was quite painful. And something was telling me another social network was going to appear from nowhere one of these days and I’ll be just horrible to try to keep it up this way.

identica-logo

So I was looking for a « scalable » solution not asking too much personal investment. Subscribing to Diaspora* some days ago on the Framasphere pod, tintouli told me to try Friendica.

Hmmm, what’s Friendica ?

f-intro

Friendica is a content manager you can plug on almost anything: social networks (Facebook, Twitter, Pump.io, Diaspora*,…), but also WordPress, XMPP, emails… I’m in fact just discovering the power of this tool but to plug it on my different social network accounts was quite a good use case for me. And I guess if you’re still reading, for you too.

I tried to use some shared public servers but I was not quite happy with the result, one connector was still missing or public servers were really unstable. So I’m at last self hosting my Friendica. Here is how.

Prerequisite for Friendica

You need to install the following packages:

# apt-get install apache2 libapache2-mod-php5 php5 php5-curl php5-gd php5-mysql mysql-server git

Having a already self-modified /etc/php5/apache2/php.ini, I encountered a small issue with libCurl and had to manually add the following line in the php.ini:

extension=curl.so

Setting up MySQL

Connect to MySQL and create an empty database with a dedicated user:

# mysql -u root -pV3rYS3cr3t -e « create database friendica; GRANT ALL PRIVILEGES ON friendica.* TO friendica@localhost IDENTIFIED BY ‘R3AlLyH4rdT0Gu3ss' »

mysql

Setting up Apache

My server hosts several services, so I use a subdomain friendica.mydomain.com. If you use a subdomain, of course check you do have declared this subdomain in your DNS zone.

apache-logo

I use SSL encryption with a wildcard certificate for all my subdomains. My Friendica data are stored in /var/www/friendica. Here is my virtual host configuration for Friendica stored in the file /etc/apache2/sites-available/friendicassl.conf :

<VirtualHost *:443>
ServerName friendica.mydomain.com
DocumentRoot /var/www/friendica/
DirectoryIndex index.php index.html
ErrorLog /var/log/apache2/friendica-error-ssl.log
TransferLog /var/log/apache2/friendica-access-ssl.log

SSLEngine on
SSLCertificateFile /etc/ssl/certs/mydomain/mydomain.com.crt
SSLCertificateKeyFile /etc/ssl/private/mydomain.com.key
SSLVerifyClient None

<Directory /var/www/friendica/>
AllowOverride All
Options FollowSymLinks
Order allow,deny
Allow from all
</Directory>
</VirtualHost>

After writing the configuration file, just launch the following commands and it should be good for the Apache configuration:

# a2ensite friendicassl && /etc/init.d/apache2 reload

friendica-trust

Setting up Friendica

Get the master zip file of Friendica, copy it on your server and decompress it. Something like :

# cd /var/www/ && wget https://github.com/friendica/friendica/archive/master.zip && unzip master.zip && mv friendica-master friendica

You need to give www-data (Apache user) the rights to write in /var/www/friendica/view/smarty3/ :

# chown -R www-data:www-data /var/www/friendica/view/smarty3 && chmod -R ug+w var/www/friendica/view/smarty3

Ok, I guess we’re all set, lets launch the installation process! Using your web browser, connect to friendica.mydomain.com. First step you’ll see the installation window which checks the prerequisite before installing. Complete if something is missing.

friendica-installation

First window of the Friendica installation process

Second step asks the host/user/password of the database, complete and the installation process starts. Hopefully all goes just fine.

Next you’ll have to create a /var/www/friendica/.htconfig.php with the content that the last page of the installation process provides. Just copy/paste, check the rights of this file and now you can connect again to see the register page of friendica at the url https://friendlica.mydomain.com/register . Pretty cool!

Register a user

That’s a fairly easy step. You just need to check before that your server is able to send emails, because the password is going to be sent to you by email. If it is ok, you should now identify on the welcome page of friendica and access your account. That’s a huge step to broadcast your short messages everywhere, but we have some last steps before being able to send your short messages on all the social networks we need.

A small break, create an app for twitter on apps.twitter.com

To send your short messages to Twitter, you need to create an app on apps.twitter.com. Just check you’re logged in Twitter and connect to apps.twitter.com. Create an app called with a unique name (apparently), then go to the Keys and Access tokens page, note the consumer key and the consumer secret. You’ll later need the name of the app, the consumer key and the consumer secret.

friendica-decentralized-network

Install and configure the addons

Friendica uses an addon system in order to plug on the different third-parties it needs. We are going to configure Twitter, Pump.io and the Diaspora* plug. Let’s go back to our server and launches some commands:

# cd /tmp && git clone https://github.com/friendica/friendica-addons.git && cd friendica-addons

# tar xvf twitter.tgz -C /var/www/friendica/addon

# tar xvf pumpio.tgz -C /var/www/friendica/addon

# cp -a diaspora /var/www/friendica/addon

You need to modify your /var/www/friendica/.htconfig.php file and add the following content at the end:

// names of your addons, separated by a comma

$a->config[‘system’][‘addon’] = ‘pumpio, twitter, diaspora’;

// your Twitter consumer key
$a->config[‘twitter’][‘consumerkey’] = ‘P4Jl2Pe4j7Lj91eIn0AR8vIl2’;

// your Twitter consumer secret
$a->config[‘twitter’][‘consumersecret’] = ‘1DnVkllPik9Ua8jW4fncxwtXZJbs9iFfI5epFzmeI8VxM9pqP1’;

// you Twitter app name

$a->config[‘twitter’][‘application_name’] = « whatever-twitter »;

banner_friendica

Connect again to Friendica. Go to settings => social networks, you will see the options for Twitter, Pump.io and Diaspora*. Complete the requested information for each of them. Important options you should not forget to check are:

Pump.io

  • Enable pump.io Post Plugin
  • Post to pump.io by default
  • Should posts be public?

Twitter

  • Authorize publication on Twitter
  • Post to Twitter by default

Diaspora*

  • Post to Diaspora by default

Done? Now it’s time to send your first broadcasted short message. Yay!

Send a short message to your different social networks

Connect to Friendica, click on the network page, write your short message in the « Share » box. Click on the lock lock, you’ll see the following setup:

friendica-post-default

It means your short messages will be broadcasted to the three networks. Or more, it’s up to you! That’s my setup, feel free to modify. Now close the lock window and send your message. For me it takes some time to appear on Twitter and Diaspora* and it immediatly appears on Identi.ca.

Last words

friendica-stand

Friendica offers to take back the control of your data, by broadcasting content on different media from a single source. While self hosting, you keep your data whatever happends and are not subject to companies losing your data like recently Twitpic. Moreover the philosophy behind Friendica pushed me to dig and test the solution

What about you? How do you proceed to broadcast your short messages? Does Friendica offer a good solution in your opinion? Are you interested in the philosophy behind this project? Feel free to share your thoughs in the comments.

LAST MINUTE: hey, this article is on Hacker News, don’t hesitate to vote for it if you liked it!