Cywin SSHD Setup

By brian | Filed in Uncategorized

Running an open ssh server (SSHD) is quite easy. And it allows you to run Linux style backups on a windows server (as I’ve previously posted about).

To set up SSHD you need to:

  1. run SSHD config setup
  2. make config changes for your environment
  3. install SSHD as a service

To run SSHD Config Setup, open cygwin prompt (and run it as administrator if on server 2008 or windows7) and run:

$ ssh-host-config -y

(Note, the -y option answers yes automatically to all yes/no prompts)

The script will ask you for a password for the cyg_server user account it will create to run sshd as a windows service

The sshd config file is now located in /etc/ssh_config. You can make config changes to secure your SSHD server. To secure the server I would normally change the port sshd listense on to something abnormal, and turn off password authentication, allowing only RSA2 key based authentication available.

Finally you need to run the service:

cygrunsrv -S sshd

Finally you need to add your public key to the user on the server you want to connect as in the users ~/.ssh/authorized_keys file if you disabled password authenticaiton in the /etc/ssh_config file.

Dont forget to open the configured ssh port on the firewall, otherwise, your connection will timeout.

After that you should be able to successfully connect to the server using open ssh client on linux or windows.

Be the first to comment
Tags: ,
del.icio.us this! Digg this! RSS 2.0 TOP

Gitolite Install

By brian | Filed in Uncategorized

Installing Gitolite is very easy, particularly if you are using Git on recent version of Ubuntu (10.10 I think).

Assumptions:

  1. You are working on a client pc of some sort
  2. You want to install gitolite onto a server which is not the client
  3. You already have openssh access from the client to the server using RSA keys

All you need to do is:

  1. Install the package (gitolite) on the server
  2. Set up Gitolite on the server
  3. Configure repositories in the Gitolite admin repository from the client

Installing Gitolite

As it is in the Ubuntu packages, all you need to do is run these commands on the server.

> sudo apt-get update
> sudo apt-get install gitolite

And your done.

Setup Gitolite

You need to have a public key somewhere public accessible (world readable) on the server to setup gitolite.

If you have ssh’d into the server using an RSA key already, copy your public key to the temp directory and make it world readable by running these commands.

> cp ~/.ssh/id_rsa.pub /tmp/[username]-id_rsa.pub
> chmod 666 /tmp/[username]-id_rsa.pub

Then you need to run the gitolite setup referencing your public key. This is so you can admin the server once you have run the setup.

> sudo su gitolite
> gl-setup /tmp/[username]-id_rsa.pub

Gitolite is now setup.

Configure Repositories

Now that you have it setup, on your client pc, you should be able to clone the admin repository to start configuring new repositories.

[sourecode language=shell]
> git clone gitolite@[serverurl]:gitolite-admin gitolite-admin
[/sourcecode]

You now have a local copy of the admin repository, which you make changes to, and then push back to the server. (You could do this on the server itself if you wanted to as well).

The admin repository has 2 directories.

  • keydir = contains user public keys in the form [username].pub
    where [username] could be their name or email address or any other identifier you find useful.
  • conf/gitolite.conf = is the default config file containing the name’s of repositories, and user access permissions
    this file can be split up if desired.

Config File Options

To define a repository insert a line like this

repo    [repository path and or name]
RW+  =  [username list seperated by spaces]

Note: username is the same as the key file name without the .pub at the end

To define a list of users in an easy list, insert a line like this

@[niceNameForUsers]  =  [useranme list seperated by spaces]

Which can make it a little easier to read who has permissions if you define a shortcut for group of users.

Permissions can be defined as these options

  • R = read permission to the repository for the users listed
  • RW = read/write permission to the repository for the users listed
  • RW+ = read/write/nonfastforwardwrite permissions to the repository for the users listed

The RW+ Non fast forward write means you can push a branch that removes history using the force option.
Standard RW permission does not allow you to remove history already part of an existing branch.

There are plenty of other config options, including exposing repositories over GitWeb if you setup GitWeb, as well as other shortcut options, but that gives you the ability to create repositories and give read, or read/write permissions to the repositories.

 

Be the first to comment
Tags: ,
del.icio.us this! Digg this! RSS 2.0 TOP

Git Hosting Options

By brian | Filed in Uncategorized

I’m in the midst of installing GitHub into our organisation. GitHub has a commercial offering called GitHub:Fi which allows you to install and run a copy of GitHub within your own network behind your firewall. Which is great for a large organisation that wants the GitHub experience, but are unwilling to let their source code outside their network. For a small business however, its not very costly, and probably out of reach.

There are a few other options though.

  1. Gitorious is a open source equivalent to GitHub, you can purchase support to install Gitorious and access the developers as consultants to work on new features or integration with existing systems. It seems they are more open to customisations than GitHub.
  2. Gitolite is a great little app that allows you to create and manage the hosting of Git repositories through a Git repository. Its simple, fast and very lightweight. It does not have a web interface to manage and view repositories, but it allows for very fine controlled security which can be favorable in a corporate environment. There are quite large installs of Gitolite, so much so that they have a dedicated “big config” flag to change the way it configures itself internally in situations where you have several thousand repositories or users. So its been well tested.
  3. Gitosis is an alternative to Gitolite. It has been around longer, but does not have as many features for controlling security(at the time I reviewed them).
  4. For small business’s I have been using the fantastic service at Repository Hosting (http://repositoryhosting.com/) which is unbeatable on price, and allows the hosting of git, mercurial or svn repositories and for each repository also runs trac for issue management, and a webdav access for file sharing on a project.

At the moment I’m using Gitolite, and for fun, I’m exploring the use of Gitorious instead (I don’t really need the web administration side, but I like the idea of it).

Be the first to comment
Tags: ,
del.icio.us this! Digg this! RSS 2.0 TOP

Automation HG -> Git

By brian | Filed in Uncategorized

Previously I manually converted HG repositories to Git as posted here.

I’ve now written a script to automate conversion of a HG repository to a Git Repository.

I run this script under Jenkins, so anyone in the organisation can use it.

The process is:

  1. They run the “HG serv” command on the repository they want converted
  2. They run the jenkins job to do the conversion which asks them for
    the http url to do the hg clone from
    and the name of the new repository
  3. The script clones the HG repo, converts to git
  4. I’m using the Jenkins publish to CIFS plugin to then push their repo to a windows file share they can access

All done, with no input.

Note, more work would possibly need to be done if the hg bookmarks plugin was being used, this script does work well for a repository that just has one head.

Source code of the script is quite simple. But it makes it very easy to migrate.

#!/bin/bash

echo "environment: "
env
# SET

HGRepoURL=$HgHttpLocation
HGRepoName=$GitRepositoryName.hg
GITRepoName=$GitRepositoryName.git
BaseLocation=$GitRepositoryName

echo " "
echo "Config... "
echo "Base Location: $BaseLocation"
echo "HG Repository URL: $HGRepoURL"
echo "HG Repository Name: $HGRepoName"
echo "Git Repository Name: $GITRepoName"
echo " "

echo "**** DO BASE LOCATION CREATE ****"
echo " "
mkdir $BaseLocation
if [ $? -ne 0 ]; then
echo "FATAL: Unable to create base location"
exit 1
fi
cd $BaseLocation
if [ $? -ne 0 ]; then
echo "FATAL: Unable to move into base location"
exit 1
fi
echo " "
echo "DONE"
echo " "

echo "**** DO HG CLONE ****"
echo " "
hg clone $HGRepoURL $HGRepoName
if [ $? -ne 0 ]; then
echo "FATAL: Unable to clone mercurial repository"
exit 1
fi
echo " "
echo "DONE"
echo " "

echo "**** MAKE GIT REPOSITORY ****"
echo " "
mkdir $GITRepoName
if [ $? -ne 0 ]; then
echo "FATAL: Unable to create git repositroy directory"
exit 1
fi
cd $GITRepoName
if [ $? -ne 0 ]; then
echo "FATAL: Unable to move into git repository directory"
exit 1
fi
git init
if [ $? -ne 0 ]; then
echo "FATAL: Unable to initialise git repository"
exit 1
fi
echo " "
echo "DONE"
echo " "

echo "**** DO CONVERSION CLONE ****"
echo " "
../../hg-fast-export.sh -r ../$HGRepoName
if [ $? -ne 0 ]; then
echo "FATAL: Unable to do conversion"
exit 1
fi
echo " "
echo "DONE"
echo " "
Be the first to comment
del.icio.us this! Digg this! RSS 2.0 TOP

MSBuild Condition Syntax

By brian | Filed in Uncategorized

I’m back doing some MSBuild work. Its idiosyncratic syntax always takes a bit to get just right.

Conditions
Some useful examples

Condition Description
‘stringa’==’$(var)’ you can also use !=, <, >, <=, >=
(note you must escape &lt;> with &lt; and &gt;
!, And, Or can all be used as Boolean logic operators and grouping functions
!, And, Or can all be used as Boolean logic operators and grouping functions
Exists(‘$(var)’) Checks if the specified file or folder exists
HasTrailingSlash(‘string’) returns true if the string ends in either forward or backward slash
Be the first to comment
del.icio.us this! Digg this! RSS 2.0 TOP

Git Command Reference

By brian | Filed in Uncategorized

I’m in the process of implemeting Git at my current employer (NAB Wholesale). Its a great win for the company that the internal communicty based standards aproach has been aloud to create a standard like this defining a tool like source control be changed from IBM ClearCase (priopriatory and awful) to Git (free, open source and awsome).

Originaly I was targeting Mercurial as the replacement, but in evaluation, Git seemed like the better choice overall. (Pro’s and Con’s exist for both from my perspective)

Thiese are some command I’ve started putting togeather as a cheat sheet

To show remote branches, and see if they are tracked, and see what the default pull/push operations are

git remote show origin

To change the default branch of a repository (particularly a bare repository)

git symbolic-ref HEAD refs/heads/[branchname]

Afterwhich you delete the local branch.

To checkout and track a remote branch

git checkout -t [remotename]/[branchname]

To delete a branch

git branch -d [branchname]

Note, this wont let you delete a branch that is not merged into your current branch. To do that, you change the command to

git branch -D [branchname]

Careful though, this allows you to make commited code unreachable (except via ref log)

To delete a remote branch

git push origin :[remotebranchname]

To prune stale remote tracked branches

git remote prune [remotename]

My repository is running slow.

This could be caused by too many objects and loose objects in your repository. Run the garbage collect command to optimise and free up space.

git gc

 

Be the first to comment
Tags: ,
del.icio.us this! Digg this! RSS 2.0 TOP

For Linux, this little application http://www.pendrivelinux.com/universal-usb-installer-easy-as-1-2-3/ makes it really easy to create a USB drive that will boot and allow you to install Linux.

For Windows 7, you don’t even need that.

Open a command prompt (type commands that start with > on the dos prompt)

> diskpart
> list disk

You need to select the correct disk as this will erase the disk. 3 in the next command corresponds to the disk number to select.

> select disk 3
> clean
> create partition primary
> select partition 1
> active
> format fs=fat32
> assign
> exit

Then you need to extract the files from the Windows ISO and copy them onto the USB disk. The easy way is to mount the ISO using something like Virtual Clone Drive. Then copy them using XCopy or Robocopy.

Assuming d:\ is your windows disk or mounted ISO image, and a:\ is your previously prepared USB disk.

> xcopy d:\* /s /e /f a:\

Done.

Be the first to comment
del.icio.us this! Digg this! RSS 2.0 TOP

I have recently decided to start using Git as well as Mercurial for a few reasons.

  • The short lived branching implementation is more mature and better implemented with the tools
  • Merge drivers
  • Widows support with msysGit for windows works brilliantly
  • Git Extensions is really good, and being one app is better than the shell extension tortoise stuff

Firstly obtain the fast export code using this command in a directory you want to work in.

git clone git://repo.or.cz/fast-export.git fast-export

Then run these commands to convert the HG repository to Git, replacing the items in square brackets with real values

mkdir [projectname]
cd [projectname]
git init
[../path/to/fast-export]/hg-fast-export.sh -r [/path/to/project.hg.directory]
git checkout master

Note, that if you have more heads than the tip, they need to be named. It will give you an error if you dont name them. It is easiest to merge in HG to one head before converting to Git.

Be the first to comment
Tags: , ,
del.icio.us this! Digg this! RSS 2.0 TOP

I like the way Linux works on the command line, you can do so much more than you can do on the windows command line, you can do it easily, and you have been able to do it for a long time. Powershell might be a good up and comer for windows, but cygwin makes it easy for people tied to windows to get some of the linux/unix goodies, I specifically like ssh, and rsync as a combination to do backups over the cloud.

I know that these days there are a hundred and one online businesses whom will backup all your data for you for very little money. However I like to actually know what’s going on when it comes to my data like pictures of my kids. Having grown up digital, I have a lot of data these days, and on various computers around the place.

SSH and RSYNC are a very powerful combination too, you use SSH to connect securely, and then you use RSYNC to backup files that have changed since your last backup. The best bit is you can make it link to a previous directory using hard links, so you can have a backup directory that contains the state of your files every time you have done a backup, but as it uses hard links it does not use excessive disk space, only new files get added, old files that have not changed simply get linked to yesterdays backup, which may be a link to the day before backup and so on.

You can also safely delete backup directories if they are too old, as the files that are linked to in more recent backups won’t get deleted, only the links from the backup you are deleting will be removed.

Some of the Cywin packages I usually use are:

  • Editors > Nano
  • Net > OpenSSH
  • Net > rsync
  • Net > tcp_wrappers

Rsync Options

  • -r recurse into directories
  • -t preserve modification times
  • -D preserve device files and special files
  • -v verbose output
  • -x dont cross file system boundaries
  • -z compress data for transfer
  • -partial to keep partially transferred files
  • –delete to delete extra files
  • –link-dest=[path] to link to files in the previous backup, rather than copying again
  • –exclude-from=[filelist] a file to exclude files from copying
  • –chmod=ugo=rwX to change the permissions of files copied

To connect using SSH utility within Cygwin, you need to use this kind of command


&gt; ssh brian@localhost -p 4022 -i /home/username/.ssh/identitys

This allows you to then run commands on the remote machine, and connect using key based authentication (which means no password required if you are scripting it, although this in itself is a security hole).

This is an example script I put together which keeps the last 400 backups, and keeps rolling, so the most recent backup is 0, and the oldest is 400. This allows me to go back in time and get deleted files, or restore an old database backup.

#!/bin/bash

# variables
BackupFromLocation=username@serverneedingbackup.com:/cygdrive/c/mydatadirectory
BackupToLocation=/cygdrive/e/backups
IdentityFile=/home/username/.ssh/id_rsa
Port=4022
BackupLastEvent=400
BackupFirstEvent=0
# Output settings applied to this script
echo "Bash version ${BASH_VERSION}..."
echo "Backup Script --- Environment variables --- "
echo "Backup To Location : $BackupToLocation"
echo "Backup From Locat  : $BackupFromLocation"
echo "SSH Port Number    : $Port"
echo "Identity File      : $IdentityFile"
echo "Backup First Event : $BackupFirstEvent"
echo "Backup Last Event  : $BackupLastEvent"
echo "Email Log To       : $EmailLogTo"
echo ""

echo "Creating backup directories if they dont exist..."
if [ -d $BackupToLocation ]; then
echo "Backup directory $BackupToLocation exists"
else
echo "Backup directory $BackupToLocation creating"
mkdir $BackupToLocation
fi
for (( dir=$BackupFirstEvent+1; dir&lt;=$BackupLastEvent; dir++ ))
do
if [ -d $BackupToLocation/$dir ]; then
echo "Backup directory $BackupToLocation/$dir exists"
else
echo "Backup directory $BackupToLocation/$dir creating"
mkdir $BackupToLocation/$dir
fi
done
echo "done"

if [ -d $BackupToLocation/$BackupFirstEvent ]; then
echo "Backup Directory [$BackupToLocation/$BackupFirstEvent] exists, rolling backups back"
echo "Removing oldest ($BackupLastEvent) backup snapshot..."
#exit
rm -r $BackupToLocation/$BackupLastEvent
echo "done"

echo "Rolling histories one snapshot back..."
let MoveTo=$BackupLastEvent-1
for ((MoveFrom=$MoveTo; MoveFrom&gt;=$BackupFirstEvent; MoveFrom-- ))
do
let MoveTo=$MoveFrom+1
echo "Moving $MoveFrom to $MoveTo..."
mv $BackupToLocation/$MoveFrom $BackupToLocation/$MoveTo
done
echo "done"
else
echo "Backup to directory [$BackupToLocation/$BackupFirstEvent] does not exist, previous run may not have completed"
fi

The only problem with this script is files that are in use are locked and can’t be accessed. You could use the vshadow service to create a readable file system, but in my case I don’t have the need. It might be a future improvement on the script to make sure scripted backups are more fool proof though.

Feel free to use, change and redistribute the script, but I take no accountability for weather it works in your scenario. All backups should be tested thoroughly and regularly.

Be the first to comment
del.icio.us this! Digg this! RSS 2.0 TOP

This site is awsome http://repositoryhosting.com/. You can host all your projects here in either Mercurial, Git or SubVersion repositories for the low cost of $6 per month, plus $1 per GB of hosting space above the included 2GB. Thats unlimited repositories, and unlimited users.

Not only that it has many add ons that can be used, like Trac http://trac.edgewall.org/ wiki and ticket tracking. As well as a bunch of trac extensions.

You can have a custom URL’s, you can have change set notifications emailed, posted to twitter, or campfire. You can allow anonymous users readonly access, as well as creating unlimited named users to access your repositories.

You can use it as a WebDAV drive which is mappable on windows, linux and mac for files that are should remain outside your code repositories.

Best of all you can backup the whole thing by daily, weekly or monthly which allows you to download a compressed file of all your data, or you can have RepositoryHosting push it up to your Amazon S3 account automatically.

It is the most cost effective way to do private repository hosting arround and has a fair bit of flexibility too.

Be the first to comment
del.icio.us this! Digg this! RSS 2.0 TOP