We’ve just had some work done on one of our systems (which runs from a Raspberry Pi v2) so, aside from the config file backups that we can collect from the systems interface, I thought it would be prudent to grab an image backup.. this is how it was done: Continue reading “Backing up the Pi”
I’m finding more and more need for a large amount of ‘always on’ storage to be available on my home network and, the recent release of Debian 3.1 (Sarge) has enticed me to repurpose a box I had around the house and get it up and running…
Debian GNU/Linux 3.1 (a.k.a. sarge) was released on 6th of June, 2005. The new release includes a bunch of major changes to it’s predecessor Debian GNU/Linux 3.0 (a.k.a. woody). You can check out the press release, review the release notes, or simply click yourself off to the Debian website to read more about it.
Now, while an official installation manual exists, and Google will help you find thousands of people who may be able to answer more technical questions, this page (or pages – depending on how much I end up caring about the documentation) will record my installation experience on my soon to be main server for the house. It’s mainly for my future reference, but you may find it useful too! I have based the instructions on the experience after booting to a CD burnt with the debian-31r0a-i386-netinst.iso file.
- The Hardware of this Server
- Initial boot from the Network Installation CD
- First boot after initial CD Configuration
- Setting a Static IP address
- Creating a Spanned Disk using LVM
- Backing up your Installation
- Creating a Network Share with SAMBA
- Monitoring the box
- Hard Disk Drives
This section is a little unnecessary as the Debian 3.1 install is actually pretty good at finding everything, but – for the sake of completeness – my hardware configuration for this server is as follows:
|Case:||Enermax FS-710B||Yes it’s big, but the case was unutilised from a previous MythTV backend server project. This machine will also be running a number of hard drives and cooling is key 🙂|
|PSU:||Enermax 550W EG651P-VE||Again
big – 550W (peak 650W) is more than the average system needs, but these
are quiet units and I’m expecting to have a large amount of disk in
this unit at the end of all this. – [Review]
(VIA KT333 Chipset)
|It’s an old board and monitoring the environmental sensors on it doesn’t appear to be too easy – if you have this board AND can direct me to an easy to follow howto to get monitoring working under Debian Sarge, please drop and email to rob [dot] inskeep [at] gmail [dot] com (with the obvious changes to unmung my address :))
I’ve found this link around running Debian on this board – I’ll need to have a bit more of a play and see if it can clarify any of the issues I’ve been having…
|CPU:||AMD Athlon 2100+|
|RAM:||Generic 512MB PC2700 DDR|
|Hard Drives:||40GB Maxtor
120GB Western Digital
|Apollo UATA133 2MB Cache (6E040L0)
|Optical:||LG DVD ROM||It may seem like overkill to put a DVD ROM into a server and I’d agree – but as the distros get bigger it’s making more sense – it also allows me to convince myself (and the accountant wife) that a DVD/RW + CD R/W drive would be a good investment for one of the LAN PCs|
|Video Card:||Power Color CM64P||I found this card in my box of bits that all us IT types tend to accumulate, I can’t find a manual or anything useful to describe the card so – google may be your best bet here.|
|Sound Card:||Onboard (C-Media CM8738)||Not planning to use this|
|NIC:||PCI 3Com 3C905B||Will probably add/replace with a Gig card at some stage|
1.0 Choose Language
2.0 Choose Region
3.0 Choose Keymap
4.0 Detection Phase
5.0 Configure the Network
5.1 Enter Hostname
[Enter a name for your machine]
5.2 Domain Name
6.0 Partition Disks
I have 3 Disks in this system:
40GB Maxtor (6E040L0)
120 GB Western Digital (WD1200BB-00DW)
200GB Seagate (ST3200822A)
Since there is nothing I want on the box I’m happy to have the install wipe out everything.
6.1 Partitioning Method
6.1.1 Erase entire disk: IDE1 Master (hda)
6.1.2 All files in one partition
This created 2 partitions,
#1Â Â Â Â primaryÂ Â Â Â 39.5GBÂ Â Â Â ext3Â Â Â Â /
#5Â Â Â Â logicalÂ Â Â Â Â Â 1.6GBÂ Â Â Â Â swapÂ Â Â swap
I don’t plan to setup the other 2 disks yet as I want to span them into a very large drive which I can add more space to later – that will take a bit of learning I imagine so for now we just:
6.2 Finish partitioning and write changes to disk
Answer yes to confirm writing the partition tables.
7.0 Write partitions / Installing Debian base system
8.0 Install GRUB boot loader
8.1 Install GRUB as default boot loader
9.0 Completing Installation
9.1 Remove the CD from the drive
10.0 Debian Configuration
11.0 Time Zone Configuration
11.1 Clock is NOT set to GMT
11.3 Root password
11.4 Users Full Name
11.6 User Password
12.0 APT configuration
12.1 Chose edit by hand
Enter the sources as listed further down in this section – some may not resolve at this point so commenting them out with a # at the start of the line will allow you to continue through the installation.
13.0 Software Selection
Note: I would not choose to install applications during the initial setup as, until this guide is complete and I can understand everything which is happening, I want to be able to install all the packages I need via the apt-get method. To do this, it pays to use local sources where it makes sense see below for my sources list which can be copy/pasted into
13.1 Unpacks a bunch of files
13.2 Configuring Exim v4 (exim4-config)
13.2.1 Local delivery only; not on a network
13.2.2 Plan to configure things later to grab mail from various POP sources and deliver locally to allow for spam/virus blocking at the network level.
13.2.3 Root & postmaster mail recipient
188.8.131.52 [default user]
13.3 Debian configuration complete
13.5.1 Note that we can run this again via base-config program.
13.4 Tries to start X & fails
13.6.1 Disables X Server until X can be configured correctly
13.5 Login Prompt
That’s it – the system is setup, now obviously there is some tidy up to do – but at least the system is running. Now, while we still have a local keyboard and monitor plugged in, we need to set the system to use a static IP address so we can find it later once it’s running headless (without a local keyboard/mouse/display).
You may find it easier to finish off via a SSH connection, first off we need to know the IP address that DHCP has assigned to this machine.
You are interested in the value of inet addr: for the eth0 card as this is the IP for the machine (and you’ll need to use it to connect your SSH client remotely.
Once you’re remotely connected, you can carry on to the next step, or go it alone and install any packages you so desire. The only package I needed at this point was vim, a text editor.
- As the root/su user, do an
apt-get updateand an
apt-get upgradefirst, to ensure you have the latest packages and package list.
- Use the command
apt-get install vimto install the vim text editor
- Edit your /etc/apt/sources.list to restore the lines we commented out during the installation.
Your file should look something like this:
deb ftp://ftp.jetstreamgames.co.nz/debian stable main
deb ftp://debian.ihug.co.nz/debian/ stable main
deb http://http.us.debian.org/debian stable main contrib non-free
# The non-US source listed below is not loving me at the moment – I need to find
# another non-US apt source.
#deb http://non-us.debian.org/debian-non-US stable/non-US main contrib non-free
deb http://security.debian.org stable/updates main contrib non-free
- Do another
apt-get updateto test your sources and ensure you have the latest package lists.
- Once all your sources are working, do a
apt-get upgradeto update your system with the latest packages
iface eth0 inet static
address [ a_valid_IP_address ] (e.g. 10.1.1.xxx)
netmask [ your_netmask ] (e.g. 255.255.255.xxx)
network [ your_network ] (e.g. 10.1.1.0)
broadcast [ your_network ] (e.g. 10.1.1.0)
gateway [ your_gateway ] (e.g. 10.1.1.xxx)
and save/exit the file (<esc> :wq <enter>)
To apply this configuration type:
or reboot the system with
shutdown -r now
Creating a Spanned Disk using LVM
So, the theroy is like this: I have a couple of largish hard drives (okay, as large as I could afford – and had lying around) and had previously spanned these within Windows XP to store a buttload of files and stuff. Since this server we’re building is going to be up all (most?) of the the time, it goes to reason that the large disk should sit here rather than on a windows box which goes up and down like a whitehouse intern.
On with the section…
- Start with unpartitioned disks
- Install LVM packages
apt-get install lvm-common lvm2
- Initialize the disk(s) (or partition(s)) for use by LVM
pvcreate /dev/[ device_name ]
pvcreate -ff /dev/hdc
pvcreate -ff /dev/hdd
The -ff option forces an initialisation
- Create a Volume Group
vgcreate [ volume_group_name ] [ device_to_include ] [ device_to_include ]
vgcreate volume_group /dev/hdc /dev/hdd
- Create a Logical Volume
lvcreate -l [ the_size ] [ y o u r _ v o l u m e _ g r o u p ]
lvcreate -l 76313 volume_group
To determine [ the_size ] you can run the command
vgdisplay [ volume_group_name ]and take the ‘Total PE’ value (or fraction of). See below for an example of this on my system.
- Make the File System
mkfs -t ext3 /dev/[ your_volume_group ]/[ y o u r _ v o l u m e _ g r o u p ]
mkfs -t ext3 /dev/volume_group/LargeDiskVolume
- Mount the Drive
mount /dev/[ Volume_Group_Name ]/[ Logical_Volume_Name ] /mnt
mount /dev/volume_group/LargeDiskVolume /mnt
- Check the Mount
Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda1 xxxxxx xxxxxx xxxxxx xx% / tmpfs xxxxxx xxxxxx xxxxxx xx% /dev/shm /dev/mapper/large_disk_volume-LargeDiskVolume 307673208 32828 292011480 1% /mnt
- Set the mount to load at boot
Add the line:
/dev/disk_span/lvol0Â /mntÂ Â Â Â Â Â Â ext3Â Â Â Â defaultsÂ Â Â Â Â Â Â Â 0Â Â Â Â Â Â Â 0
Save the file
Right – the mount appears, it’s all done! The next step is to install Samba and create a share that windows machines can see and write to.
Displaying the Volume Group Information
|— Volume group —|
|Metadata Sequence No||2|
|VG Size||298.10 GB|
|PE Size||4.00 MB|
|Alloc PE / Size||76313 / 298.10 GB|
|Free PE / Size||0 / 0|
Displaying the Logical Volume Information
|— Logical volume —|
|LV Write Access||read/write|
|LV Size||298.10 GB|
|Read ahead sectors||0|
Reference: LVM HowTo
If you’re following through this in the order that it’s written, it’d probably a good time to take a snapshot of your raw system before we go installing anything (clever) which might cause problems further down the track. This is a good idea as it give you a clean build of your system after all the initial setup has completed – and saves the setup bandwidth should you need to start again 🙂
dd as the tool to create the backup as it’s reletivly well documented so googling will pretty much sort out most issues you’re likely to run into (very important when you’re talking about your digital assets!). The ddcommand will do a bit for bit backup of a Hard Disk. I’m using this to give myself an image that I can always come back to as I try changing and installing new ‘stuff’ on my system (given my tendency to haphazardly experiment as I go).
- Create a directory to store the backup
mkdir /[device to store the backup on]/[backup directory name]
e.g. mkdir /mnt/backup
- Copy the HDD to an image file
dd if=/dev/[hard drive name] of=/mnt/[backup directory name]/[backup name].iso bs=65535
e.g. # dd if=/dev/hda of=/mnt/backup/mydisk.iso bs=65535
BE ROOT FOR THIS! Actually I don’t know for sure, but it stands to reason that if you don’t have the access of root, then you won’t be able to back up files owned by root.
- Be patient. THIS WILL TAKE QUITE AWHILE (You are after all copying an entire disk) when it completes, you will see something similar to the following (just with numbers not xxxs 😉 )xxxxxx+1 records in
xxxxxx+1 records out
xxxxxxxxxxx bytes transferred in xxx.xxxxxxx seconds (xxxxxxxx bytes/sec)
- Compress the Image
If you have a large amount of free space on the disk you were copying, compressing it will shrink it’s size.
gzip /[device to store the backup on]/[backup directory name]/[backup name].iso
e.g. gzip =/mnt/backup/mydisk.iso
Again, this will take a fair while to complete so I’d recommend taking the dd and the gzip command lines and pasting them to the command line together so your system runs them one after the other – you could of course automate this with a script and run it via cron too. As an indication, I was able to turn my 41,110,142,976 byte .iso into a 32,757,683,474 byte .gz file – that’s about a 20% space saving!
Getting it back (Restore [for the benefit of searchbots]):
- From some kind of command line (you could use a Knoppix CD or similar CD based distro) you need to run:
dd if=/[device the backup is stored on]/[backup directory name]/[backup name].iso
e.g. dd if=/mnt/restore/mydisk.iso of=/dev/hda bs=65535
shutdown -r now
Note – I haven’t tried a restore at the time of writing this, so I’d recommend testing your backup before you need it.
Creating a Network Share with Samba
So, by this point you should have a machine setup on your network and you may even have a spanned disk with a heap of space which you can make available to your other network machines.
I’m going to use Samba to create the shared drive and control its access so that my Windows machines can also make use of the space.
- Install the Samba packages
apt-get install samba smbfs smbclient
How do you want to run Samba: [daemons]
Create samba password database, /var/lib/samba/passdb.tdb? [Yes]
- Add a Linux user for linking to the Samba share
Enter a username to add: [Share Username]
Adding user ‘[Share Username]‘…
Adding new group ‘[Share Username]‘ (1001).
Adding new user ‘[Share Username]‘ (1001) with group ‘[Share Username]‘.
Creating home directory ‘/home/[Share Username]‘
Copying files from `/etc/skel’
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for [Share Username]
Enter the new value, or press ENTER for the default
Full Name : [Share Username]
Room Number :
Work Phone :
Home Phone :
Is the information correct? [y/N] [y]
- Add users to the Samba server
smbpasswd -a [Share Username]
New SMB password:
Retype new SMB password:
Added user [Share Username]
- Create a directory to share
I’m assuming that your LVM is already mounted, if not check the LVM section for the correct syntax
mkdir /mnt/[Share Directory Name]
e.g. mkdir /mnt/MediaStore
- Allow others to reand and write to the directory
chmod 777 /mnt/[Share Directory Name]
e.g. chmod 777 /mnt/MediaStore
At this point, you need to decide if you’re GUI or CLI, for me a Graphical User Interface is quicker and suits my impatient nature quite nicely, others may wish to do all their config from the Command Line Interface – either choice is fine, but for the sake of swift sharing, and given that this server will sit on the inside of my network, I’ve chosen to use a package called SWAT (Samba Web Administration Tool) to help me setup my Samba shares
- Install the SWAT package
apt-get install swat
As the package configures, you will receive a warning that your smb.conf file is about to get overwritten. Choose [Ok]
- Enable SWAT by allowing the service
Find and uncomment the line:
#<off>#Â swatÂ Â Â Â Â Â Â Â Â Â Â Â streamÂ Â tcpÂ Â Â Â Â nowait.400Â Â Â Â Â Â rootÂ Â Â Â /usr/sbin/tcpdÂ Â /usr/sbin/swat
The line should now read:
swatÂ Â Â Â Â Â Â Â Â Â Â Â streamÂ Â tcpÂ Â Â Â Â nowait.400Â Â Â Â Â Â rootÂ Â Â Â /usr/sbin/tcpdÂ Â /usr/sbin/swat
Save the file
- Enable SWAT by allowing the port
Add the line:
Save the file
- Reboot the server (or figure out how to reload the affected configuration files – and let me know)
To REBOOT use
shutdown -r now
To RELOAD without a full reboot, use
Right, now you should be able to point a browser on your LAN to
http://[Server Hostname or IP Address]:901/
You should be presented with an authentication dialogue (which you’ll need to use your root credentials for – don’t worry, you can setup other users once you’re in)
- Click the shares link
- Create a Share by entering your [Sharename] and clicking the ‘Create Share’ button
- In the ‘Base Options’ section, enter the path that you created above ‘/mnt/[Share Directory Name]’
- In the ‘Security Options’ section, add the users you want to have access to the relevant list (valid users, read list, write list) e.g. I will have a user who can read and write from my windows machines, but the media devices which also connect to this share will be write only. In the ‘hosts allow’ line, you may want to add you whole network with
or just specific IP addresses
- Click ‘Commit Changes’ when done setting options
Reference:There is a pretty good article over on Linux Questions on SAMBA, click here to view
This section covers a number of aspects of monitoring which I feel are necessary, but then – I do have a soft spot for numbers and graphs!
This is a great tool used to graph pretty much anything, seriously! One of the features I like the best about it is how the databases it creates never get any bigger than they are when you create them. I’ve decided to store all my databases in /var/lib/rrd/ since a number of others out there seem to use this location and it just makes using other peoples scripts a lot easier.
- Install the RRDTool package
apt-get update && apt-get install rrdtool
Read the documentation on the web for how it all works – it’s explained better there than I can do in this guide.
SmartMon tools is a package which allows you to interrogate your SMART enabled hard drives for problems. There is a pretty good article over at the Linux Journal here which details the what’s and how’s of the package.
To monitor the hard drive(s) on this system I followed these steps:
- Edit /etc/apt/sources.list to add the location of the SmartMon tools package
- Add the lines:
# Location of SmartMon Monitoring Package
deb http://honk.physik.uni-konstanz.de/~agx/linux-i386/debian smartmontools/
- Update your sources and install the package
apt-get update && apt-get install smartmontools
- smartd was disabled and to enable I needed to edit /etc/default/smartmontools
- Uncomment the smartd line
- Start smartd
- Check your disks
/usr/sbin/smartctl -a [disk name]
e.g. /usr/sbin/smartctlsmartctl -a /dev/had
I got output which looks like this:
smartctl version 5.32 Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION === Device Model: Maxtor 6E040L0 Serial Number: xxxxxxxx Firmware Version: xxxxxxxx Device is: In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: ATA/ATAPI-7 T13 1532D revision 0 Local Time is: xxx xxx xx xx:xx:xx xxxx xxxx SMART support is: Available – device has SMART capability. SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
Total time to complete Offline
data collection: (1021) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
Offline surface scan supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
No General Purpose Logging support.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 17) minutes.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
3 Spin_Up_Time 0x0027 220 219 063 Pre-fail Always – 8211
4 Start_Stop_Count 0x0032 253 253 000 Old_age Always – 1110
5 Reallocated_Sector_Ct 0x0033 253 253 063 Pre-fail Always – 0
6 Read_Channel_Margin 0x0001 253 253 100 Pre-fail Offline – 0
7 Seek_Error_Rate 0x000a 252 252 000 Old_age Always – 1
8 Seek_Time_Performance 0x0027 250 234 187 Pre-fail Always – 64735
9 Power_On_Minutes 0x0032 232 232 000 Old_age Always – 983h+23m
10 Spin_Retry_Count 0x002b 253 252 157 Pre-fail Always – 0
11 Calibration_Retry_Count 0x002b 253 252 223 Pre-fail Always – 0
12 Power_Cycle_Count 0x0032 252 252 000 Old_age Always – 403
192 Power-Off_Retract_Count 0x0032 253 253 000 Old_age Always – 359
193 Load_Cycle_Count 0x0032 253 253 000 Old_age Always – 1698
194 Temperature_Celsius 0x0032 253 253 000 Old_age Always – 37
195 Hardware_ECC_Recovered 0x000a 253 252 000 Old_age Always – 1136
196 Reallocated_Event_Count 0x0008 253 253 000 Old_age Offline – 0
197 Current_Pending_Sector 0x0008 253 253 000 Old_age Offline – 0
198 Offline_Uncorrectable 0x0008 253 253 000 Old_age Offline – 0
199 UDMA_CRC_Error_Count 0x0008 199 199 000 Old_age Offline – 0
200 Multi_Zone_Error_Rate 0x000a 253 252 000 Old_age Always – 0
201 Soft_Read_Error_Rate 0x000a 253 252 000 Old_age Always – 3
202 TA_Increase_Count 0x000a 253 252 000 Old_age Always – 0
203 Run_Out_Cancel 0x000b 253 252 180 Pre-fail Always – 0
204 Shock_Count_Write_Opern 0x000a 253 252 000 Old_age Always – 0
205 Shock_Rate_Write_Opern 0x000a 253 252 000 Old_age Always – 0
207 Spin_High_Current 0x002a 253 252 000 Old_age Always – 0
208 Spin_Buzz 0x002a 253 252 000 Old_age Always – 0
209 Offline_Seek_Performnce 0x0024 184 184 000 Old_age Offline – 0
99 Unknown_Attribute 0x0004 253 253 000 Old_age Offline – 0
100 Unknown_Attribute 0x0004 253 253 000 Old_age Offline – 0
101 Unknown_Attribute 0x0004 253 253 000 Old_age Offline – 0
SMART Error Log Version: 1
ATA Error Count: 1
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It “wraps” after 49.710 days.
Error 1 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours)
When the command that caused the error occurred, the device was in an unknown state.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
— — — — — — —
04 51 50 40 97 03 00 Error: ABRT
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
— — — — — — — — —————- ——————–
ef fe 00 00 00 00 00 00 00:03:20.976 SET FEATURES [Reserved for CFA]
ec 00 01 01 00 00 00 00 00:03:20.944 IDENTIFY DEVICE
c1 00 00 00 00 00 00 00 00:03:20.944 [VENDOR SPECIFIC]
c0 00 01 01 ff ff 00 00 00:03:20.928 CFA ERASE SECTORS [VS IF NO CFA]
c1 00 ff 01 ff ff 00 00 00:03:20.928 [VENDOR SPECIFIC]
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 4348 –
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Loads of info huh? You can obviously pull out what is of interest to you from here.