Arkeia

Arkeia/Backups

Arkeia is the software currently being used to control all aspects of tape backups/restorations/archives. The web interface for Arkeia is: (see google docs) Arkeia also has a command line interface, which is described after how to use the web interface section.

How to Install an Arkeia Client

Reference: http://wiki.arkeia.com/index.php/WD_Arkeia_Version_10.1_Adding_Linux_Clients_Guide

First scp the rpm file to the server. I stored both 64 and 32 bit versions in /nfs/home/tdemers/arkeia/

Then rpm -Uvh arkeia...rpm

Then edit the following files like this (say dalet):

Add this to the bottom of the /opt/arkeia/arkeiad/arkeiad.cfg file:

VIRTUAL_HOSTNAME "dalet.cluster.ucsf.bkslab.org"
NLP_FORCE_IP "10.20.0.22"        ⇒ This is the ip address of dalet
NLP_GET_PEERNAME "0"

~ Then make sure the file /opt/arkeia/arkeiad/admin.cfg only has this line:

tav.ucsf.bkslab.org:617

On the backup server first: service arkeia restart

Then on the client (dalet): service arkeia restart

You also have to add INPUT and OUTPUT lines in iptables on the client for port 617:

-A INPUT -m state --state NEW -m tcp -p tcp -s 10.20.0.10 --dport 617 -j ACCEPT
-A OUTPUT -m state --state NEW -m tcp -p tcp -s 10.20.0.10 --dport 617 -j ACCEPT

The client should be added okay now in the webui

Important Note on Backup Versions

The server you are moving to and from must be of the same revision of Arkeia.
Please DO NOT attempt a move from a different version of Arkeia such as v10.0 to v10.2.

If an upgrade of Arkeia is required, first move the configuration to the same version of Arkeia on the new server. Ensure things are stable for a few days, then upgrade the new server as needed

When moving an Arkeia DiskStorage, the DiskStorage path must be the same location on the new server as it was on the old server. Using an alternate path requires special steps not covered in this article.

The new Arkeia backup server version should be backwards compatible with your older client versions. So there is no need to upgrade every client to the latest version. However, every client must be added to the new backup server.

How to set up a brand new backup

First determine what files you want to backup and the size of the directories.
I’ve found that the following command works the best to determine the directory size.

du -ch filename | grep total 

Backup Procedure

1) Shutdown all Arkeia processes on the old server

/opt/arkeia/bin/arkboot stop
ps -ef | grep ark
kill PIDs

2) Create a tarball of /opt/arkeia

cd /opt
tar -zcvf arkeia.tar.tz arkeia

3) ftp or scp the "arkeia.tar.tz" file to the new server

scp arkeia.tar.tz root@newserver.domain.com:/opt/

4) Unpackage "arkeia.tar.tz" to /opt/arkeia

cd /opt
tar zxvf arkeia.tar.gz

5) Download "arkeia master" and the "arkwui" package from www.arkeia.com/download

6) Install the packages on the new server using rpm, dpkg or ./install

rpm -Uvh arkeia master package
rpm -Uvh arkwui

OR

dpkg -i arkeia master package
dpkg -i arkwui

7) If the hostname has changed on the new backup server, then issue the following command to rename everything in Arkeia:

/opt/arkeia/bin/arkmvhost  --host oldserver.domain.com --newhost newserver.domain.com

8) Stop Arkeia

/opt/arkeia/bin/arkboot stop

9) Change the hostname in the admin.cfg file to the FQDN of the new server

vi /opt/arkeia/arkeiad/admin.cfg
newserver.domain.com

10) If you are migrating an Arkeia Disk Storage, the data must also be moved to the new server.

NOTE: It is important that the DiskStorage is the exact same directory path of the old server.

EXAMPLE: If the old server has the DiskStorage in /vol0/vtl0 then the new server must have the DiskStorage located in /vol/vtl0.

11) Edit the usr.lst file to reflect the new hostname for all users

vi /opt/arkeia/server/dbase/f3sec/usr.lst
"NODE"  "newserver.domain.com"

12) Start Arkeia

/opt/arkeia/bin/arkboot start

13) Check the Library and Tape Drive paths to and update the settings using the WebUI/

IMPORTANT NOTE:

• You will need to stop the Arkeia Library in the WebUI to adjust the settings

Arkeia Detection

/opt/arkeia/bin/arkdetect

Tape Library in dmesg

dmesg | grep "type 8"

Tape Drive(s) in dmesg

dmesg | grep tape

13) Start the Arkeia Tape Library and verify the slot inventory is accurate.

First determine what files you want to backup and the size of the directories. I’ve found that the following command works the best to determine the directory size.

du -ch filename | grep total

Tapes

To see a list of all tapes

arkc -tape -list

or save it in a file tape_list:

[root@tav ~]# for TAPE_NAME in `(arkc -tape -list | cut -d'=' -f2)`;do `echo $TAPE_NAME >> /root/tape_list`; done

To add tapes

1) Select Where to backup -> backup to tape -> TapeStorage -> Tapes
2) Click 'green circular plus sign' to add a tape. Make sure you have the barcode of the tape you want to add.
3) Enter information regarding tape

 Tape Name: (typically tape name followed by three digit number)
 Tape Type: LTO Ultrium 800GB (this is the tape type we have in BKS) 
 Comment: (information up to your preference)
 Click the 'Show more options' checkbox so you can enter the tape barcode.  
 Put in the barcode of the tape.  Tape auto loader will identify tapes by the barcode on their labels so this is important.  
 Our tapes are mostly LTO4 so the barcode must be suffixed by L4.  
 Ex: if my tape barcode is 001029, I will put 001029L4 as the barcode.

If the tape is in the autoloader when the tape is created, the you will have to initiate a rescan of the tapes. You can do this by ejecting the tapes from the autoloader and sliding the tape magazines back in.

Alternatively, you can access the Backup -> Where to Backup -> Tape Storage -> Tape Libraries; Click Actions on drop down menu -> Manage Drives and Slots; Manage Slots; Then you can select the Unknown Tape by barcode number, select a tape name from 'Compatible Tapes' and click the leftward facing arrow so the name will get ported to the unknown tape.

How to see which device is the tape library and drives

dmesg | grep Attached

Attached scsi tape st0 at scsi6, channel 0, id 3, lun0 <---- TAPE DRIVE on st0 Attached scsi tape st1 at scsi6, channel 0, id 4, lun0 <---- TAPE DRIVE on st1 Attached scsi generic sg1 at scsi6, channel 0, id 3, lun 1, type 8 <---- TAPE LIBRARY on sg1

OR

Tape Library

type 8 devices are library robotics and control arms.

dmesg |  grep "type 8"

sg 6:0:1:0: Attached scsi generic sg1 type 8

To see the details of the tape library

/usr/sbin/mtx -f /dev/sg1 status /usr/sbin/mtx -f /dev/sg1 inventory

How to Reindex Tapes (Unknown Tape)

• Note: When you reindex a tape there cannot be any other arkeia jobs happening (ie no other backup jobs, restore jobs, etc).

Insert the tape you want to reindex into the tape library.

Log into https://tav.ucsf.bkslab.org:20618
login: admin
password: clust2pass(same as machine)

Go to Backup -> Where to backup -> TapeStorage -> Tape Libraries Then in the drop down menu under Actions select Manage Drives and Slots Then click the Move Tape tab Then on the right hand side select the tape you want to reindex and on the left select a Tape Drive (usually just Tape Drive 1) and then click the left directional button. (Take note of the slot number the tape is in). Once this is done go to Tools -> Tape reindex Unclick the Reindex a known tape box. Enter the Source drive, Pool to attach, Source tape type and the Tape slot num and click Start Reindex. (This step will identify the name of the tape and add this name to your Tapes collection. It is up to you to correctly allocate the tape name to the tape barcode in the Tape Libraries menu.) Then when it’s done go to tapes, select the slot on the left and the name of the tape on the right and then click the left arrow (you have to scroll down to see it).

I don’t know about the following: (if you don't encounter the reindex'd tape ID, then that's okay. Proceed to Tape Reindex.) Once this is done, ssh to tav and as root do the following: cat /opt/arkeia/server/dbase/f3tape/tpmaster.lst | more and look for the tape you just created/reindexed. Then vi /opt/arkeia/server/dbase/f3tape/oldcid.lst and delete the entire entry for that tape. There should be something like: ITEM { "CID" "43cf5073" } You would delete all three lines.

Then go back to the web ui and go to Tools-> Tape Reindex again and leave the Reindex a known tape box checked, enter in the Source tape and the Source drive and click Start Reindex. A recently reindex'd tape may not show up immediately (wait about five minutes). Once this is finished (it will probably take a while) you can then restore the files by going to Restore.

If arklib is not running when you start the tape library

On the server:

cd /opt/arkeia/bin
./arklib

How to Overwrite a Tape Label in Arkeia

First, if the tape that you’re overwriting already exists, delete it. Then move the tape (it should say unknown tape or something) to the tape drive. Pretend that the device name for the tape drive is /dev/st0 and there exists a file on the backup server /home/backup/testing

tar -cvf /dev/st0 /home/backup/testing
tar -tvf /dev/st0              #To check it successfully overwrote the label
Then move the tape from the drive back to the slot.
Then enter the “new” tape information.
Write the label to the tape (Tape Libraries -> Manage Slots and Drives -> Manage Slots)
Then when you use the tape again, it will write the new label.

To create a new savepack

In the Arkeia web UI, click on What to backup and then Savepacks
On the right hand side of the screen, click on add a new savepack.
Enter a savepack name
The compression type should be LZ1 or LZ3_3
Encryption type allowed should be no encryption
Filesystems allowed normal filesystems
Hit the create button

You will then be redirected to the list of savepacks.
With the new created savepack highlighted, click on List of Trees tab.
Then click on Manage trees for your_savepack on the righthand side.
Then click on Add a new tree on the righthand side.
If the server is nfsheadN and the directory you want to backup is /raidM/people, you would enter the pathname like this:
nfsheadN.bkslab.org!file:/raidM/people
Then at the bottom click Add tree

How to Restore data

1) Click 'Restore' on the left menu tab
2) Click 'Restore files, system info, databases, and applications'. Next.
3) Use the date sliders to customize the date you want the restores from. Usually, you will want to adjust this option because the tapes can contain data from ancient history and will try to restore irrelevant data from tapes we don't use.
4) Click on the server and it's respective filesystem that you want to restore. Then click the redirection tab.
5) Source: I always put the filesystem I want restored. Destination: I put the name of server and filesystem location I want the restored data to be written to. I always use this option because by default it will write back to the original server the data came from. Usually, I will write to an empty location on a server then copy the data back from that location.
6) Click 'Start Restore'

Licenses

Different Types of Licenses

THE DIFFERENT TYPES OF LICENSES:

Arkeia classifies clients using three Type categories. The following ONLY applies to clients/ backup server

Type 1

AIX, Solaris, True64, IRIX, HP-UX, SCO OpenServer / Unixware, Novell Netware

Type 2

Windows Servers (2000, 2003, 2008, 2012) Windows Server and Advanced MacOS X Server SLES RHAS RHEL Mandriva Corporate Server NOTE: Any Linux on IA64, PowerPC, x86_64 architecture

Type 3

Free Linux on x_86, (Gentoo, CentOS, Fedora, SuSE) Windows Workstations, Professional and Home Editions (XP, Vista) MacOS X Workstations , Free BSD, Open BSD, NetBSD

Our Licenses

Here are all licenses for Account UCSF_PHARMA
License customer:     UCSF_PHARMA
License name:     ARK102
License serial:     TE92R1ZFRHMDPN
License checksum:     M59D5SAWVYN8XA
Comment:     Arkeia 10.2
Creation date:     2014-05-13 20:46:56
License created by:     sbrown
Expire time:     Never
Type-1 agents:     0
Type-2 agents:     4
Type-3 agents:     8
Number flows:     11
Number drives:     1
Compression:     Yes
Encryption:     No
Multiflow:     Yes
Periodic backup:     Yes
Archive:     Yes

License customer:     UCSF_PHARMA
License name:     TAPE_AUTOMATION
License serial:     TE91GNVGRHLDP1
License checksum:     S99ZJRKW43VVYK
Comment:     Tape Automation
Creation date:     2009-08-18 21:06:05
License created by:     bspagnuolo
Expire time:     Never
Number:     1

License customer:     UCSF_PHARMA
License name:     ARKADD_FLOW
License serial:     TE91GNVTRHLDP1
License checksum:     9ADJYG7PF4LT78
Comment:     Add Flows
Creation date:     2009-04-09 18:18:54
License created by:     jscheldt
Expire time:     Never
Number:     11

License customer:     UCSF_PHARMA
License name:     ARKADD_DRIVE
License serial:     TE91GNVGRHLDP1
License checksum:     JXS6B46BENNC9Z
Comment:     Add Drives
Creation date:     2009-08-21 17:01:38
License created by:     bspagnuolo
Expire time:     Never
Number:     1

License customer:     UCSF_PHARMA
License name:     ARKADD_CLI3
License serial:     TE91GNVTRHLDP1
License checksum:     XMWWFR9YHZSNCW
Comment:     Add Clients type 3 (Arkeia >= 5.2)
Creation date:     2009-04-09 18:18:13
License created by:     jscheldt
Expire time:     Never
Number:     11

License customer:     UCSF_PHARMA_a
License name:     ARKADD_CLI3
License serial:     TE91GNVHRHLDP1
License checksum:     72JA45ACJD93VA
Comment:     Add Clients type 3 (Arkeia >= 5.2)
Creation date:     2010-03-11 02:08:30
License created by:     hbrackett
Expire time:     Never
Number:     2

To list licenses

/opt/arkeia/bin/arkchklic

To remove a client (remove a machine from being backed up)

/opt/arkeia/bin/arkrmhost hostname

PROBLEM: The web UI freezes when trying to start a backup

"Hello,

SUMMARY:

The backup is failing due to a hostname change on the backup server itself.

DETAILS:

1) uname -a reports the real $HOSTNAME

backup.uoft.bkslab.org

2) However, the name that was present when Arkeia was installed was "backup.bkslab.org"

2014/01/14 14:32:33 E19010060 arkvlib (19004) get_hostlist [L799]: Error: the host backup.bkslab.org is not server
2014/01/14 14:32:33 E19010090 arkvlib (19004) get_hostlist [L204]: Bad end
2014/01/14 14:32:33 E19010060 arkvlib (19004) get_hostlist [L799]: Error: the host backup.bkslab.org is not server
2014/01/14 14:32:33 E19010090 arkvlib (19004) get_hostlist [L204]: Bad end

SOLUTION:

1) Stop Arkeia

/opt/arkeia/bin/arkboot stop

2) The /etc/host file looks OK

142.150.250.19 backup.pharmacy.uoft.bkslab.org backup.uoft.bkslab.org

3) Delete the existing entry in the admin.cfg file and replace it with the current $HOSTNAME

vi /opt/arkeia/arkeiad/admin.cfg backup.uoft.bkslab.org

4) Manually edit the rhost.lst file and delete the backup servers corrupted entry located at the bottom of the file

vi /opt/arkeia/arkeiad/rhost.lst

ITEM { "OLDNAME" "backup.bkslab.org" "IS_MEDIA_SERVER" "1" ";" "INETADDR\"\"" "169.230.26.116" "INETADDR" "142.150.250.19" "INETNUM" "335189646" "DISTRIBNAME" "Red Hat CentOS release 5.4 (Final)" "OSNAME" "Linux 2.6.18-164.11.1.el5 x86_64" "VERSION" "Arkeia Network Backup 8.2.9" "NUMPOINTS" "997" "ID" "N" "ISDIR" "h" "PORT" "617" "NOM" }

5) Issue the arkmvhost name command to rename Arkeia

/opt/arkeia/bin/arkmvhost --host backup.bkslab.org --newhost backup.uoft.bkslab.org

6) Start Arkeia (will create a new entry in the rhost.lst file at the top of the file)

/opt/arkeia/bin/arkboot start

7) Verify the library slots inventory has completed and the issue should be corrected.

Regards, S B

Hello,

> If I add trees or rename trees, do I need to register them somewhere?

All trees in the Arkeia Savepack must have a registered client machine on the backup server. Failure to backup a client that is not registered will result in a Type 1 client license error.

> If so, how and where?

Please see the "Adding Client Section" at this URL to properly add/register clients to the backup server. http://wiki.arkeia.com/mediawiki/index.php/Arkeia_Network_Backup_Documentation

> Adding or renaming trees doesn't affect the # of licenses, does it?

Only if you add a hostname to the Savepack that is not registered to the backup server or exceeds the license limit.

S B

If I add trees or rename trees, do I need to register them somewhere? If so, how and where? Adding or renaming trees doesn't affect the # of licenses, does it?

Hello,

You can run the command /opt/arkeia/bin/arkchklic too see how many licenses and what type are allowed on the backup server and which hostnames are claiming and in violation of the license.

• Clients and their indexes can removed using the /opt/arkeia/bin/arkrmost command.
• Clients that are (unknown Type 1) are no longer registered to the backup server.

These clients must be re-registered or deleted

To see what ports Arkeia is running on

netstat -pant | grep ark

To Check and See Which Drive is Okay to Use

On the server type:

mt -f /devst0 status 

and

mt -f /dev/st1 status

Whichever one responds with BOT ONLINE IM_REP_EN is the device the drive should be set to.

External References

http://wiki.arkeia.com/index.php/WD_Arkeia_Version_10.1_Linux_Installation_Guide

http://wiki.arkeia.com/index.php/Network_Backup_Version_8.1_Tape_Library_Guide#Tape_Library_Details

http://wiki.arkeia.com/index.php/Arkeia_Network_Backup_Version_10.0_Quick_Start_Guide_-_English#Attach_the_Tape_Drive