Old documentation! Read the manual for the latest version in the Netatalk v2 release series.
Table of Contents
Version 2.x of the Netatalk suite includes significant changes and enhancements in functionality compared to 1.x. Netatalk 2.x is now supported which allows UTF8-MAC encoded filenames of up to 255 bytes (85-255 chars) in length amongst other things. The Catalogue Node ID (CNID) subsystem has been reworked as well and should now be much more robust. For an overview of what CNIDs are and why you need them please see the CNID section in the manual.
The downside of these enhancements is that upgrading to Netatalk 2.x is not a process that can be easily automated. Too many factors depend on site specific configuration and administrators have to make choices that suit their requirements. This document attempts to clarify the issues and outline the steps that need to be taken for a successful upgrade. As usual, the first of these steps should be to make a complete backup of all volumes and home directories that were in use with Netatalk before. Afterwards, you'll have to decide
what encoding to use for filenames in the future and how to convert existing filenames
what storage scheme to use for CNIDs and maybe convert an existing database to that scheme
The following two sections deal with each of these areas in turn.
Netatalk 1.x saved filenames in the so called CAP encoding by default. Alternatively, there was the NLS system, that allowed you to convert filenames to other codepages, like ISO-8859-1.
For Netatalk 2.x the charset conversion routines had to be completely rewritten to support AFP 3.x. For more indepth information on character sets please read the Unicode/charsets section in the manual.
As a consequence, Netatalk 2.x now stores filenames in UTF-8 by default. Additionally you have to specify a maccodepage in afpd.conf, if your Mac clients are not using MacRoman.
The format of the metadata files stored in the .AppleDouble folders has changed from AppleDouble v1 to AppleDouble v2. Netatalk 2.x is still able to use AD1 files, if configured. Otherwise ADv1 files will silently be updated to the new ADv2 format, which will prevent you from using this volume with 1.x again.
Do not share a 1.x volume with Netatalk 2.x without setting the proper options!
In case of MacChineseTraditional, MacJapanese or MacKorean, This method cannot be used.
Before upgrading, copy all files to the client. And restore all files to netatalk 2.0.4 or later.
You should consider 'upgrading' your volumes using the new defaults UTF-8 and AppleDouble v2, even if this is a time consuming process. AFP 3.x uses UTF-8 and it is impossible to fully map UTF-8 to any of the old volume formats.
To convert the 1.x CAP or NLS encoded volumes on the server, we provide the uniconv(1) utility. Please see the man page for details.
Another option to perform an upgrade, is to copy all files using a Mac client. Either copy the volume to a Mac while you are still running 1.6, then install 2.x and copy the data back to a fresh share, or try to set up the volume with the compatibility options described below and do a share to share copy.
Using a 1.x CAP encoded volume is still possible with Netatalk 2.x. To work properly, the following options need to be set, matching your 1.x setup:
afpd.conf:
maccodepage
AppleVolumes.default:
volcharset
adouble
You have to make sure maccodepage matches your
Apple clients codepage. For western users the default
Mac_Roman should be fine.
Set volcharset to
ASCII.
Set adouble:v1, this will make sure the metadata
files will not be changed to AppleDouble v2. If you do not set this
option, it will not be possible to use the volume with Netatalk 1.x
anymore.
Example:
afpd.conf:
- -transall -maccodepage:MAC_CENTRALEUROPE
AppleVolumes.default:
/path/to/share "1.x Volume" adouble:v1 volcharset:ASCII
Whether you can still use an 1.x NLS encoded volume with Netatalk 2.x mainly depends on which NLS setting you used with 1.x.
Make sure you set the correct maccodepage in
afpd.conf !
Use the following settings in
AppleVolumes.default:
/path/to/share "1.x Volume" adouble:v1 volcharset:ISO-8859-1
Sorry, you're out of luck. This NLS contains a non standard mapping and is not supported by afpd anymore. You'll have to convert the volume to a supported encoding.
Using the following settings in
AppleVolumes.default might work, but is
untested:
/path/to/share "1.x Volume" adouble:v1 volcharset:CP437
Using the following settings in
AppleVolumes.default might work, but is
untested:
/path/to/share "1.x Volume" adouble:v1 volcharset:CP850
Using the following settings in
AppleVolumes.default might work, but is
untested:
/path/to/share "1.x Volume" adouble:v1 volcharset:KOI8-R
All of the above require iconv to be installed and to supply the volcharset codepage!
Previous versions of Netatalk allocated CNIDs either on the fly or CNIDs were recorded in a persistent database. "On the fly methods" work by either generating a CNID from the device and inode number or simply by using a counter that is increased by one on each access to a file or directory from the client. The counter only lasts for the lifetime of an afpd daemon process and inode numbers are reused for a different file once the original file has been deleted. These methods therefore violate a fundamental assumption: A CNID once assigned must never be reused for the lifetime of a volume. Netatalk 2.0 supports one "On the fly scheme" called last. It computes CNIDs for files from device and inode of the file and uses a counter for directories. You should think twice about using it in production. Depending on your needs and the semantics of the underlying file system it might be OK on read only volumes, but even there we are not certain if OS X clients will work properly.
That leaves the CNID schemes that use persistent storage for CNIDs. Netatalk 2.0 supports two: cdb and dbd. Both are based on the Berkeley DB database library as before. One difference is, though, that you are not restricted to using a single scheme for all of your volumes that has to be determined at compile time. The CNID scheme (also called a "CNID backend") is now a runtime option for a volume. That means that you can make the choice per volume based on your requirements. Here are the properties as well as the advantages and disadvantages of the three supported schemes:
cdb: Roughly analogous to the Netatalk 1.6.x versions with what was called then the "DID scheme" option set to "cnid" and the "CNID with Concurrent Data Store" option set to "yes". Access to the CNID database for a volume happens directly from the Netatalk afpd daemons. A Berkeley DB locking scheme (the "Concurrent Data Store" bit) is used to avoid database inconsistencies. Robustness is much improved compared to previous releases. The CNID database can only become corrupted if an afpd daemon crashes unexpectedly, is killed by the administrator or the whole machine crashes.
dbd: There is only a single daemon that accesses the CNID database for a given volume. Any afpd process that wishes to retrieve or update CNIDs for that volume needs to do it via the daemon. The CNID can database be (this is a compile time option) updated under Berkeley DB transactional protection. This design combined with the transactional updates makes the CNID database crashproof: Any of the participating afpd daemons, the database daemon itself or the whole machine can crash and the CNID database should still be in a consistent state. The downside to this is that the speed of updates and retrieval is slower than with the cdb scheme. If this is a problem, you might want to disable transactions at Netatalk compile time (currently, the default is to compile without transactions anyway). That will give you safety against afpd crashing, but not if the machine goes down unexpectedly. Also, have a look at the nosync option documented in the cnid_dbd manual page.
It is also possible to switch between cdb and dbd for a
given volume, since they use the same database format. You just have to
shut down all processes accessing the database cleanly, make the necessary
configuration changes and restart. Please note, that you can easily
specify a default CNID backend for all shares by applying the
cnidscheme option to the ":DEFAULT:" share (compare
with the AppleVolumes.default(5) manual page for details).
Note that the dbd backend needs an auxiliary daemon, called cnid_metad, to work. It should be started together with afpd. If the dbd backend is compiled into afpd (the default), this should happen automatically. If you cannot find it in the process list even though the dbd backend is used please check for errors in the startup scripts.
If you compile Netatalk 2.0 yourself and invoke configure --help, you'll notice that there are in fact more CNID backends to chose from. Don't use any of them. They are either broken or incomplete. Some of them might turn into something useful in the future.
That is easy. Just pick a CNID backend from above, configure it
properly in afpd.conf and the AppleVolumes file and start up the
necessary Netatalk processes. The databases will be automatically
created in a subdirectory .AppleDB of the volume in question.
In that case the CNID databases need to be upgraded. A script called cnid2_create that comes with Netatalk 2.0 does most of the work. The steps you have to take depend on what version of Berkeley DB is installed on your system. If you already use one of the supported versions of Berkeley DB (4.1.25 or 4.2.52) for your old Netatalk installation and plan to use it for Netatalk 2.0 as well just use the db_dump and db_load utilities that came with it as indicated below. Otherwise it is probably best to have the old and the new (to be used with Netatalk 2.0) version of Berkeley DB installed side by side until you have finished the upgrade. The reason for this is that we will dump out the old databases with the currently installed version of Berkeley DB in ASCII format and reload them with the new version. This avoids any incompatibility problems between Berkeley DB releases with respect to the on-disk format.
For each volume to be upgraded, follow these steps
Stop all afpd daemons accessing the volume.
Change to the database directory for that volume, most
likely the .AppleDB subdirectory of the
volume toplevel directory in question.
Dump the contents of cnid.db and
didname.db using the old (installed) version
of Berkeley DB like this:
db_dump -f cnid.dump cnid.db db_dump -f didname.dump didname.db
Make sure the db_dump utility you are using is the correct
(currently used) one. Use the full path to the db_dump executable
if in doubt. So if this database was created with Berkeley DB 3.xx
installed in /usr/local/db3 use
/usr/local/db3/bin/db_dump instead. This will
create two files, cnid.dump and
didname.dump in the current directory.
Run the cnid2_create script:
/path/to/netatalk/bin/cnid2_create
The script assumes that .AppleDB is a
subdirectory of the volume directory to be upgraded. If that is
not the case give the full path name of the volume as the first
argument to cnid2_create. The script will
create a file cnid2.dump in ASCII
format.
Remove the old Berkeley DB environment and logfiles (if present):
rm __db.* log.*
Load cnid2.dump into the new database.
You should use the db_load utility of Berkeley
DB that will be used with version 2.0 of Netatalk. So if Berkeley
DB 4.xx lives in /usr/local/db4 use
/usr/local/db4/bin/db_load -f cnid2.dump cnid2.db
This will create the new database file,
cnid2.db. Remove the old database files
cnid.db, didname.db and
devino.db. The intermediate files
cnid.dump, didname.dump
and cnid2.dump can be removed now or at some
later time.
If you do not want to have two versions of Berkeley DB installed side by side during the upgrade, you should first dump out the old databases as indicated above for all volumes, upgrade Berkeley DB and then load them with db_load. The cnid2_create script can be run before or after the upgrade. Berkeley DB environment and logfiles should still be removed before running db_load.
If you are absolutely sure what you are doing, you can also just
clear out all database files from the .AppleDB
directories. They will be recreated, but will not contain the same CNIDs as before!! That might
lead to all sorts of problems, like aliases not working any more on
clients. As I said, make sure you know the consequences and don't mind
them.
Providing a test environment in parallel with the existing
production installation is not difficult and, if done properly, it should
not in any way disrupt the normal operation. However, as always, it is
recommended to make a backup of the existing installation before
proceeding. When compiling a newer netatalk version you should also take
care that you do not overwrite the binaries of an older version (make use
of the --prefix= configure option).
There could be more than two afpd servers running on one UNIX box. You just have to be careful to keep them from running into each other:
the shares/volumes
(AppleVolumes.default)
the PID file (afpd -P
command line option)
the port number (-port option in
afpd.conf)
no use of AppleTalk (-noddp option in
afpd.conf)
You should test the new Netatalk version with both a freshly created new share and another one that has been duplicated/converted from an already existing volume. This helps finding mistakes you probably made in the upgrade process when the first share behaves well and the latter not.
First, you have to provide some space for the test share. Just create a directory on one of your data filesystems. However, this directory must not be accessible from the production afpd server. Don't forget to set appropriate permissions for the share. For example:
mkdir /macdata/testshare chown root.macusers /macdata/testshare chmod g+wrx,g+s /macdata/testshare
Ensure that users cannot access the share in question and copy the
whole contents (including all the metadata directories like
.AppleDB) to another location.
cp -pr /production/testshare /macdata/
Then do the somewhat extensive upgrade of CNID databases and filename encodings outlined earlier in this chapter.
Normally the test afpd cannot listen on the standard afpovertcp
port, because that one is already bound by the production afpd. So the
afpd.conf should look like
- -noddp -nouservol -port 5480 -loginmsg "WARNING: test server"
You also have to present the share to the Mac users by editing the
AppleVolumes.default file. Remove the line
containing a single "~" at the end of the file and append something
like:
/macdata/testshare "Test Volume (not production)"
That should be the only uncommented line in the file. Do not forget to adjust encoding and AppleDouble setting when you're not using the recommended defaults UTF8 and ADv2.
In case you have many users and want to restrict access to the
test server, there is a provision for that in the
AppleVolumes.default file. First, create a group
named, say, afpdtest, and put in it all users you would
want to enable access to the test volumes. Then, instead of the line
above, append a line like this one:
/macdata/testshare "Test Volume (not production)" allow:@afpdtest
The test afpd server can be started now:
$TESTDIR/sbin/afpd -P /var/run/afpd-test.pid
You can also put this line in the production netatalk start script, in the "start" case. In the "stop" case, you should insert
[ -f /var/run/afpd-test.pid ] && kill `cat /var/run/afpd-test.pid`
The production server has to know about the test server. Otherwise
the Mac users would not be able to see the test server in their
choosers. Append the following line to the production
afpd.conf (usually in
/etc/netatalk/afpd.conf):
"Test server (not production)" -proxy -uamlist "" -port 5480
and restart the production netatalk. Note that the port directive
here should match the one which appears above in the test
afpd.conf.
Note that there is a limit of 31 characters for the server's name. Should the name be longer, then afpd will just refuse to register the server.
The test server should appear in the chooser on Macs. You can also test that from the UNIX command line:
nbplkup =:AFPServer
or
netstat -an | grep 5480
If everything went fine spread the word about the test server among your more experienced Mac users and see whether things work as expected.
