TMG to GRAMPS how I did it using tmg2gramps script.
I have used The Master Genealogist version (TMG9) for years, and have maintained it running under Linux wine & crossover office. However, it is showing its age with lack of UTF-8 support. Not long ago the developer of TMG decided to stop developing the program and the last version ever to be available is 9.05. Gradually I am going over to GRAMPS, currently v4.2 under LinuxMint 17.. For the moment I am running a the two in parallel in an attempt to get GRAMPS to produce web pages similar to those from John Cardinals SecondSite (v5). GRAMPS is open source and multi-platform so I can give copies to all my relatives. It has the advantage that special reports can be written using python code and I think this will more than compensate for the beautiful sentence statements which could be constructed in TMG. I still think TMG is a program that sets standards to be met by others. If there is a TMG9 which runs easily on Linux I would still use it. I think the simplest way of using TMG9 in Linux is to emulate WinXP as a guest within VirtualGBox. I think you could within any version of windows use VirtualBox to run WinXP as a guest.
tmg2gramps is Perl program which will translate a TMG9 file. If you intend to try it please read the GRAMPS-project wiki before going any further. It gives a well organised simplified version of what I describe below. These here are just my own notes which may help someone.
Note that the wiki suggest that you download an altered version of tmg2gramps-v8 altered to cope with TMG8. There are three lines changed between tmg2gramps.pl and trm2gramps-v8. The lreatter should really be named tmg2gramps-v8.pl as it is an executable Perl file. No comments are given regarding the changes or who made them so I list them below for clarity.
See also: https://github.com/sam-m888/tmg2gramps for tmg2gramps-v8
l117 my $sthbio = $dbh->prepare( "select etypenum from $tagtable where etypename='Father-Bio' or etypename='Mother-Bio' or etypename='Parent-Bio'" ); my $sthbio = $dbh->prepare( "select etypenum from $tagtable where etypename='Father-Biological' or etypename='Mother-Biological' or etypename='Parent-Biological'" ); l168 # print STDERR " does exist\n"; print STDERR " does exist\n"; l683 my $sthtypenums = $dbh->prepare( "select etypenum from $tagtable where etypename='Father-Bio' or etypename='Mother-Bio' or etypename='Parent-Bio'" ); my $sthtypenums = $dbh->prepare( "select etypenum from $tagtable where etypename='Father-Biological' or etypename='Mother-Biological' or etypename='Parent-Biological'" );
ERROR: DBD::mysql::st execute failed: You have an error in your SQL syntax; ... near ')' t line 1 at ./tmg2gramps-v8.pl line 130
If you get the above error you might try using a very slightly adjusted version of the file tmg2gramps-v8.pl which I have made. It addresses this error & runs with files created by TMG9.05 or as early as TMG6 (perhaps 5). It is temporarily here while it is tested. Navigate to your directory holding the original tmg2gramps.pl file and change its name to, say. tmg2gramps.pl.original. Then copy the new file to the directory renaming it as tmg2gramps.pl. Then alter its permissions so that it is executable and run it as before. Let me know how you get on.
My notes on using tmg2gramps.
There is one oddity though. If you try to translate the Sample file which comes with TMG9 tmg2gramps gives an error and stops.
"Generating families DBD::mysql::st execute failed: You have an error in your SQL syntax;
I suggest you try the tip given above in "ERROR: DBD::mysql::st execute failed."
It may help in TMG9 to run File > Maintenance > Validate File Integrity which corrects any relationship errors.
However, it worked well with my own c700 person TMG9 data base with external exhibits, witnesses, roles, citations and so on.
Download tmg2gramps.pl and its associated files from GRAMPS wiki mentioned above.
See the README.tmg2gramps on that site for more information. Various other files give interesting details of the TMG database but are not needed.
What follows below is a modified version of the README above as it applied to my TMG7 data and should be read alongside it with the two links below. However, my text below is self contained and should work with the necessary additions for your system. There is an example using the TMG Sample project.
My system LinuxMint13 (based on Ubuntu Precise 12.04 LTS) with Gramps 3.4.5-1.
tmg2gramps was written to convert TMG v6 to GRAMPS but here I used it to convert from v7. There has been little change since v6 and the latest v8 appears very similar to v7 I would imagine tmg2gramps will work with it equally well.
For more information on how it works I relied on these links**, which outline some of the problems you may encounter. However, most of the salient points are included in my instructions plus information on how to deal with the situation where my media files were on a drive mapped in Windows to s: or S:. Windows seems inconsistent in its use of upper and lower case in filenames and Linux will see these as different files. So I found I had to translate all filenames to lower case and the method is described below.
** Additional links here:
http://gramps.1791082.n4.nabble.com/TMG-to-GRAMPS-td1810437.html
http://gramps.1791082.n4.nabble.com/attachment/1810442/0/Report_on_TMG2GRAMPS_and_GRAMPS.odt
The above report points out a number of things including that the Reports Text> Complete Individual Text Report is not very complete and includes entry like "Source Text" from a GRAMPS Note without including the contents of it. However, the Web Pages> Narrated Web Report contains all the contents of the Notes and thus seems to constitute a complete report if you follow the pages for an individual. So check the Web Narrative Report before concluding that information has not been transferred. Allow the option to see Individual IDs and you will see how the TMG components have been translated and where to look. Unfortunately, the Web Narrative Report does not include the Note IDs and you will need to search them out in GRAMPS Note View.
How to set up MySQL
Before using tmg2gramps you first need to set to MySQL and this is how I did it, after looking at this link:
http://www.debuntu.org/how-to-create-a-mysql-database-and-set-privileges-to-a-user/
In a terminal do the following substitute your TMG project name without extension for MYNAMEtmg7
mysql -u root -p
create database MYNAMEtmg7;
grant usage on *.* to ian@localhost identified by 'PASSWORD';
grant all privileges on MYNAMEtmg7.* to ian@localhost ;
to check connection do
mysql -u ian -p'PASSWORD' MYNAMEtmg7 (NB This must be running when you continue with tmg2gramps in a second terminal.)
Running tmg2gramps
Before doing anything in TMG9 run File > Maintenance > Validate File Integrity to make sure that there are no relationship or other errors before proceeding.
The example below makes the following assumptions:
name of TMG project: MYNAMEtmg7
home directory: /home/ian/
directory under home holding tmg database files: GRAMPS/Projects
name of MySQL database to use: MYNAMEtmg7
username for MySQL access: ian
password for MySQL access: PASSWORD
name of destination Gramps xml file: MYNAMETMG7_Gramps.xml
From the directory where you unpacked the tmg2gramps program in a terminal:
1. convert all filenames to lower case so that Linux will find them with command: (see below for web site with explanation)
find . -type f | perl -n -e 'chomp; system("mv", $_, lc($_))'
2. In a new terminal in the directory where you unpacked it, run tmg2gramps as follows, all on one line, swapping your directories and filenames:
./tmg2gramps.pl /home/ian/GRAMPS/Projects MYNAMEtmg7 MYNAMETMG7_Gramps.xml MYNAMEtmg7 ian PASSWORD
(If using the TMG sample file you might do this: [NB the TMG9 Sample fails unless easliy corrected in TMG9- see above: File > Maintenance > Validate File Integrity .]
./tmg2gramps.pl /home/ian/GRAMPS/Projects/Sample sample SampleTMG7_Gramps.xml MYNAMEtmg7 ian PASSWORD)
3. Rename the file to MYNAMETMG7_Gramps.xml.gramps. GRAMPS will not accept the xml file for import without this extension. DO NOT open it as a GRAMPS database, it must be imported.
4. Create a new project in GRAMPS. Be careful this must be empty. GRAMPS will import the file and add it to an existing database causing a bit of chaos.
5. Open that new project in GRAMPS and do Family Tree>Import the file just renamed into it.
ERRORS on Import
If you get errors and the program stops. Look at the error message and note the problem line number. Then in an editor like Kate look at that line. You may find some character that is not legal in the filename. Change the filename and try again.
In the xml file lines like this in the from the Sample_UK file:
<placeobj id="P0181" handle="_800000181" change="1401860845"><location ="caroline.gurney@ntlworld.com" />
</placeobj>will cause an error which is reported by line number & column as GRAMPS imports it and will require to be edited individually. There is a token missing before ="caroline.gurney@ntlworld.com". It should be edited to email="caroline.gurney@ntlworld.com" or E-mail="caroline.gurney@ntlworld.com".
SOLVE THE EMAIL problem by commenting out line 24 - in the perl script TmgGramps/Place.pm like this:
# 'Temple'=>'',
After that change Place.pm & Repository.pm will give a warning about the email field (synonym Temple in TMG) which it cannot handle, just like Lat/Long and Addressee fields.
This summary gives a clue to what is happening but ignore it if you don't understand it.
Under Master Repository List
Repository # 15
Abbreviation: * GURNEY Caroline
Under Edit Place
Place Style: Repository
Abbreviation * Mrs C M Gurney
Name ID 0
Name - Other Caroline Gurney
L1 Archivist Mrs C M Gurney Master Place: Addressee
L10 E-mail caroline.gurney@ntlworld.com Master Place: Temple
The external media files could not be found being on another drive. So I did the following by editing the xml file: (However there is a simpler method using GRAMPS Media Manager - see below.)
Once generated edit MYNAMETMG7_Gramps.xml using an editor like kate with line length set to c2000 or more:
In lines like: <file src="S:\Maps\MaidenDetail1.jpg" mime="ERROR: cannot open `S:MapsMaidenDetail1.jpg' (No such file or directory)" description="MaidenDetail1"/> replace s:\ or S:\ with /media/AcerWinData/AcerIansDocs/TMGMYNAMEFamilyTree/ Change Maps\Maiden to Maps/Maiden & similar
mime="ERROR: cannot open `S:MapsMaidenDetail1.jpg' (No such file or directory)" to mime="image/jpeg; charset=binary"
mime="ERROR: cannot open replace with mime="image/jpeg; charset=binary" THEN remove rest to descriptionThe result should look something like this:
<file src="/media/AcerWinData/AcerIansDocs/TMGMYNAMEFamilyTree/Maps\MaidenDetail1.jpg" mime="image/jpeg; charset=binary" description="MaidenDetail1"/>
Choose mime type from this list in Exhibit.pm. However, I think you probably can omit the mime type tab altogether and the default will work fine.
my %mimes = ( 'jpg'=>'image/jpeg', 'jpeg'=>'image/jpeg', 'tiff'=>'image/tiff', 'tif'=>'image/tiff',
'gif'=>'image/gif', 'png'=>'image/png', 'svg'=>'image/svg+xml', 'bmp'=>'image/x-ms-bmp',
'au'=>'/audio/basic', 'wav'=>'audio/x-wav', 'mp3'=>'audio/mpeg', 'wma'=>'audio/x-ms-wma',
'ogg'=>'application/ogg', 'pdf'=>'application/pdf', 'doc'=>'application/msword',
'ps'=>'application/postscript', 'xml'=>'application/xml', 'odt'=>'application/vnd.oasis.opendocument.text',
'html'=>'text/html', 'htm'=>'text/html', 'txt'=>'text/plain', 'rtf'=>'text/rtf',
'fli'=>'video/fli', 'mpeg'=>'video/mpeg', 'mpg'=>'/video/mpg', 'qt'=>'/video/quicktime',
'mov'=>'video/quicktime', 'avi'=>'video/x-msvideo'
);For .txt
<file src="exhibit00001.txt" mime="text/plain; charset=us-ascii" description="Letter from Frank Alexander to his children shortly after the death of his wife. From a copy made by Louise."/>.txt" mime="image/jpeg; charset=binary" .txt" mime="text/plain; charset=us-ascii"
.gif" mime="image/jpeg; charset=binary" .gif" mime="image/gif; charset=binary"
.tif" mime="image/jpeg; charset=binary" .tif" mime="image/tiff; charset=binary"
.html" mime="image/jpeg; charset=binary" .html" mime="text/html; charset=ISO-8859-1" This is probably right
Change the s: or S: into /media/MyDisk in GRAMPS using the Media Manager in GRAMPS which is probably much easier than the method described above. Here are the stages:
Tools>Utilities>Media Manager > Replace substings in the path:
Replace \ with / To alter the paths required by Linux.
Replace upper case S: with ../../media/AcerWinData/AcerIansDocs/TMGMyNameFamilyTree
Replace lower case s: with../../media/AcerWinData/AcerIansDocs/TMGMyNameFamilyTree
(Both s & S needed as TMG in Windows seems to use them randomly)
After that all c300 images appeared. Editing as above was not required. NB replace upper and lower case s because s and S seem to be used randomly by windows as it is not case sensitive.
How to Correct "mime="ERROR: cannot open2" in Gramps
Apply the path correction mentioned above. Then in the Media tab click on each media entry with an error and click on path. The file should be there and on clicking OK (twice) the mime type entry will be corrected. This is important because without it you will not see thumbnails in, for example, the Narrative Web Report.
How to convert all TMG datbase files to lowercase
Files copied to /home/ian/GRAMPS/Projects/ changed to lowercase names. TMG has extensions etc some in uppercase some lower. Make all lowercase using the Perl command line below.
cf http://stackoverflow.com/questions/13051871/change-filenames-to-lowercase-in-ubuntu-in-all-subdirectories
Perl has a locale-aware lc() function which might work better:
find . -type f | perl -n -e 'chomp; system("mv", $_, lc($_))'
Note that this script handles whitespace in filenames, but not newlines. And there's no protection against collisions, if you have "ASDF.txt" and "asdf.txt" one is going to get clobbered.
Resulting GRAMPS file
I was surprised that everything seemed to be transferred. I checked this out by creating a Web Narrative Report going through evey option and ticking it for the Entire Database. As noted above, the Complete Individual Text Report is not complete because it does not appear to include the contents of, for example, Source Texts which is where TMG Memos seem to end up. But they are only shown as "Source Text" etc. and can be found on the Note View. The web version shows them in detail.
However, there seemed to end up being huge numbers of notes.
Each Source Note in TMG ends up as a separate Source Text in GRAMPS with its own unique ID like N0001. I found that I had to manually merge 100s two at a time. I have looked for a way of merging more than two at once but I think I will have to write this myself.
Another possibility is to use a macro with Autokey. However, the installation of this package is broken at the moment and it will not work (20121126
<No Citation> peculiarity
Another GRAMPS peculiarity completely confused me. Under views like Events with the Citations Gramplet you may see under the Source/Citations column <No Citation>. In fact there probably is a citation which I could see if I click on <No Citation>. What is missing ia a Volume/Page entry. This is a misleading in GRAMPS; it might be preferable to say <empty>. If I added anything free text like "Jenny Smiths Birth Cert in my possession" in the Volume/Page this would show in place of <No Citation>. For a long time I thought that the citation had not been transferred over by tmg2gramps. It had and the details were in a note. It took me a while to work out that the note was attached to the Citation. The Volume/Page entry is supposed to reflect where the Citation is in the Source. The key for me was the concept that the Citation is the link between the Source and the Event. (There are plans, I believe, to introduce Evidenced based sources & citations which are more in line with what I would like but there is no time-frame for this that I can see: this "Evidence style sources" may confuse you if you are a beginner and I would skip it.)
Further tweeking by editing an exported GRAMPS file
Peter, referred to below, suggests editing the GRAMPS file and reimporting it to GRAMPS to make changes. I found it necessary to go through the following to make it work. The KDE editor Kate would open but not save the file because of encoding issues or because the file is compressed: Here is the way to handle this.
1 Export GRAMPS file, 2 select & copy all the text, 3 paste all the text into a new text file, 4 save the file with the same name. (The file size increased from 144k to 1.1Mb.) Now carry out any edits required on this file and import it into GRAMPS.
I have been given support with this by Peter who found my website recently and has expanded on what is written above by explaining his discoveries transferring a TMG6 database with over 5,000 people to GRAMPS. Please see his additions to the GRAMPS wiki under tmg2gramps. This explains how tmg2gramps handles Repositories, Citations and Notes which are mentioned above.
********************************************************************************************************************************************************
REFERENCES
http://www.whollygenes.com/
http://www.gramps-project.org/
http://www.cohsoft.com.au/tmg2gramps/