Results 1 to 6 of 6

Thread: how to MERGE Zbk Archives

  1. #1
    Junior Member
    Join Date
    Jan 2016
    Posts
    3

    how to MERGE Zbk Archives

    Hi all,

    years ago I've posted here (the other section) asking if there was a way to merge two or more zbk sets (zarafa-backup's output) it seems ugly but without having other way to keep the mail we do several full backup every three months then a daily incremental backup.

    After this work we ends with two nas full of huge files.. which all almost contain the same messages.

    So the idea is:

    read: fullbackup1/user1.index.zbk as A
    read: fullbackup2/user2.index.zbk as B
    diff (B-A) of the two lists to identify new messages/folder/items

    create: full-and-mergedbackup/user1.index.zbk
    that contain A + (B-A).

    What I've done till now:

    read A, B ->Ok
    Diff A, B ->Ok
    Extract the single message:
    identified the start of the element (easy, from the index.zbk row)
    identified the end of the element (assumed as the start -1 of the next row)
    now the strong part: what is the format of the message stored in the archive? do I have to consider it as binary then read and write byte by byte or there is a structure somewhere that I can use to further analyze the file or that can help in some manner?

    next step: store in the new data file the message and write index file emulating the zarafa-backup utility

    obiously the resulting file have to be zarafa-restore compliant, we should have to be able to restore messages from it.

    Do someone has seen somewhat on this topic? I'm aware that is not directly zarafa-python api related but I've posted here because my entire program is written in python. (if it can solve this problem I can switch to c++ or php)

    My zarafa setup is 7.1.5-4205

    Thanks a lot for any help

  2. #2
    Zarafa

    Join Date
    Jan 2009
    Location
    Hanover, Germany
    Posts
    1,891
    Sorry, but I have to ask. Why do you want to merge the data files from zarafa-backup? to deduplicate and reduce required storage? Actually I would recommend to do a fresh bricklevel backup more often, since in case there is a foul increment in between you can forget the whole backup set.

    In general zarafa-backup is primarily to have an easy restore of single mailboxes or messages, if you are looking for long term archiving of backups then using mysql backups is probably more space efficient (can be better compressed and attachments on disk can easily be deduplicated).
    Last edited by fbartels; 14-01-2016 at 09:58 AM.
    Regards Felix

    How to get Kopano

    Zarafa ALPHA/BETA/RC feedback in BETA forum please.
    Zarafa IRC chat: irc.freenode.com > #zarafa
    Zarafa documentation: http://documentation.zarafa.com/

    No support via PM! Please contact our sales team for an offer if you want my full attention.

  3. #3
    Junior Member
    Join Date
    Jan 2016
    Posts
    3
    Yep, Deduplicate and reduce.. mainly I've to do that because between backups some user can have deleted some messages from their store (accidentally or by their will) but I've to keep them in any case.

    I'm agree with you that mysql backups could have done the trick but I've to merge zbk backups since 2011 and I simply cant start from today on.

    I'm with you that is not the "right" way but the only other way I've found is to do a fresh zarafa installation, import every single archive, do a duplicate check with some unknown outlook plugin then re-export all.. two months of work at least

    Hope to get the information on how zarafa backup writes his data files? if not I've to move to the option b..

    I almost forgot, thanks for replying!

  4. #4
    Zarafa

    Join Date
    Jan 2009
    Location
    Hanover, Germany
    Posts
    1,891
    But why since 2011? I find it hard to imagine that someone will realise tomorrow, that he needs to restore a mail he deleted over four years ago.

    To my knowledge you are correct with the following: "do I have to consider it as binary then read and write byte by byte" thats basically what zarafa-restore does.
    Regards Felix

    How to get Kopano

    Zarafa ALPHA/BETA/RC feedback in BETA forum please.
    Zarafa IRC chat: irc.freenode.com > #zarafa
    Zarafa documentation: http://documentation.zarafa.com/

    No support via PM! Please contact our sales team for an offer if you want my full attention.

  5. #5
    Junior Member
    Join Date
    Jan 2016
    Posts
    3
    Thanks, is for a sort of data retention, keeping communications from customers, long projects, old changes.. the phrases like "he told me that in the 2009" is not so uncommon in this place.

    I've to switch to somewhat better like zarafa archive or some system like mailarchiva, but this can be from now on, that still not cover the past.

    I'll try byte by byte approach and I'll let you know.

  6. #6
    Zarafa

    Join Date
    Jan 2009
    Location
    Hanover, Germany
    Posts
    1,891
    Ah, I see. Your goal is to have a mail archive (hence the importing and finding duplicates) to have your complete mail history and not to reduce storage of your backup archive. Then I've gotten your initial question wrong.

    Just theoretically speaking.. python-zarafa has the ability to export a mailbox as maildir (here someone already wrote a script around that functionality ), so you could

    - import one users bricklevel backup
    - export es maildir (linked script can be called with "zarafa-export.py -u username" to run only for a specific user)
    - unhook store of import user, create a new store for the user and delete the unhooked orphan (else your database will grow and grow and grow)
    - repeat step 1 with another backup for the same user
    - once your'e done (or even in between depending on the available space) you can use a file base duplicate finder to do the cleanup
    afterwards you could import that structure back into your new mail archive. The individual steps will take time, but in general that sounds easily scriptable.
    Regards Felix

    How to get Kopano

    Zarafa ALPHA/BETA/RC feedback in BETA forum please.
    Zarafa IRC chat: irc.freenode.com > #zarafa
    Zarafa documentation: http://documentation.zarafa.com/

    No support via PM! Please contact our sales team for an offer if you want my full attention.

Similar Threads

  1. Searching in Archives (attachments)
    By deisenlord in forum Installation, Configuration and Maintenance
    Replies: 2
    Last Post: 18-06-2013, 05:02 PM
  2. z-merge won't start
    By stony999 in forum Z-Merge Development Archives
    Replies: 4
    Last Post: 08-02-2013, 03:37 PM
  3. Z-merge archive
    By rhannink in forum Z-Merge Development Archives
    Replies: 1
    Last Post: 14-03-2012, 03:32 PM
  4. Z-Merge sync.php works only in /usr/lib/z-merge/tools
    By StefanK in forum Z-Merge Development Archives
    Replies: 0
    Last Post: 30-03-2011, 04:34 PM
  5. Z-merge
    By alex2010 in forum Z-Merge Development Archives
    Replies: 1
    Last Post: 13-05-2010, 02:37 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •