MirrorSync

Overview

MirrorSync is a simple, elegant FileMaker synchronization product that connects FileMaker Server with FileMaker Pro and FileMaker Go. Users can sync the hosted file with their FMGo or Pro clients, go off network to make changes, and sync again when a connection is available.

Why use MirrorSync?

MirrorSync is designed to solve the problem of how to work productively on a hosted FileMaker database without being tied to FileMaker Server. If, for example, you have multiple users connecting to FileMaker Server over a wide area network (WAN) you could notice a substantial slowdown, as every change is immediately written back to FileMaker Server. MirrorSync allows each user to take a local copy of the database which would run on the users machine and not be affected by the WAN performance. The user could sync when they first came into the office, work on their local copy throughout the day, and sync at specified times (or on demand) to push and pull the latest data to the server.

Another use case would be users who work in the field and may have limited or no connectivity to the FileMaker server network. The user could take a local copy on their iPad or iPhone, input the information from the field, and then sync when they regain a network connection. If the connection is spotty, they could even specify a one-way sync where their information is pushed to the server, but no data is pulled down, thus speeding up the process.

Server Requirements

Windows 2003r2 or later, or Mac OS X 10.5 or later FileMaker Server 10, 11, or 12.

Configuration utility requirements

OS X 10.6 or later, or Windows 2007 with Java 10.6 or higher installed FileMaker Pro 11 or 12 (FileMaker Advanced is recommended and reduces setup steps, but not required) This is only required for one machine on the network

Sync Client Requirements

Mac or Windows with FileMaker Pro 11 or 12 OR FileMaker Go running on iPad, iPhone, or iPod Touch

Important MirrorSync Information

Before moving further in MirrorSync, there are a few important points you need to know about. MirrorSync is a web application that is installed on the FileMaker Web Publishing Engine, and accessed on client machines through a FileMaker Web Viewer. To configure and run MirrorSync, follow the instructions in the configuration utility to modify the hosted copy of the database you wish to sync. User will then download local copies of the hosted database and run the MirrorSync script to sync the local copy to their devices and the server.

Q: So I've downloaded my local database to my laptop and want to work on my iPad. Can I just email a copy of the database to myself?

A: Yes, as long as you have not run the sync script in the database. Once the sync script is run for the first time, the database is linked to your device's MAC address. Using a previously synced file on devices with different MAC addresses will result in errors. To get a copy of the file that is not linked to any device, simply navigate to your MirrorSync server address, login to the main screen, and choose "Download Database". You'll download a new copy that you can then email to other devices.

Q: I was so excited about MirrorSync that I downloaded a local copy of my database and immediately started making changes. But when I synched, all of my changes disappeared. What gives?

A: It is very important that when you first download your local copy of MirrorSync that you run the sync script before making any changes. This links the local copy to your device and ensures that you have the most recent information from the hosted database. During the initial sync, information from the hosted database takes precedence, so any changes you've made will be reverted.

Q: What happens if two devices try to sync simultaneously?

A: Devices cannot sync simultaneously. If a user gets a "Sync in Progress" message, they will need to try again in a few seconds to allow the first sync to finish.

MirrorSync With Hosting Providers

As of 1.25, MirrorSync supports hosting provider installations of MirrorSync.

The following companies have told us that they are set up or willing to configure their servers to host MirrorSync:

• John May, Point in Space ( http://www.pointinspace.com/ )

For instructions on how to install MirrorSync for hosting providers, please see the MirrorSync for Hosting Providers page.

MirrorSync Quick Start Tutorial: Synching the Demo File

MirrorSync comes with a fully featured demo. You may use the demo for your own files, or walk through a sample configuration below. The only demo limitation is that once you have created an offline copy of the database, that particular copy can only sync for 24 hours. You can renew this 24 hour window at any time by downloading a new offline copy from the server.

MirrorSync ships with two example files: "Contact Management.fp7" and "Contact Management.fp12", along with a MirrorConfig.txt file. These files are copies of the Contact Management Starter Solution from FileMaker Pro 11. The only changes that have been made are the addition of the table generated by the MirrorSync Configuration Utility. To use the demo file: 1) Host the file on FileMaker Server 3) Run the appropriate Mac Installer.app or Windows Installer.jar 4) On the client machine, navigate to the URL provided at the end of the installation to download the MirrorSync utility. Be sure to match the capitalization of the word 'MirrorSync' in the URL. 5) Run the configuration utility using the username and password you created during installation. From the starting screen, click import and choose the configuration file (MirrorConfig.txt) from your download.

http://demo.360works.com/wiki/ImportConfiguration.png

6) With the new configuration selected, click "Edit" from the main screen and proceed through the configuration utility. If using a three-machine FileMaker Server deployment, be sure to enter the host name of the web server. Otherwise, use 'localhost'. Be sure the "Contact Management" database that you hosted on FileMaker Server is selected, then click "Next" through the next six configuration screens. On the final screen, copy the scripts, layouts, and steps as indicated.

http://demo.360works.com/wiki/ScriptStepChange.png

7) Download the file and run the MirrorSync script. This is a local copy of the FileMaker file and is linked with your current machine during the initial sync. To sync this file with multiple devices, follow the steps below for a custom configuration.

MirrorSync Setup Tutorial: Custom Configurations

In this tutorial, we'll walk through a MirrorSync configuration from start to finish, using the Contact Management.fp7 FileMaker Starter Solution. If you are using FileMaker 12 and would like to follow along, you can download a converted file at http://mirrorsync.360works.com/static/Contact%20Management.zip

1) Prepare the file. This involves: - enabling the Access via XML Web Publishing extended privilege set in File > Manage > Security - ensuring that the database is contained in a single file - ensuring that each table has a primary key field, either with an auto-entered, unique serial number or UUID - ensuring that each table has a modification timestamp field, also set to auto-enter - creating a new "sync" layout for each base table (not table occurrence). The layout should only contain those fields that should be synched. This means no global fields, summary fields, or unstored calculations. While this is technically an optional step, it is highly recommended.

2) Host the file on your FileMaker Server and run the MirrorSync installer. If you have a multiple machine deployment, run the installer on the machine where the web publishing engine is running. Make note of the URL at the end of the installation process.

3) On your client machine, navigate to the MirrorSync URL and launch the MirrorSync Client. Follow the instructions in the configuration utility. This involves setting the primary keys and relationships, choosing the layouts to sync, confirming the fields, and inserting the table, script, and layout that MirrorSync generates. If you get stuck, look for the blue question marks - they have helpful tooltips.

4) During the final screen of the configuration, MirrorSync will generate a table, script, layout, and script steps for synching. Copy and paste these into the hosted database (if you don't have FileMaker Pro Advanced, you'll need to import the XML for the table). Once these changes are made, click finished.

5) From the MirrorSync Dashboard, choose the correct configuration and click "Download Database". This will download a copy of the database to your local machine. Keep in mind that after the initial sync this copy is tied to your device. If you would like to have the database on multiple devices, you'll need to navigate to the server address on that device, launch the client, and download the database directly, or download a fresh copy of the database and email it to the device.

6) Before making any local changes, run the MirrorSync script.

Syncing

To sync changes to the server copy, simply run the MirrorSync script. This script is attached to the button on the MirrorSync layout, but users may choose to attach it to whatever is required. Be sure to sync once before making changes to assure that both copies are identical.

Please note: if the database has multiple access accounts with different privilege sets, make a new copy for each user. Do not sync an offline copy of the database to the server using multiple accounts.

Main Screen

http://demo.360works.com/wiki/mainscreen.png

After configuration, users can view information about current sync configurations. The left window displays all the current sync configurations. Users may Add new sync configurations, edit existing ones, duplicate, or delete them. If a sync with MirrorSync has been set up on another server, that configuration may be imported using the Import button.

If a sync configuration needs to be changed, there are a couple of things to keep in mind. Choose Edit… from the Main Screen and walk through the configuration again. After making the changes required, be sure to delete everything in the MirrorSync script first and then paste in the new script steps. After walking through the configuration utility, make a fresh copy of the database and run the MirrorSync script again.

The right window displays information about the currently selected sync and the option to run the sync script.

The bottom window displays the log for MirrorSync.

FAQ

Deployment and installation questions

Q: How should MirrorSync be installed?

A: For one- or two-machine deployments where the web server and the Web Publishing Engine are on the same computer, run either the Mac Installer.app or the WindowsInstaller.exe to install the MirrorSync application on the server. If you are using a multiple machine deployment, make sure to install the application on the server that has the Web Publishing Engine running. For three-machine deployments where the web server is on a different computer than the Web Publishing Engine, see the installation instructions for three-machine deployment below.

After installation is finished, a message will be displayed pointing to the address where MirrorSync can be configured. Typically the address will display as http://servername/MirrorSync, where servername is replaced with the address of the server. Please keep in mind that the address requires the correct capitalization. From a client computer, navigate to that address to launch the configuration utility.

Q: How do I upgrade MirrorSync to a new version?

To upgrade to a new version, run the new install file (Mac Installer.app or WindowsInstaller.exe) and select Upgrade.

Q: Are plug-ins required?

A: No, the only software to install anywhere is the MirrorSync server application, which is installed on the FileMaker Web Publishing Engine. You can send a copy of the FileMaker database to anybody with a copy of FileMaker Pro, and they will be able to run the sync script.

Q: I had to restart the Web Publishing Engine on FileMaker Server and now the MirrorSync configuration utility is not working! What do I do?

A: Any time you restart the WPE, you should also restart the MirrorSync Configuration Utility. Navigate to your MirrorSync server address, launch the client, and proceed through the steps.

Q: Does MirrorSync work on SSL enabled FMS?

A: Yes. For SSL support, choose the appropriate checkbox when selecting a database in the configuration utility. If you get an error saying that you have a self-signed certificate, then you will need to follow the Custom SSL Certificate setup instructions.

Q: My solution is behind a firewall. Will MirrorSync still work?

A: MirrorSync communicates with FileMaker Server using a web viewer, so as long as the firewall allows standard HTTP traffic on port 80 (most firewalls are configured to allow this), all of your sync devices should be able to access it through the firewall.

Q: Can MirrorSync sync between two FileMaker Servers?

A: No, MirrorSync depends on web viewers and scripts steps that are only available in regular FileMaker Pro. You can sync between FileMaker Pro and FileMaker Server, but not between FileMaker Server and FileMaker Server. If this feature is important for you, please e-mail us and let us know: plugins@360works.com

Q: Does MirrorSync work with runtime versions of FileMaker Pro?

A: No, MirrorSync is not supported in this configuration and may not work correctly.

Q: I have a three-machine FileMaker Server deployment. How do I install MirrorSync?

A: You'll need to perform a manual installation:

• Modify the Tomcat forwarding configuration, so that Apache or IIS will pass MirrorSync requests through to Tomcat. If you have your web server (Apache/IIS) on a separate machine than your Web Publishing Engine, this step needs to be done on the web server machine.
  • On OS X running FileMaker 11, modify the file located at '/Library/FileMaker Server/Admin/admin-helper/WEB-INF/conf/mod_jk.conf'. Copy and paste the following lines onto the end of the file:

JkFmMount /MirrorSync cwpe
JkFmMount /MirrorSync/* cwpe

• On OS X running FileMaker 12, modify the file located at '/Library/FileMaker Server/Admin/admin-helper/WEB-INF/conf/mod_proxy.conf'. Copy and paste the following lines onto the end of the file:

ProxyPass /MirrorSync ajp://127.0.0.1:16021/MirrorSync
ProxyPassReverse /MirrorSync ajp://127.0.0.1:16021/MirrorSync

• On Windows, modify the file located at 'C:\Program Files\FileMaker\FileMaker Server\Admin\admin-helper\WEB-INF\conf\uriworkermap.properties'. If you are running FileMaker 11, copy and paste the following lines onto the end of the file:

/MirrorSync=cwpe
/MirrorSync/*=cwpe

For FileMaker 12, copy and paste the following lines onto the end of the file:

/MirrorSync=jwpc
/MirrorSync/*=jwpc

The rest of the steps are done on the computer where the FileMaker Web Publishing Engine is running.

• If you are deploying to FileMaker Server 12, copy the file at 'Installer data/logging.properties' into the directory at 'FileMaker Server/Web Publishing/publishing-engine/jwpc-tomcat/conf'

• Copy the file at 'Installer data/MirrorSync.war' into the FileMaker Tomcat web applications directory. For FileMaker 11, this is 'FileMaker Server/Web Publishing/publishing-engine/cwpe-tomcat/bin'. For FileMaker 12, this is 'FileMaker Server/Web Publishing/publishing-engine/jwpc-tomcat'. Within 20 seconds, you should see a folder that is automatically created in this directory called 'MirrorSync'.

• Once you see the 'MirrorSync' folder, the application is deployed. You now need to set an administration username and password. The username and password can be whatever you want - they do not need to match your FileMaker Server Admin password, or your operating system login password. You will need to open the MirrorSync.xml file in a text editor. For FileMaker 11, this is located at 'FileMaker Server/Web Publishing/publishing-engine/cwpe-tomcat/conf/Catalina/localhost/MirrorSync.xml'. For FileMaker 12, the path is 'FileMaker Server/Web Publishing/publishing-engine/jwpc-tomcat/conf/Catalina/localhost/MirrorSync.xml'. Set a value for the the 'adminUsername' parameter. You can either enter the password in plain text format for the 'adminPassword' parameter, or you can do an MD5 encoded version of the password and put it into the 'adminPasswordHash' parameter. To generate an MD5 encoded hash, use one of these methods:

• On OS X, you can type 'md5 -s "yourPassword"' on the command line
• On OS X or Windows, you can use the web site at http://www.adamek.biz/md5-generator.php to generate the MD5 hash.

• Restart IIS if on a Windows platform, or Apache if on OS X.

• After completing these steps, go to http://yourServer/MirrorSync in your browser to configure synchronization.

Q: Does MirrorSync support ESS tables?

A: Yes! To use MirrorSync with ESS tables, there are additional steps. First, configure MirrorSync in its entirety. You should be able to select your ESS tables and files in the configuration utility. After you have pasted the last script step, download a copy of the file as usual.

At this point, you need to make some modifications for the file to work offline. Open the downloaded copy, and make the following modifications to the local copy ONLY! Do not run the sync yet.

• First, delete the data source. This is vital to preventing duplicate records and other artifacts of using ESS. Open File > Manage > External Data Sources and delete the OBDC data source.
• Next, open the File > Manage > Database and navigate to the Tables tab. Copy any ESS tables. Then delete them. Make sure NOT to delete the table occurrences. After all the italicized tables are gone, paste the tables back in. This ensures they are local FileMaker files, not references to external data.
• Make sure the primary keys and modification timestamps are correct. The primary key needs to be an auto-entered serial number or generated UUID.
• Open the relationships tab in the File > Manage > Database and relink any missing tables.

You can now distribute this file for synchronization to your users. Make a copy of this file and send it to each user who needs a copy. These copies are then ready for synchronization.

Q: How do I uninstall MirrorSync?

A: To uninstall, first stop the Web Publishing Engine. After that, run the original install file (Mac Installer.app or WindowsInstaller.exe) and select Uninstall.

Configuration questions

Q: What do I need to do to prepare my database for MirrorSync?

A: MirrorSync only supports sync with a single-file database, so if you have multiple files, create a new table with external table references to the other files.

Be sure that XML extended privilege is enabled in FileMaker Pro. Choose this option under File > Manage > Security… Double click the FileMaker user, and choose Edit… from the Privilege Set.

http://demo.360works.com/wiki/image002.png

Make sure the Access via XML Web Publishing option is enabled and has a checkmark.

Most tables that you would like to synchronize need to have a unique identifier, also known as a primary key. This can either be a sequential serial number, a universally unique identifier (UUID) generated with the new FileMaker 12 Get(UUID) function, or any other value that is guaranteed to be unique, non-empty, and unchanging for every record. If you are using UUID primary keys, it is important that you do NOT check the box prohibiting modification of these values. The exception for this requirement is many-to-many join tables, which often do not have a primary key, and instead have two foreign keys linking them to primary keys in other tables. It is OK if these tables do not have primary keys.

Every table (including join tables) must have a timestamp field which is set to auto-enter a modification timestamp.

It is not a requirement, but we highly recommend that you create a new layout for each table that you plan on synchronizing. Be sure to include the modification timestamp, primary key, and any fields that you want to synchronize. Be sure to NOT include summary fields, global fields, and unstored calculations. These will make synchronization slower. These are easy to spot, because if you enable the 'View->Show->Quick Find' menu in FileMaker, those field types will either have a yellow magnifying glass, or no magnifying glass. Remove those fields from the layout, and keep the ones with the green magnifying glass.

http://demo.360works.com/wiki/greenglass.png

MirrorSync checks with the initial sync which fields are read or writeable. If you have options selected that may prohibit writing to a field, MirrorSync may not be able to sync those fields appropriately. These options include "Prohibit modification of value during data entry" and "Validated by Calculation" under the options section for a field.

Host the database on FileMaker Server. Open the file as a guest with FileMaker Pro Advanced, using a full access account.

Q: I tried to follow the configuration utility, but don't quite understand the process. Do you have a walkthrough somewhere?

A: There is a MirrorSync configuration walkthrough page that explains this in more detail.

Q: How do I configure synchronization for solutions with multiple database files?

A: MirrorSync will work fine with multi-file solutions. Whether you're using the separation model, migrated from an older version of FileMaker, or just prefer to split our your tables into separate files, we've got you covered. You will need to select one file as your main file. Make sure that the main file contains table occurrences from the other database files that you want to sync, and when the configuration screen instructs you to create layouts for each table, be sure to create the layouts in the main file.

When you complete the setup process and get to the step where you click the button to download an offline copy of the database from your server, select the option for a multi-file solution. This will show a checkboxes of all of the database files available on the server. Be sure to select all the database files in your solution.

Q: What happens if I edit my configuration, or make changes to the sync layouts on my hosted database?

A: If you change your configuration or edit your hosted database in a way that affects the sync layouts, then your need to go back through the configuration and re-generate the sync script steps. Delete the current contents of the MirrorSync script and replace with the steps you just downloaded. Then send a copy of this new file to each offline user to replace their database.

Q: I configured my file in FM11 and everything worked great. I just converted it to FM12 and now it doesn't sync at all. What's going on?

A: After converting your file, you'll need to edit the configuration and replace the MirrorSync script steps with ones generated for the FileMaker 12 file. This is due to the fact that the "Insert from URL" script step doesn't exist in FileMaker Pro 11.

Also, don't forget to re-run the MirrorSync installer on the FileMaker Server 12 machine or the machine where Web Publishing Engine is running.

Q: My external IP addresses are different than my internal IP addresses. What do I need to do?

A: Use your internal IP addresses for configuration and testing. When you are ready to deploy this to users who will access it from outside your network, modify your startup script (or any script that runs before your MirrorSync script) and set the variable $$MIRRORSYNC_URL to "http://putYourExternalAddressHere/MirrorSync/sync".

Q: Is FileMaker Pro Advanced completely necessary for configuration?

A: If you do not have a license for FileMaker Pro Advanced, you'll need to import the table data from the XML schema that came with your MirrorSync download.

In your FileMaker Database, go to File > Import Records > XML Data Source. In the "Specify XML Data Source" window, choose File then "Specify...". Navigate to your MirrorSync download folder and choose the MirrorSync.xml in the XML Schema subfolder. Click continue to bring up the import field mapping window. In the "Target" drop down, select "New Table" (MirrorSync). Verify that the source fields and target fields have been matched correctly, and click import.

Next you need to change some options and settings on the fields you just imported. Open the Manage Database dialog and make the following modifications:

1. In the fields tab, set "id" to Auto-Enter Serial Number generated on creation. Choose "Prohibit Modification" and on the Validation tab choose "Not Empty" and "Unique Value".
2. Select the "modstamp" field, select "Modification: Timestamp", "Prohibit Modification" and "Not Empty".
3. Change the "container" field to be a container field.

Close out of the field modification windows and carry on with the MirrorSync configuration.

Q: I never saw the foreign key mapping screen! What do I do?

A: If you selected UUIDs as your primary keys at the beginning of the configuration, the foreign key screens will not appear. This is normal and expected behavior. When using serial numbers, primary keys can be (and usually are) re-numbered when they are transferred to another device. MirrorSync needs to know which foreign keys which point to the primary keys, in order to write the new primary key value to each foreign key. When using UUIDs, the primary keys stay the same on every device because there is no possibility of conflict. Therefore, there is never a need to re-write foreign keys.

Sync questions

Q: Why can't I transfer my database to another computer after I've synced it?

A: The MirrorSync server stores information about each device that syncs with it. This includes the primary key, as well as the count of modifications for that record on that particular device. If this file were to be synced from two different devices, this information would conflict and cause problems. If you do try to send a database to another device and sync it, an error message will appear and the sync will be prevented.

Q: Can I log into my local copy with different accounts/different privilege sets?

A: No, you'll need to download a new copy of the database for each of the user accounts you use to the server using multiple accounts with the same file may generate errors.

Q: Does MirrorSync sync container fields?

A:Yes, MirrorSync works with FileMaker container fields as of version 1.1. If container fields are slowing down your synchronization, consider moving containers to a separate table and create new layouts for them. MirrorSync scans for modification timestamps to find records that have been changed, so by moving containers to a separate table with a separate timestamp, they are only uploaded and downloaded when containers change.

Q: What about SuperContainer? Does MirrorSync work with that?

A: Yes, it does. However, with SuperContainer the approach is very different. Only the URLs are synchronized, not the actual files - they remain on the SuperContainer server. The advantage is syncing is very fast - there is no binary data being transferred. The disadvantage is that you'll only have access to the files stored in SuperContainer when you have working network access from your computer or iOS device.

Q: Does MirrorSync do conflict resolution?

A:MirrorSync 1.2 introduced a choice of manual conflict resolution or automatic conflict resolution. If a user syncs their changes and a conflict arises, the user has an option of selecting either. Selecting manual conflict resolution will bring up an interface that highlights differences in the server and client copies. For each conflict, users can select which version of the record will win and MirrorSync will use that record. MirrorSync does not 'merge' changes - it will always pick one entire record as the winner of the conflict.

Previous versions of MirrorSync, and automatic conflict resolution keep track of which change was made last, and selects that as the winner in a conflict. So if a record is changed at 2PM on one device, and 3PM on another device, the 3PM change will win the conflict. The exception is that changes always win over deletions, regardless of which one happened last. A common misconception is that conflict resolution is based on who syncs first or last - MirrorSync will pick the record that was EDITED last, not the one that was SYNCED last. Again, MirrorSync does not 'merge' changes - it will always pick one entire record as the winner of the conflict.

Q: Is MirrorSync transactional? What happens if the connection is lost while syncing?

A: MirrorSync is not 'transactional' in the strictest sense. If the connection is dropped during a sync, it is possible for records from one table to be written to the server, while records from another related table may not be written. However, MirrorSync keeps track of which records have been written and which ones have not, and it will resume from where it was on the next sync and complete everything as if the first sync had finished. If a record on the server cannot be written because it is being edited, the offline user will get a warning error message telling them this. MirrorSync will continue to retry that edit operation each time the sync script is run.

Q: Can I use external authentication with MirrorSync?

A: Yes, MirrorSync supports externally authenticated accounts with no modifications. So long as the username on the local copy matches the username on the server, you can set up one password for local access and another for external authentication with the server. The passwords do not need to match between the local file and the hosted file.

Q: How do I make changes to my hosted database if I have offline users who are syncing? Can I add fields and tables?

A: MirrorSync will match your fields by name whenever you sync. What this means is that as long as you don't rename or delete fields, you can freely add additional tables and fields to your hosted database without causing problems for your users working with the older databases running offline. Don't forget to add new fields to your sync layouts. To distribute your updated version of the database to users, re-run the configuration process and select any new fields and tables to add to the sync configuration. Be sure to re-copy your script steps for the MirrorSync script. Then download a copy of the database from the server and send it to your users to get them the newest version. Users running the old version will be able to sync until they are ready to download the newest version of the file.

Primary key / serial numbers

Q: I'm running FileMaker 12, and my primary keys are UUIDs. When I try to sync I get errors. What's going on?

A: Make sure that if you are using UUIDs as primary keys, the "Prohibit Modification of these Values" box is unchecked in the specify field window. MirrorSync must be able to write to these fields.

Q: Do I need to change how my primary keys are generated?

A: No. MirrorSync does not require primary keys to be unique across devices. It will take care of keeping track of primary and foreign keys that are re-numbered when they are transferred to other devices.

Q: After a couple of syncs I noticed that the serial numbers on my records don't match on my iPad and my laptop. Is this a problem?

A: No, serial numbers are expected to be different between devices, because when a record is written from a mobile device to the server, another record may already exist with that same serial number. MirrorSync keeps track of which serial numbers are assigned to which records. For example, if the contact record 'Jesse Barnum' is ID #5 on your laptop, and the same record is #12 on the server, and the same record is #15 on your iPad, Then when a change to record #5 is written from your laptop to the server, MirrorSync knows to make the change to record #12. When you download the change on your iPad, MirrorSync knows to change record #15 on the iPad. It also updates all foreign keys that link to the primary key (this is why the foreign key mapping step is important in the setup process).

Q: What if I need to assign a user-friendly serial number to my records that stays the same when it is synced? For example, an invoice number?

As the previous FAQ points out, if you are using serial numbers for your primary keys, MirrorSync will renumber these serial numbers when it writes to another device to prevent number conflicts. However, this can be a problem for things like invoice numbers, that need to always be the same for all devices. This isn't really MirrorSync's "fault", because obviously if the devices are off-line from each other, there is no way to know what the next available serial number is if it might have been used by another off-line device.

Before talking about solutions to this problem, it's important to emphasize that this user-friendly number should NOT be what you use as a primary key in your database. Primary keys are internal database identifiers, and should not do double-duty as a user-readable value, so even if you are not using MirrorSync, you should have two fields in your table: One hidden field used as a primary key that the user never sees, and then a second user-visible serial number that users can refer to. With that in mind, here are two potential solutions to the problem:

1) Have a separate database file that is hosted on the server (not off-line) with a table that contains serial numbers. When you want to generate the next serial number, go to that online table, create a new record, and use that next serial number for your invoice number (or whatever user-visible field this is). Since this database is online, everybody shares it and nobody gets the same serial number. The downside of this is that you won't be able to create new records unless you have a network connection to the server, but the upside is that you can still get all of the speed benefits of working on your own offline copy of the database (since only the serial number table is hosted on the server).

2) Either use a prefix for each device (so that records created on machine 'ABC' would be numbered ABC1, ABC2, ABC3, while records created on machine 'DEF' would be numbered DEF1, DEF2, DEF3), or assign a serial number range to each device (so records created on 'ABC' would be numbered 10,001, 10,002, 10,003, and records from DEF would be numbered 20,001, 20,002, 20,003). The downside to this approach is that your user-visible number will be longer, and also that you need to customize the database slightly for each device (either by changing the next serial number or by setting the prefix).

If you'd like our assistance doing the integration and development for this, we'd be happy to help - please contact us for an estimate.

Q: Does the sync run any faster if you use UUIDs rather than serial numbers?

A: Although we have not done benchmarking on these configurations to compare them, it is unlikely that this would substantially affect the speed.

Performance question

Q: How fast is MirrorSync?

A: From a speed standpoint, there are three types of operations that MirrorSync does: 1) Initial sync, 2) syncing inserts and edits, and 3) syncing deletions. #1 and #3 are affected by the number of records in your database, but #2 is not.

For each sync, there is an overhead associated with sync that is typically 1/4 - 1/2 of a second per table on a local network. Therefore, if you sync 20 tables, expect about 5 or 10 seconds in addition to the actual time to sync the records. For iOS devices, this is more like 2-3 seconds per table.

For syncing inserts and edits, we find that 20 records per second going from server to client is a fairly typical benchmark. Therefore, if 100 records have been edited on the server, it should take about 5 seconds to sync these records to the client. This number will be higher or lower depending on how much information is stored in your records, and is more like 4 or 5 records per second for iOS devices.

Syncing inserts and edits from the client to the server is about 5 records per second for FileMaker Server 11, and 40 records per second for FileMaker Server 12.

Detecting deletions takes about 1 second per 1,000 records in your table. This can be time consuming if you have a large number of records. See the performance tips section below for tips on improving this.

Initial sync takes about the twice as long as deletion - 2 seconds per 1,000 records in the table. This only happens the first time that a user syncs with an offline copy of the database.

Q: How well does MirrorSync perform on slow networks?

A: MirrorSync is very efficient over slow networks, because it sends very little data other than the actual record changes, and it transmits this information in large chunks, which is more efficient. There are 2 HTTP request at the beginning of the sync, and usually 2 HTTP requests per table that contains any modifications. For example, if you have a 20 table solution, and there are only changes in 5 of the tables, it will usually take 12 HTTP requests (2 + 2*5) to complete the sync. By comparison, the FileMaker home page takes 65 HTTP requests to load in a web browser.

Performance / speed optimization tips

Limit fields on the sync layout

It's a good idea to remove unnecessary fields on your sync layouts. Unstored calculations, summary fields, and related fields can cause significant slowdowns and are not synced anyway, so they should definitely be removed from the layout. Global fields and stored calculations do not affect speed as much, but they are not used for sync either so it's best to remove them if speed is important.

Move container fields into their own separate table

When MirrorSync checks if a record has been modified, it looks at the modification timestamp. If a record has been changed, it downloads the entire record, including container fields, every time. By moving container fields off the regular sync layouts, MirrorSync will only download and upload containers when they have actually changed. To implement this optimization, create a new table that is used only to store the container fields. Put your containers fields in that table, and add a primary key and modification timestamp. Be sure to import all the actual container data from the old table to the new one. Then, take the containers off your current sync layouts (if you have already configured MirrorSync), and create a new layout for the container table. Be sure to re-run the MirrorSync configuration utility so that it knows about the new layout.

Create layouts with just the ID and the modification timestamp

FileMaker 11 transmits data differently than FileMaker 12 to MirrorSync. This technique doesn't have any noticeable effect with FileMaker 12, but it may make the initial synchronization as well as deletion scanning much faster for FileMaker 11 tables with more than 1,000 records. For each Sync layout, create a new layout with the suffix _small. For example, you may have a table called Contacts. When you set up MirrorSync, you'd make SyncContacts. Now create another layout called SyncContacts_small. On this layout, place the primary key and modification timestamp ONLY. MirrorSync will automatically detect this layout and quickly scan for modification timestamps and records to add and delete. There is no need to rerun the configuration utility or script steps with this option. 

Flag records instead of deleting them

If you want to optimize performance here, it might make more sense to flag records as being deleted instead of actually deleting them. That makes it into an edit operation, which is not affected by the number of records in the table.

Customizing MirrorSync

Q: I don't want to sync my entire database to my offline users. Can I just sync certain tables, fields, or records?

A: That's a three-part question with a three-part answer. Just syncing certain tables is very easy: When you're configuring the sync using the MirrorSync configuration utility, only include layouts corresponding to the tables that you want to sync.

Syncing certain fields is also very easy: Just don't include any fields on your sync layouts except the ones that you want to sync. You can also simply uncheck unwanted fields in the MirrorSync configuration utility.

Syncing certain records is only slightly less easy: Utilize the record level access privileges in FileMaker's security management dialog to create custom restrictions. For example, you can easily restrict each user to only see their own records by creating a calculated restriction that checks to see if Get(AccountName) is equal to a field with an Account Name auto-entry option. With a little creativity, you can set up similar restrictions to only sync records with certain statuses, or which fall in certain date ranges.

If you decide to use this technique to limit records based on access level privileges, you may choose to distribute an empty clone to your users instead of a full copy of the database. That way, the initial sync will only pull the records into their offline file that they have access privileges to on the database. Keep in mind that if they have access to many thousands of records, that could make the initial sync slower than if you distributed a full copy of the database that already contains all of the record data.

Q: I don't want users to have to click on the dialog that comes up at the end of the sync, can I hide that?

A: Yes, set the global variable $$SILENT_MODE to "1" in your startup script. This will prevent MirrorSync from showing a dialog unless an error occurs.

Q: Can MirrorSync run automatically when my users finishes a job/invoice/widget? Can it run at startup?

A: Yes, absolutely. Remember that the MirrorSync script is a regular FileMaker script that can be set to run at any time by the developer. For example, if there is a critical section of your application that you want to avoid conflicts, trigger the sync when the user navigates to that screen, and again when they are finished editing in that screen. You could make it so that checking a box, clicking a button, or any other condition that you choose can trigger the sync to happen. If you'd like to incorporate MirrorSync into your startup script, but you only want it to run the first time that user opens an offline file, you can switch to the MirrorSync table and do a search for any record that has a value in the 'lastClientToken' field. If there is at least one record matching the search, then this file has been synced before.

Q: Can MirrorSync run automatically every x seconds/minutes?

A: Yes, you can use an On Timer script to trigger this. However, keep in mind that when the script runs, it commits the current record and pops open a new window that could be open for a few seconds or longer. This could get annoying if it happens in the middle of something that the user is working on. For this reason, we recommend the approach in the previous FAQ entry of linking your script sync operations to some user-initiated action. The exception to this is unattended computers - if you want to have a computer that is running MirrorSync all the time for purposes of having an extra copy of the database replicated, without a user working on it, then by all means go ahead and use an On Timer script step (and be sure to see the FAQ above about disabling dialogs).

Q: I have users in different time zones. Does MirrorSync account for that?

A: Yes. When an offline client syncs with the server, one of the first pieces of information it sends to the client is the client's current time. The server compares this to its own clock to see how different the client is (whether from being in a different time zone or just the client's clock being wrong), and then it applies this offset amount to all the timestamps it receives from the client for writing modification timestamps and resolving conflicts.

For online users connecting directly to the server, this offset cannot be calculated. Therefore, it is possible that online users in a different time zone, or whose clock is incorrect by more than 10 minutes (MirrorSync automatically includes a 10 minute "overlap"), could have their modifications missed by the sync process. To solve this problem, we recommend adding a new field, called something like 'Host modification timestamp', in addition to a regular modification timestamp field. Set the field type to 'Timestamp', and configure this auto-entered calculation:

Evaluate ( "Get ( CurrentHostTimeStamp )"; Modification timestamp ) //Change modification timestamp to be whatever the name of your modification timestamp field is

Be SURE to uncheck the box titled 'Do not replace existing value of field (if any)'.

Repeat this step for every table. This Host modification timestamp should now be configured as the modification timestamp for MirrorSync.

Licensing

Q: Is MirrorSync included in the 360Works Portfolio Bundle?

A: Yes, the Portfolio Bundle (http://360works.com/portfolio) includes MirrorSync. This is the same license as if it were purchased separately - up to 10 devices and 2 FileMaker solutions.

Q: From a licensing standpoint, what is a 'solution'?

A: From a business standpoint, a solution is one or more FileMaker database files that are integrated together for the purpose of meeting a particular business need. From a technical standpoint, there must be a single FileMaker file that contains external table occurrences in its relationship graph pointing to tables stored in other FileMaker data files. To be considered a single solution for purposes of MirrorSync licensing, a file must satisfy both the business and technical requirement.

Q: Is there an unlimited device license for MirrorSync? What about for vertical market solutions?

A: No, we do not offer an unlimited device license for MirrorSync. The same applies for vertical market solutions. For vertical market applications, we recommend that you pre-integrate your solution with MirrorSync, using the demo version of MirrorSync. Then tell your customers that if they would like to enable the offline sync feature, they can purchase MirrorSync separately, with no development work required. All that is needed is to install the MirrorSync software on their server, enter the license key using the admin utility, and customize the FileMaker sync script with the IP address of their MirrorSync server (this can be stored in a settings table, so that it doesn't have to be customized for each client / device).

Miscellaneous questions

Q: Is MirrorSync localized into any non-English languages? Does it work in other languages?

A: At this time, MirrorSync is only localized to English. Our current focus is on adding features, but we anticipate translating this into other languages when the product is more mature. If you would like to offer help with translation, please contact plugins@360works.com to let us know. You should still be able to add MirrorSync to your solution in other languages, although we have discovered that copying and pasting the script steps does not work with Japanese languages active - you must switch to English for this step of the configuration, and then you can switch back. We are investigating solutions for this problem, as well as checking to see if the problem happens in other languages besides Japanese.

More Help Sources

If you still need help, there are several resources available! FMForums hosts a support forum for MirrorSync. Please look through the posts and see if your question has been posted before making a new topic.

Support is also available via email at plugins@360works.com. You can also call us at 770-234-9293.

We're confident you'll be able to configure MirrorSync quickly and easily. However, if you want to have us integrate MirrorSync your solution for you, it will take approximately an hour at our rate of $165. We'll take a look at your files first, and if the integration will take more than an hour we'll contact you. Contact us to learn more and get started!