MirrorSync advanced topics
m (→MIRRORSYNC_CLIENTID for user-visible numbers)
m (→MIRRORSYNC_CLIENTID for user-visible numbers)
|Line 162:||Line 162:|
$$MIRRORSYNC_CLIENTID * 10000 + serialField copy
$$MIRRORSYNC_CLIENTID * 10000 + serialField copy
* For primary key configuration in MirrorSync, select Developer-managed.
* For primary key configuration in MirrorSync, select Developer-managed.
Revision as of 14:54, 22 January 2014
Welcome to the advanced section of the MirrorSync documentation! This guide will help you customize MirrorSync to suit your development needs, and provide detailed instructions for various situations. If you would like a good general overview of the software, please read through MirrorSync basic setup first!
How do I uninstall MirrorSync?
To uninstall on Mac, run the 'Mac Uninstaller.pkg' that comes with the MirrorSync. On Windows, run 'C:\Program Files\360Works\Uninstall.
Installation for hosting providers
To install multiple instances of MirrorSync, choose the Hosting provider option in the installer. This is exactly the same as the regular installation process, except that it will allow you to rename the instance of MirrorSync. You can continue running the installer as many times as you like, once per client, renaming each instance to something unique. These copies can then be managed via the 360Admin utility, which is found either in your Program Files or Applications folder. When installing additional instances of the application, only a single Tomcat process will be installed which is shared by all of the MirrorSync instances.
We recommend you use the installer. We at 360Works use the installer for our hosting clients as well, and find it easy to update and manage. If you are curious as to what the installer actually does, it modifies and adds the following:
- Creates a folder at '/Library/360Works' on Mac or 'C:\Program Files\360Works' on Windows that is readable and writeable.
- Downloads and installs an instance of Tomcat 6 into that folder. (Only one copy of Tomcat is installed, regardless of how many copies of MirrorSync are running)
- Copies and renames the installer's MirrorSync.war, which deploys the web app, as well as the other supporting material into the folder.
- Adds a launch daemon in Library/LaunchDaemons in OS X, or creates a Windows Service to automatically start Tomcat.
- Modifies the http.conf file for Apache to allow for URL redirection. If using Windows with IIS, we create an ISAPI filter or a URL Rewrite rule for the URL redirect.
- Copies a lightweight Admin Utility JAR file to manage Tomcat to either C:\Program Files\360Works or /Applications.
If upgrading from an older version of MirrorSync, we'll copy over the SyncData and remove the URL redirects for the old MirrorSync.
If you prefer MirrorSync to be deployed to your own instance of Tomcat that you've set up, please note and follow these instructions:
- Create a folder at '/Library/360Works' on Mac or 'C:\Program Files\360Works' on Windows. Make sure that it is readable and writeable to the process that Tomcat is running as. This is where MirrorSync stores its private data including the internal sync database and the stored configurations. If you have a strong preference for locating this someplace else, because of disk space for example, set the '360directory' system property to point to some other path.
- Rename the MirrorSync.war file to whatever name you'd like the instance to run as for your hosting customer, ie. 'CustomerXSync.war' If you decide to run multiple instances of MirrorSync with the same name (using multiple instances of Tomcat), you'll need to set '360directory' separately for each Tomcat instance, otherwise those multiple instances will overwrite each other's private data.
- Drop that .war file into the webapps directory in your Tomcat instance.
- Modify the the MirrorSync.xml context descriptor and set the administrative username and password for MirrorSync. In Tomcat 6, this file is automatically written to the Tomcat/conf/Catalina/localhost. If you are running Tomcat 7, the file is located in the webapps/MirrorSync/META-INF/context.xml file, or you can follow the instructions at http://tomcat.apache.org/tomcat-7.0-doc/config/host.html regarding copyXML to get the same behavior as Tomcat 6 (we recommend doing this, so that you don't lose the context.xml file every time the application is redeployed).
- If necessary for your configuration, set up URL forwarding from IIS / Apache to your Tomcat connectors. See tomcat documentation on how to do this.
Installing without a network connection
The MirrorSync installer normally needs a network connection during the installation process. It uses this to download a copy of Apache Tomcat 6 from the Apache web site, as well as doing a network license check to validate the license key. If you are in an environment that has no outbound network capabilities, you can still install MirrorSync, but you'll need to follow these extra steps:
- Contact email@example.com and ask for a customized build of MirrorSync that does not need a network connection for license checking. Be sure to include your license key and registration information.
- Manually create a new folder at /Library/360Works (on Mac) or C:\Program Files\360Works (on Windows).
- Download Apache 6 from http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.36/bin/apache-tomcat-6.0.36.zip and place it into the 360Works directory.
Now you should be able to run the installer without any network connection.
Split server deployments
If your FileMaker deployment is split onto multiple machines, we recommend installing MirrorSync on the FileMaker Server machine. This allows the download database feature in MirrorSync to work, which greatly simplifies deployment to end users. If you will not use the download database feature, and you do not plan on sending download URLs to your users, then install MirrorSync on the web server, for maximum speed during syncing.
Regardless of which computer you install on, be sure that when you are configuring MirrorSync, you specify that the MirrorSync / FM Server machines are on different computers. This option is on the first screen in the new configuration process.
Does MirrorSync work with runtime versions of FileMaker Pro?
MirrorSync is untested and unsupported for use in this configuration. In addition, the legal licensing agreement for creating runtime versions of FileMaker specifically disallows any automated transfer of data between the runtime version and FileMaker Server. This restriction means that no sync process can legally be used to transfer data between the runtime edition and FileMaker Server, whether that is from a 3rd party or a home-grown automation process. Contact your FileMaker Business Account Manager for volume pricing on FileMaker Pro licenses.
What if I need to redeploy FileMaker Server?
If using a Mac, no additional steps should be required. However, if you are running Windows, you'll want to rerun the installer for MirrorSync.
I am running MirrorSync 1. What do I need to do to upgrade to 2?
Any extra configurations or devices you purchased with MirrorSync 1 will carry over to MirrorSync 2 after purchasing the upgrade.
- First, have all of your offline users run a sync. They will need to replace their offline files with new versions, so this will prevent them from losing any data.
- MirrorSync 2 requires a creation timestamp field, in addition to the modification timestamp. Create that field if you do not already have one, and be sure to add it to all the MirrorSync layouts.
- Run the MirrorSync installer. MirrorSync 1 configurations will not be preserved, so you'll need to create new configurations and distribute new copies of the files to your users. The good news is that since you already have all of your sync layouts ready to go, and also since MirrorSync 2 automatically detects foreign keys (requires FileMaker Server 12 or later), this shouldn't take much time at all.
- Download new copies of the file for your users, and enjoy all the great features and speed of MirrorSync 2!
Can MirrorSync send encrypted data? What about SSL enabled FMS?
Yes, it can. To make sure data is encrypted, you'll need a valid SSL certificate installed on the web server where MirrorSync is running. This will ensure that all plain-text data is sent and received encrypted with SSL. To sync container data using SSL, first set up FileMaker server as SSL enabled (see page 20 of these docs). After doing so or if FileMaker Server is already is SSL enabled, check the SSL checkbox when configuring MirrorSync.
If you get an error saying that you have a self-signed certificate, then you will need to follow the Custom SSL Certificate setup instructions.
I need my database to be HIPPA compliant, what should I do with MirrorSync?
HIPPA compliance depends on a large number of requirements. However, one of those requirements is to transmit only encrypted data. See the question above if you experience difficulties. Please keep in mind that encryption is one piece of HIPPA compliance, so consult the Department of Health and Human Services for more information.
What ports are required for MirrorSync? Can I change them?
MirrorSync transmits container data on port 5003, which is the regular port for communication betwen FileMaker Pro and FileMaker Server. All other field types (text, number, date, time, timestamp) are transmitted using regular HTTP communication, which typically runs on port 80, or port 443 if SSL encryption is being used. These port numbers are set in your web server configuration (IIS on Windows or Apache on OS X), and can be customized to any value you prefer. A good way to test this is to test the FileMaker Web Publishing Engine - if the URL 'http://yourServer:somePort/fmi/xml/FMPXMLRESULT.xml?-dbnames' returns an XML list of database names, then it should work for MirrorSync as well.
Will MirrorSync work on a VPN?
Yes. MirrorSync runs over standard HTTP protocols (which are VPN compatible) for non-container data, and standard FileMaker protocols (which are also VPN compatible) for container data.
My solution is behind a firewall. Will MirrorSync still work?
MirrorSync communicates with FileMaker Server using a web viewer, so as long as the firewall allows standard HTTP traffic on port 80 (almost all firewalls are configured to allow this), all of your sync devices should be able to access it through the firewall. The exception to this is container fields. If you are syncing container fields, you will need to make sure that port 5003 (FileMaker's standard port) is open on the firewall, because the client needs to connect as a guest of the server to transfer container data.
Is MirrorSync compatible with Network Address Translation (NAT)?
Yes. When configuring MirrorSync, select the option that says 'My internal and external IP addresses are different.' This will write the sync script so that if it detects it's running on the same LAN as MirrorSync, it will use the internal IP address, and if it's running outside the MirrorSync LAN, it will automatically switch to the external IP address.
Primary key / serial numbers
What is the difference between 'MirrorSync-managed' and 'Developer-managed' primary keys? Do I need to change how I do my primary keys? Which one should I pick?
Before answering this question, it's first necessary to explain why primary keys are a complex issue with synchronization. With traditional incrementing serial numbers, one database might have records numbered 1 through 10. Another database might have records 1 through 50. If we create a new record on the first database, it will get assigned the next number in the sequence (11). However, if we try to write that record to the second database, it will conflict with record 11 that already exists there. Here are several approaches to solving that problem:
- Use Universally Unique Identifiers (UUIDs) as primary keys. These are typically long, 36-character strings that look like this: "D2EF9F69-5DEA-4FE3-9095-162C77F76FBF". They are sufficiently random to statistically eliminate the possibility of duplicate values. This makes them ideally for syncing databases - you can safely write records from one database to another without worrying about conflicting IDs. MirrorSync (and most other sync frameworks) supports UUIDs.
- Combine a traditional serial number with some delimiter, such as each user's initials or a file ID. This way, user 1 would have primary keys "1.1", "1.2", "1.3", and so on. User 2 would create primary keys "2.1", "2.2", "2.3", etc. This has the advantage of being shorter and more readable than UUIDs, but the added management consideration of assigning unique identifiers to each user. MirrorSync can manage this for you, as explained later.
- A variation on the second solution would be to assign each user a particular numeric range, so that user 1 generates primary keys in the range of 1-10,000; user 2 generates primary keys 10,001-20,000; and so on. This has the advantage of pure numeric values (instead of text), but the disadvantage of the possibility of conflicts if a user exceeds the expected number of records. MirrorSync can manage this approach as well.
- A very different approach is to allow conflicting primary keys to exist on each separate database, without ever writing those primary keys to other databases. When record #11 is written from the first database to the second (in our original example), instead of being written with primary key 11, it is written with primary 51 (the next number in the sequence on the second database). This has the advantage of the shortest possible primary keys, which are pure numeric values, with no possibility of conflicts. It is also the way that the majority of existing databases are designed. MirrorSync supports this method (and is the only sync framework that does, to our knowledge). It creates an internal table to translate between the primary keys on all database that are syncing, so that when record #11 is later updated, MirrorSync knows to change record #51 in the second database. MirrorSync also re-writes foreign keys when they are written from one database to another, so that foreign keys that contained '11' on the first database will be re-written with '51' in the second database.
Options 1, 2, and 3 are considered 'Developer-managed', because MirrorSync writes the primary keys unmodified between the databases being synced. It is the developer's responsibility to pick some scheme that ensures that the same primary key is never used for different records on different databases. There are other variations on the same theme (ie. one database gets odd numbers and the other gets even numbers), but they all are treated the same by MirrorSync.
Option 4 is considered 'MirrorSync-managed.' The developer is not responsible for the uniqueness of primary keys across different databases; MirrorSync takes care of this for you.
As to which you should pick, the answer is usually this: If your FileMaker database is currently using UUIDs, use developer-managed. If your database is written using traditional serial numbers, you should use MirrorSync-managed. However, if you need 'user-friendly' numbers for things like invoice numbers, job numbers, or check numbers, read the next FAQ on user-friendly serial numbers before making a decision.
If you are building a new database and can use whichever approach you want, here are the relative advantages:
- Serial numbers are more efficient because they are smaller. See the performance section of this documentation for more discussion on this.
- UUIDs are easier to use if you need to ever sync or import manually without using MirrorSync, or switch to some other sync tool (which only support UUIDs).
What if I need to assign a user-friendly serial number to my records that stays the same when it is synced? For example, an invoice number?
The problem with MirrorSync-managed serial numbers is that they are not suitable for user-visible numbers, such as invoice numbers. That's because the primary key will be different on one device than another. If you're using the primary key serial number as your invoice number, it is clearly a problem if your invoice number is different between your laptop and the server!
There are several solutions to this problem. One of the first things to establish is whether your database uses a single field as both the primary key and the user-visible value (we'll refer to it as the 'invoice number'). It is always preferable (even when not syncing) to have these be separate fields from each other. Invoice numbers / job numbers / user visible numbers should NOT be what you use as a primary key in your database. Primary keys are internal database identifiers, and should not do double-duty as a user-readable value.
If the fields are separate, the problem becomes simpler to solve, because you have the flexibility to change the value of the invoice number without breaking relationships. See the section below on write-back values for the recommended approach.
If the same field is being used for the primary key and the invoice number, and if it is feasible to split this into two separate fields (one for relationships, and one for display/searching), you should do so. Unfortunately, this can sometimes be a very big job - creating the new field is easy, but then you either need to find every place that that field is used in the user interface and point it to the newly created invoice number field, or else you need to find every place that it is used in the relationship graph and re-point that to the newly created primary key field. If you do not want to do this, then see the section below on MIRRORSYNC_CLIENTID for the recommended approach.
|One mistake you should be careful to avoid is this: If your database is currently designed with serial numbers for primary keys, do not try to short-cut the solution by simply creating a new UUID field and try to trick MirrorSync into using that as your primary key. MirrorSync won't re-number your foreign keys, which means that the wrong children records will point to the wrong parent records when the records are synced with the server.|
Write-back values for user-visible numbers
Let's say that you have a UUID as your primary key field, as well as an invoice number field. Using the write-back approach, the invoice number field will be blank when a record is created in an offline file. When that invoice record is synced to the server, it will get assigned the next invoice number, and that number will be written back to the invoice number field on the offline file. This approach does not work unless the invoice number field is separate from the primary key.
Advantages of this approach:
- Invoice numbers are in a simple, short numeric sequence, just like they would be without syncing.
- The offline file does not get an invoice number until the record is synced for the first time.
To use this approach, follow these steps:
- If you do not already have a serial number field in your table, create one. If you have a serial number which is used as a primary key, that's usable. We'll call that field 'serialNumber' in our example.
- Define your invoiceNumber field this way:
If( Get( MultiUserState ) = 2; serialNumber; "" ). This will leave the invoice number blank when working offline, but fill it in when connected directly to the server.
- In the primary key selection screen in MirrorSync, select invoiceNumber as the write-back field. This will cause MirrorSync to write invoiceNumber from the server back to the offline file when the record is first synced.
- Your choice of MirrorSync-managed or Developer-managed is unaffected by this setting; pick appropriately depending on whether you are using serial numbers or UUIDs as your primary key.
MIRRORSYNC_CLIENTID for user-visible numbers
Let's say that you have a single serial number as your primary key, which is doing double-duty as an invoice number field. You can use the $$MIRRORSYNC_CLIENTID global variable to help. Using this approach, MirrorSync ensures that each file being synced gets assigned a unique sequential number, starting from 1. You can use this number in conjunction with a traditional serial number to create a unique number than can be used as a user-visible field as well as a primary key. The serial numbers will either be text (ie. "1.1", "1.2", "1.3"…) or numeric (10,001 for user 1, 20,001 for user 2, 30,001 for user 3…).
Advantages of this approach:
- Reasonably short user-visible numbers
- IDs are assigned immediately upon record creation, without needing to wait for syncing
- Offline users must do an initial sync before they can create records.
- Requires a startup script to run, which means it won't work with custom web publishing applications.
- IDs are not sequential, so you can't assume that an ID with a higher value was created before or after another ID.
- If you are using numeric version, limits users to creating a fixed number of records before they start conflicting.
- If you are using text version, you must switch all primary key and foreign fields from 'number' type to 'text.'
- Setup is a bit more complex than write-back fields.
To use this approach, follow these steps:
- Set the 'MirrorSync setup' script as your startup script. If you already have a startup script, call the 'MirrorSync setup' script from your existing script. This will set the $$MIRRORSYNC_CLIENTID global variable. This clientId field is stored in the MirrorSync table, and is assigned when the user does their first sync. Each synced database will get a unique, sequential clientId. Manually run the script now to set that global variable before proceeding.
- We assume you have a serial field called 'serialField'. Start off by duplicating this field, to create a new field called 'serialField copy.'
- Change serialField to an auto-enter calc. Uncheck the box that says 'Do not replace existing value'. Also make sure that the 'Prohibit modification of value during data entry' is unchecked. Set the formula to one of the following:
For text keys, use this formula. This will result in primary keys that look like "1.1", "1.2", "1.3" from device 1, and "2.1", "2.2", "2.3" from device 2. Feel free to modify as desired. Be sure to change the field type from number to text. Also change all foreign keys that relate to it to text.
$$MIRRORSYNC_CLIENTID & "." & serialField copy
for numeric keys, use this formula. This will result in primary keys that look like "10001", "10002", "10003" from device 1, and "20001", "20002", "20003" from device 2. Replace whatever number you want instead of 10,000 to give a wider or narrower range of numbers.
$$MIRRORSYNC_CLIENTID * 10000 + serialField copy
- For primary key configuration in MirrorSync, select Developer-managed, and set 'serialField copy' as your primary key.
How do I configure and use two foreign keys as a primary key?
This is a common configuration for join tables in many-to-many relationships. In the primary key configuration screen, use the drop down menus to select the two foreign keys.
Please keep in mind primary keys need to be unique. If using two foreign keys, there may be an instance where an error occurs about a duplicate node ID, since FileMaker is not validating the fields together and making sure they are always unique in a table. To avoid this, make sure:
- Ensure that the combination of two foreign keys only ever occurs once in the database.
- If you need to be able to have multiple join table records with the same foreign keys, add a regular serial number field to the join table and use that as the primary key, instead of the compound foreign keys.
- The foreign key fields must both have the 'Not empty' validation in order to appear in the pull-down menu of eligible fields. If your join table needs to have foreign keys that may be empty, you should add a traditional single primary key to the table instead of using compound foreign keys as an identifier for MirrorSync.
Does MirrorSync need the databases being synced to be identical?
No, not in MirrorSync 2. MirrorSync is able to sync between databases with completely different tables and fields from each other. In addition, it can sync just a subset of records, fields, and tables. If you are syncing different databases, for example a dedicated mobile file with a much larger server database, select the option 'Sync with a separate mobile file' (for FileMaker Go) or 'Sync with a separate server file' (for server-to-server sync). With this option selected, MirrorSync will allow you to match up your layout and field names with a simple drag and drop interface.
Why does MirrorSync need to know about foreign keys?
MirrorSync uses foreign keys for two purposes. If you are using MirrorSync-managed primary keys, then it needs to know about the foreign keys because it needs to rewrite them when it writes between databases (see the primary key section above).
In addition, even if you're using developer-managed primary keys, MirrorSync needs to know about relationships so that it can sync parent tables before children tables. This allows validation rules that check referential integrity to work correctly.
MirrorSync 2 adds a new feature that automatically detects foreign keys, if you are using FileMaker 12 or later, so this step should not add any time to the configuration process. If you are using FileMaker 11 AND you are using developer-managed primary keys AND if you do not have any referential integrity validation making sure that foreign keys point to valid parent records, you can safely skip this step.
Make sure XML publishing is enabled for the account you are using in MirrorSync. Also be sure the database is accessible from the machine you are installing MirrorSync on.
To test the XML Web Publishing Engine, try going to the URL
http://yourServer/fmi/xml/FMPXMLRESULT.xml?-dbnames. You should see an XML document listing the databases in FileMaker Server. If you do not, or find a 404 or 401 error, then the XML Web Publishing Engine may not be configured correctly. Contact FileMaker tech support for help getting this running.
If there is an authentication required at that test URL, try disabling IIS authentication. FileMaker Server authenticates its password-protected databases; however, this method disables the IIS authentication layer. This will also disable authentication for other websites that are using IIS on that server.
- From the Control panel, navigate to Administrative Tools > Internet Information Services (IIS) Manager
- Select the website in IIS and choose Action > Properties
- Navigate to the Directory Security pane, and select Edit for authentications. This button may differ in various Windows versions.
- In the Authentications Methods pop up:
- Make sure Anonymous Access is enabled
- For Authenticated access, disable the authentication methods.
- Press OK
Configuration without FileMaker Pro Advanced
Since FileMaker Pro does not have the ability to copy and paste tables, you'll need to follow these instructions to create the MirrorSync table.
- Locate the XML schema that came with your MirrorSync download.
- Open FileMaker Pro and navigate to File -> Import Records -> XML Data Source.
- In the "Specify XML and XSL Options" window, choose the "File" radio button, then click "Specify...".
- Navigate to your MirrorSync download folder and choose the MirrorSync.xml file from the XML Schema subfolder.
- Click continue to bring up the Import Field Mapping window.
- Open the "Target" drop down menu, and select "New Table ("MirrorSync")"
- Verify that the source fields and target fields have been matched correctly.
- Click import.
After the table is created, you'll need to modify some of the fields:
- Navigate to File -> Manage -> Database...
- Navigate to the "Fields" tab, highlight the id field and click the "Options..." button.
- Check the box next to "Serial Number." Set it to generate on creation.
- Check the box next to "Prohibit modification of value during data entry."
- Navigate to the "Validation" tab and check the boxes next to "Not Empty" and "Unique Value" then press OK.
- Next, highlight the modstamp field and click the "Options..." button.
- Check the box next to "Modification" and ensure that the drop down menu says "Timestamp (Date and Time)."
- Check the box next to "Prohibit modification of value during data entry."
- Navigate to the "Validation" tab and check the box next to "Not Empty" then press OK
- Next, highlight the lastErrorMessage field and click the "Options..." button.
- Navigate to the "Storage" tab and check the box next to global storage and press OK.
- Repeat the last two steps to make webServerResponse, _gSync1, _gSync2, _gSync3, _gSync4, _gLastInsertTable, and _gLastInsertResult all be global fields.
- Finally, highlight the container field, and select "Container" from the "Type" drop down menu, next to the "Options..." button.
You are now finished manually configuring your MirrorSync table. Click the OK button to close the database management window, and return to the MirrorSync configuration. (Be sure to save your changes!)
Separation model / multiple file solutions
MirrorSync will work fine with multi-file solutions. You need to make sure that one of the files in your solution has a reference to every table that you want to sync (for separation model solutions, it is better to use the data file than the UI file, because MirrorSync stores some internal data in a new MirrorSync table that it creates in the file). When configuring MirrorSync, use this file. When using the MirrorSync download feature, be sure to select the multi-file option so that your offline users will receive all of the files for the solution.
How do I configure External SQL Source (ESS) tables?
MirrorSync supports ESS tables, but it's very slow compared to using MirrorSync to simply sync the database from the external SQL database into your FileMaker Server. We do not recommend ESS unless syncing is not an option for some reason.
If you would like to use ESS tables, there are additional steps. First, configure MirrorSync in its entirety. You should be able to select your ESS tables and files in the configuration utility. After you have pasted the last script step, download a copy of the file as usual.
At this point, you need to make some modifications for the file to work offline. Open the downloaded copy, and make the following modifications to the local copy ONLY! Do not run the sync yet.
- First, delete the data source. This is vital to preventing duplicate records and other artifacts of using ESS. Open File > Manage > External Data Sources and delete the OBDC data source.
- Next, open the File > Manage > Database and navigate to the Tables tab. Copy any ESS tables. Then delete them. Make sure NOT to delete the table occurrences. After all the italicized tables are gone, paste the tables back in. This ensures they are local FileMaker files, not references to external data.
- Make sure the primary keys, modification timestamps, and creation timestamps are correct. The primary key needs to be an auto-entered serial number or generated UUID.
- Open the relationships tab in the File > Manage > Database and relink the missing tables.
You can now distribute this file for synchronization to your users. Make a copy of this file and send it to each user who needs a copy. These copies are then ready for synchronization.
I configured my file in FM11 and everything worked great. I just converted it to FM12 and now it doesn't sync at all. What's going on?
After converting your file, you'll need to edit the configuration and replace the MirrorSync script steps with ones generated for the FileMaker 12 file. This is due to the fact that the "Insert from URL" script step doesn't exist in FileMaker Pro 11.
Does MirrorSync work with timezones?
Yes and No.
For offline users who are syncing, Yes. When MirrorSync gets a list of all modifications, it will automatically detect the offline user's timezone and clock drift and compensate for it.
For server-to-server sync, Yes. MirrorSync can detect FileMaker server's current time and compensate for it, just like it does for offline users.
For non-syncing users connecting directly to FileMaker Server, No, not automatically. You'll need to perform a small work around to make this work. It gets a little complicated, so an example makes it more clear. Let's say your FileMaker Server is hosted in New York. A user in New York runs a sync at 12:30 PM Eastern Time. 30 minutes later, a user in San Francisco modifies a record on the server at 10 AM Pacific Time (1 PM Eastern Time). Due to the way that FileMaker Server works, that modification timestamp on the server in New York will say 10 AM. It will not be converted to Eastern Time, nor will FileMaker Server store any time zone information for anybody to tell that it was actually representing Pacific time. Essentially, it's just an incorrect value. When the next sync runs, MirrorSync is going to request records modified since the last sync at 12:30 PM, which is going to miss the change made by the San Francisco user, even though it was made since the last sync.
The solution to this problem is very easy - it actually takes less time to do than it did to read that explanation! Let's say that your table contains a regular modification timestamp called 'Modification timestamp'. Create a new field called 'Host modification timestamp' (or whatever you prefer), and set it to auto-enter this calculated value (substituting your own field name in place of 'Modification timestamp'):
Let( x=Modification timestamp; Get ( CurrentHostTimeStamp ) )
Be SURE to uncheck the box titled 'Do not replace existing value of field (if any)'.
Don't forget to add it to your sync layout, and REMOVE the regular modification timestamp from the sync layout!
Use this field as the modification timestamp in the MirrorSync configuration process, instead of the regular modification timestamp.
This works by using the FileMaker Server clock, instead of the user's clock, for the modification timestamp. Now, when the San Francisco user modifies a record at 10 AM Pacific Time, FileMaker Server will store 1 PM (the server's local time when that change was made) in the Host modification timestamp field, and MirrorSync will work with time zones correctly.
Why can't I transfer my database to another computer after I've synced it?
The MirrorSync server stores information about each device that syncs with it. This includes the primary key, as well as the count of modifications for that record on that particular device. If this file were to be synced from two different devices, this information would conflict and cause problems. If you do try to send a database to another device and sync it, an error message will appear and the sync will be prevented.
If all users on that device log in with the same username, then it's probably OK, as long as you make sure that you are not using record level access privileges on the offline device that would grant different record access to different users.
If you want each user who shares a device to log in with different usernames, then the solution is to put a separate copy of the file on the device, one for each user. Make sure that each user is only able to login to the file that is assigned to them.
MirrorSync prevents multiple users with different usernames from sharing the same offline file, because if record level access privileges or script filters were configured differently for each user, that would lead to the sync process incorrectly deleting the wrong records on the server, which would be very bad.
Does MirrorSync sync container fields?
A:Yes, MirrorSync works with FileMaker container fields. Container fields do take longer than other field types to synchronize, so remove them from your sync layouts if you don't need them to be synced.
What about SuperContainer? Does MirrorSync work with that?
Yes, it does. However, with SuperContainer the approach is very different. Only the URLs are synchronized, not the actual files - they remain on the SuperContainer server. The advantage is syncing is very fast - there is no binary data being transferred. The disadvantage is that you'll only have access to the files stored in SuperContainer when you have working network access from your computer or iOS device.
Does MirrorSync sync external container fields?
Yes, MirrorSync works with external container fields. There are no extra steps to take when syncing with FileMaker Pro or FileMaker Server. If you are using external container fields on an iOS device, there are a few steps after configuration that are required.
- Download a copy of the database from MirrorSync or FileMaker Admin
- Open the copy on a computer, and navigate to File > Save a copy as...
- Under the type drop down, choose "self-contained copy (single file)"
This ensures the copy of the file embeds the containers into the file for easy use with FileMaker Go and offline functionality.
PLEASE NOTE: there is an unfortunate behavior in FileMaker in this process: when saving the self-contained copy, FileMaker updates the modification timestamp for every record that has an external stored container field. That's not a problem normally, but when used with MirrorSync, it will make the initial sync very slow. The reason for this slowdown is due to MirrorSync assuming that all of those records have been modified on the client, so it writes all of the server data to the client (including the container data).
The workaround for this is to uncheck the modification timestamp auto-entry before saving the self-contained copy, and then turn it back on after the copy has been saved. This modification timestamp behavior has been reported to FileMaker, Inc. and will hopefully be resolved soon.
Does MirrorSync require users to sync with full access accounts?
No. MirrorSync requires a full access account only once: to paste in the table, scripts, and layout. When users run the sync, they will use their own user account privileges, with whatever access that grants them. See the 'Customizing MirrorSync' section below for tips on restricting user accounts.
Does MirrorSync do conflict resolution?
Yes, MirrorSync has robust conflict detection and field-level merging (added in version 2). It will detect when the same record(s) are modified on both database and take action. There are two settings in the configuration client that will control how this works:
- Merge the changes together: If MirrorSync sees that all of the changes made in the databases were to different fields (ie. firstName on the server and lastName on the client), it will automatically merge the changes together. It will not tell the user that a conflict occurred, it will just write the merged record to both the server and the client. If any changes were made to the same field, it is treated as a conflict and not merged.
- Flag the edit as a conflict: MirrorSync will treat all modifications to the same record as a conflict, regardless of whether they were made to the same fields or different fields.
If a conflict occurs, then MirrorSync will resolve it based on the next configuration option:
- User picks: MirrorSync will present a web-based interface to the user, summarizing all conflicts and showing them exactly which fields were changed on both databases. It color-codes the actual sections of text that were changed, making it easy for even inexperienced users to make the best choice. The user is able to pick one entire record over another, pick individual fields from each record, and even manually edit the result to combine changes from both records. They also have the option to automatically select the most recent change (see next option).
- Most recent change wins: MirrorSync will pick whichever record was last modified as the winner of the conflict. If Joe changes a record on his iPad at 3:00 PM, Kate changes the same record on her iPhone at 3:30 PM, and Tom changes the record on the server at 4:00 PM, then when Joe or Kate sync their database, Tom's record will be selected as the winner, regardless of when Joe and Kate do their sync.
- Hub always wins: MirrorSync will always pick the value on the hub (typically FileMaker Server) as the conflict winner, regardless of other considerations. This is not generally recommended, because it is essentially first-sync wins behavior: Whoever syncs to the hub first will win conflicts with other offline users who modify the same record and sync. This setting can be useful if changes made by directly connected users should be given priority over offline, syncing users.
- Email administrator: This is very similar to 'User picks', except that instead of the user who is doing the sync deciding, an e-mail is sent to the administrator email address specified in the configuration. The administrator can resolve the conflict using the same web-based user interface described above, after which the user will be able to sync normally.
Is MirrorSync transactional? What happens if the connection is lost while syncing?
MirrorSync is not 'transactional' in the strictest sense. If the connection is dropped during a sync, it is possible for records from one table to be written to the server, while records from another related table may not be written. However, MirrorSync keeps track of which records have been written and which ones have not, and it will resume from where it was on the next sync and complete everything as if the first sync had finished. If a record on the server cannot be written because it is being edited, the offline user will get a warning error message telling them this. MirrorSync will continue to retry that edit operation each time the sync script is run.
Can I use external authentication with MirrorSync?
Yes, MirrorSync supports externally authenticated accounts with no modifications. So long as the username on the local copy matches the username on the server, you can set up one password for local access and another for external authentication with the server. The passwords do not need to match between the local file and the hosted file.
How do I set up server-to-server syncing?
For the most part, the process for setting up server-to-client sync is almost identical to server-to-server sync. Here are a few differences to be aware of:
- If you will be syncing container fields across a Wide Area Network (WAN), both servers must have public IP addresses, because they each make HTTP calls to the other.
- In the spoke configuration, you select 'copy or clone of existing file' or 'separate file.' If the first option is selected, you don't actually put the file on the other server during the setup process - go through the setup process, and when you're finished, save a copy or clone from the hub server using the MirrorSync download feature and upload it to the spoke server. If you select 'separate file', then you will need to have the file up and running on the spoke server during configuration.
How do I run a server-to-server sync?
You do not use the MirrorSync script from within FileMaker, like a server-to-client sync. Instead, use the 'sync' button in the MirrorSync admin utility. Once you've run the sync and are satisfied that everything is working correctly, you may enable auto-sync by checking that box and specifying the frequency to run the sync.
How can I be notified of a problem that occurs during server-to-server sync?
There is an administrative e-mail address that is set during MirrorSync configuration. If you set this, you will receive e-mail notifications whenever the sync fails (or succeeds, depending on what granularity you set admin e-mails to). These e-mails are sent via Amazon Web Services, and thus do not require you to configure an SMTP server.
How well does MirrorSync perform on slow networks?
MirrorSync is very efficient over slow networks, because it sends very little data other than the actual record changes, and it transmits this information in large chunks, which is more efficient. There are 3 HTTP request at the beginning of the sync, and usually 2 HTTP requests per table that contains any modifications. For example, if you have a 20 table solution, and there are changes in 5 of the tables, it will usually take 13 HTTP requests (3 + 2*5) to complete the sync. By comparison, the FileMaker home page takes 65 HTTP requests to load in a web browser.
My file is very large, and I want to give MirrorSync more memory allocation. How do I do this?
MirrorSync has a 512 megabyte allocation by default. This is typically sufficient for syncing change batches of around 200,000 records, although that number gets much smaller if the records contain a lot of information. Now divide that number by the number of devices simultaneously syncing - so if you expect up to 4 people to be syncing simultaneously, MirrorSync can probably handle up to 50,000 changes from each user with the default memory allocation. If you need more than that, you can increase the memory allocation by modifying the setenv file. Find the 512 and change it some new memory allocation, like 1024. The file is at:
Mac: /Library/360Works/Applications/bin/setenv.sh Win: C:\Program Files\360Works\Applications\bin\setenv.bat
What can I do to speed up syncing?
MirrorSync is pretty fast without any special tweaking, but faster is always better. Here are some tips for squeezing out the best possible sync speed from MirrorSync:
- SSD drives: MirrorSync does many thousands of tiny read and write operations when syncing large record batches. This type of usage will benefit greatly from the ultra-low latency of SSD drives. You will see a noticeable change in sync speed by installing MirrorSync on an SSD drive. You don't need to use an SSD for your entire hard drive - you can just put MirrorSync on the SSD by using a symbolic link in OS X or changing the install location on Windows. The SSD performance benefits will improve all sync operations, but it will be most noticeable when doing very small syncs with few or no changes, and initial syncs for large database with hundreds of thousands of records.
- When MirrorSync fetches changed records from FileMaker Server, it requests all changes since 10 minutes prior to the last sync. This 10 minute overlap compensates for users whose clocks are slow. If you're using the Host modification timestamp trick outlined in the timezone section above, then the user clock doesn't matter, and you don't need this 10 minute overlap. You can get rid of it by editing the MirrorSync.xml file and change the 'fmServerOverlapSeconds' parameter from 600 (10 minutes) to 5 (5 seconds). The file is at:
Mac: /Library/360Works/Applications/conf/Catalina/localhost/MirrorSync.xml Win: C:\Program Files\360Works\Applications\conf\Catalina\localhost\MirrorSync.xml
- Use serial numbers instead of UUIDs, especially for large record sets with narrow tables. The difference between a 5 character serial number and a 36 character UUID adds up, especially when transferring large numbers of records that mostly consist of foreign keys, such as join tables. Don't make this change if you have a good reason for using UUIDs.
- Container fields hold lots of data, require a guest connection to FileMaker Server over port 5003, and are sent individually instead of in batches like non-container data. If there are containers that you don't need to sync, remove them from the sync layouts. Removing all container fields will give much better sync performance, as well as eliminating firewall concerns, because MirrorSync does not need to connect as a guest of FileMaker Server.
- If you want to just sync a subset of records, instead of the entire database, you can use either record level access privileges or scripted filters (see the customization section below for more details). Scripted filters are much, much faster then record level access privilege restrictions.
- If you're on FileMaker 11, switch to 12 or later. All of MirrorSync's communication with FileMaker Server is via the XML Web Publishing Engine, and this was just not very fast in FileMaker Server 11. It is greatly improved in later versions of FileMaker Server.
- Flag records instead of deleting them: MirrorSync 2 is very fast at detecting deleted records. However, it's always faster to sync edits and inserts than to scan for deletions, especially in large tables with more than 100,000 records. If you want to optimize performance, it might make more sense to flag records as being deleted instead of actually deleting them. That makes it into an edit operation, which is not affected by the number of records in the table.
I don't want to sync my entire database to my offline users. Can I just sync certain tables, fields, or records?
That's a three-part question with a three-part answer. Just syncing certain tables is very easy: When you're configuring the sync using the MirrorSync configuration utility, only include layouts corresponding to the tables that you want to sync.
Syncing certain fields is also very easy: Just don't include any fields on your sync layouts except the ones that you want to sync.
Syncing certain records is only slightly less easy: Edit the 'MirrorSync Customization' script and follow the instructions in the 'FindChanges' documentation there to constrain the found set to just the records you want to sync for the current user. Remember that the script will be running as the user doing the sync, so you can use the Get( AccountName ) function to determine which records are available. Be careful regarding the use of $$globalVariables - they will work correctly for the duration of the current sync, but they will not still be set on the next sync (except for $$MIRRORSYNC_USERTOKEN - see next question).
Any records that are excluded by your search criteria will not be synced to the user. In addition, if those records were previously synced, they will be deleted from the user's device. This makes it very easy to create check-in / check-out workflows with MirrorSync, where jobs are synced to iPads, completed, and then deleted from the device.
To test your customization script, go to the sync layout that you want to test, show all records, and run the MirrorSync customization script. If you don't get the results you expect, use the Script Debugger to tweak it.
Another approach to limiting record access is with FileMaker's record level access restrictions. Any restrictions you place on the user's privilege set will be respected by MirrorSync, and the user will only receive those records when they do a sync. Records which were previously visible which switch to non-visible will be deleted on the next sync. Keep in mind that scripted record filters will be much faster than using record level access privileges.
If you decide to limit the synced records, you may choose to distribute an empty clone to your users instead of a full copy of the database. That way, the initial sync will only pull the records into their offline file that they have access to on the database. Keep in mind that if they have access to many thousands of records, that could make the initial sync slower than if you distributed a full copy of the database that already contains all of the record data, and let the initial sync delete the extra records.
How can I filter records on a per-user basis if everybody is logging in with the same Filemaker account?
Use the $$MIRRORSYNC_USERTOKEN to accomplish this. This is a special global variable that MirrorSync passes from the client to the server when a sync runs. You can set this global variable to whatever you want on the client, and it will be set on the server so that the MirrorSync customization script can use it when searching for records. This global variable is the one exception to the 'no global variables' rule for server-side filtering, so use it any way you want. You'll notice that it is nested in a conditional in the MirrorSync customization script that sets it. That 'If' statement prevents it from being set on the server, which is important because that would override the value sent by the client.
How can I tell if a record created in FileMaker Go has been synced with the server?
During configuration, select any field as a write-back value (see write-back values in the primary key section, above). Make sure that value is only set to auto-enter a value on the server, by checking that
Get(MultiuserState) = 2. If the field is blank, the record has not yet been synchronized. If the field contains a value, then it was either created on the server, or created in FileMaker Go and synced with the server.
Another approach you can take is by modifying the didInsert section of the customization script (see next question), but the write-back approach is significantly faster when syncing.
How can I run a script on the server when a record is synchronized?
Maybe you want to send an e-mail to an administrator when a user completes a delivery, creates a request for a quote, or deletes a job in production. To do these types of operations, modify the MirrorSync customization script. There is a section in the script for 'DidUpdate', 'DidInsert', and 'WillDelete'. Make sure you modify those in the 'Hub' section - it's very rare that these should be modified on the spoke. In the relevant section of the script, check the current layout, and if it's the one where you want to take action, go ahead and do whatever you want in this section. You can freely go to other layouts, do searches / inserts / updates / deletions, send e-mails, or take any other action, as long as you finish the script on the same layout and record where you started.
If MirrorSync detects that the record was modified as a result of a 'DidInsert' or 'DidUpdate' script, it will re-do the client search, taking into account any scripted record restrictions in the 'FindChanges' section. For example, if you want jobs to be removed from the user's iPad when a record on the 'SyncJob' layout has its status changed to 'Completed', take these steps:
- Modify the 'FindChanges' to exclude records whose status is 'Completed' when Get(LayoutName) = "SyncJob". (To do this type of search in FileMaker, enter Find mode, Set the status field to 'Completed', Omit the current record/request, and then Constrain Found Set)
- Modify the 'DidInsert' section to blank out the modification timestamp when Get(LayoutName) = "SyncJob".
- Do the same thing in the 'DidUpdate' section.
Now, whenever an offline user updates or inserts a record in the 'SyncJob' table, the script will blank out the modification timestamp. This triggers a change, which causes MirrorSync to re-do the search, which will exclude the complete job from the search, which will automatically remove it from the iPad.
You might ask, why blank out the modification timestamp? The answer is that it actually doesn't matter what you do, as long as you make some change to the record at all. Blanking out the modification timestamp is a harmless change, because FileMaker will immediately replace the blank value with a new timestamp. You could just as well set any field on the record to its current value, or make some more meaningful change, such as filling in a date field with the data that the job was completed.
I don't want users to see any dialogs when the sync runs, can I prevent them from showing?
Yes, set the global variable $$SILENT_MODE to "1" in the MirrorSync customization script. This will prevent MirrorSync from showing any dialogs unless an error occurs, and for required information like login and conflict resolution.
- If you would like to also skip the login dialog, AND if you know the user's password (ie. if it's stored in a users table) OR everybody is syncing with the same account, you can hard-code the $$MIRRORSYNC_USERNAME and $$MIRRORSYNC_PASSWORD values in the MirrorSync customization script. If these are both non-empty values, then MirrorSync will not prompt the user to enter their password.
- If you would like to prevent the conflict resolution dialog from displaying, pick any option other than 'user decides' in the conflict resolution section of the MirrorSync configuration process.
I don't want users to see the MirrorSync window at all when a sync runs. Can I prevent that from showing?
If you are running MirrorSync on FileMaker Pro, then yes. Set the $$MIRRORSYNC_HIDE_WINDOW variable to 1 in the MirrorSync customization script. This will move the MirrorSync window off-screen, where the user cannot see it. This is not recommended for large sync operations that will take longer than a few seconds, because without that status display, users may think that their computer is frozen.
If you are running MirrorSync on iOS, then this cannot be hidden. That is because MirrorSync needs to open a new window to manipulate the found set without losing the current selection, and new windows cannot be hidden offscreen on iOS devices. The $$MIRRORSYNC_HIDE_WINDOW variable will be ignored on iOS.
Can I block other MirrorSync dialogs from being displayed to the user, or change the wording?
Yes. Using the MirrorSync customization script, all dialogs, even error messages, can be suppressed or modified. Look towards the bottom of the MirrorSync customization script, and you will see a section that includes a dialog for each type of message that is displayed to users during a normal sync. You may freely customize the message displayed to the user, or the title of the buttons (but be sure that the first button still has the same meaning, and the second button still has the same meaning). If you want to hide the dialog from showing, you can disable the script step that displays it, but you must decide which option you would like to select, and exit the script with that button number. You can use this technique to display messages to your users in their preferred language.
Can MirrorSync run automatically when my users finishes a job/invoice/widget? Can it run at startup?
Yes, absolutely. First of all, we recommend setting the 'MirrorSync setup' script as your startup script, or calling it from the startup script. This will check to see if the file is running offline, and whether it has ever been synced before. If not, it will prompt the user to do the initial sync. In addition, the MirrorSync setup script sets the $$MIRRORSYNC_CLIENTID global variable, which is necessary for some primary key strategies (see the Primary Key section above).
Remember that the MirrorSync script is a regular FileMaker script that can be set to run at any time by the developer. For example, if there is a critical section of your application that requires users to see what the latest value is before they make a change, trigger the sync when the user navigates to that screen, and again when they are finished editing in that screen. You could make it so that checking a box, clicking a button, or any other condition that you choose can trigger the sync to happen.
Can MirrorSync run automatically every x seconds/minutes?
Yes, you can use an On Timer script to trigger this. However, keep in mind that when the script runs, it commits the current record and pops open a new window that could be open for a few seconds or longer. This could get annoying if it happens in the middle of something that the user is working on. For this reason, we recommend the approach in the previous question of linking your script sync operations to some user-initiated action. The exception to this is unattended computers - if you want to have a computer that is running MirrorSync all the time for purposes of having an extra copy of the database replicated, without a user working on it, then by all means go ahead and use an On Timer script step (and be sure to see the FAQ above about disabling dialogs).
Customizing the MirrorSync layout
Feel free to make cosmetic changes to the 'MirrorSync' layout that is pasted into your solution. However, follow these tips before making changes:
- Make sure you are able to successfully sync on FileMaker Pro and FileMaker Go before making customizations.
- Make a backup copy of the unmodified MirrorSync layout first.
- It is important that you preserve the tabs on the layout, and that the same fields remain in those tabs. The other thing to be careful about is that there are two 1x1 pixel web viewers on the MirrorSync layout. Make sure that those remain visible on the layout.
- If the sync stops working after you make the changes, then you may have inadvertently lost the web viewers on the layout. Restore them from the backup copy of the layout.
How does licensing and pricing work for MirrorSync?
MirrorSync is completely free to use, with no limitations on features, for syncing FileMaker Server with a single copy of FileMaker Pro or FileMaker Go. You can optionally purchase additional server configurations or devices.
The supported configuration types are:
- FileMaker Server with FileMaker clients (Go or Pro)
- FileMaker Server with FileMaker Server
- FileMaker Server with any SQL database (Oracle, MySQL, and MS SQL Server are fully supported. You may also sync with any database with a JDBC driver).
- SQL database with SQL database
- SQL database with FileMaker clients (Go or Pro)
Devices are sold in quantities of 1, 5, 10, 40, and unlimited. These can be purchased in addition to the single device that comes with MirrorSync for free.
Is MirrorSync included in the 360Works Portfolio Bundle?
Yes. MirrorSync 2 is free for one device, but Portfolio Bundle (http://360works.com/portfolio) subscribers will receive an additional 10 device pack (for a total of 11 devices). Server-to-server configurations are not included in the Portfolio License, and must be purchased separately. There is also a special licensing program for FileMaker Business Alliance (FBA) members; contact us at firstname.lastname@example.org for details.
Is there a vertical market license type for MirrorSync?
Yes. Contact email@example.com for details about how this works.
Is MirrorSync localized into any non-English languages? Does it work in other languages?
At this time, MirrorSync is only localized to English. Our current focus is on adding features, but we anticipate translating this into other languages when the product is more mature. If you would like to offer help with translation, please contact firstname.lastname@example.org to let us know. You should still be able to add MirrorSync to your solution in other languages. See the previous section on customizing dialogs for tips on localizing your solution.
I only have an offline copy of my file left, and need to upload it back to the server. What do I do?
If necessary, you can re-upload an offline copy of the file to FileMaker Server and use it with MirrorSync. Be sure to remove the "Client" record in the MirrorSync table, and keep only the "Server" record. To do this, navigate to the MirrorSync layout, switch to the Debugging tab, and perform a find for "Server" in the type field. Select the inverse of the records found by pressing the pie icon in the status bar. Then, delete those other records.
The maximum number of users are currently using this copy of FileMaker Pro
There is a bug in FileMaker Pro and FileMaker Pro Advanced that may occur when pasting the MirrorSync script, or editing the MirrorSync script. This occurs due to the references found in the script, which will reference both the internal and external IP address. FileMaker then opens both of those references, which triggers the licensing server as two copies of FileMaker Pro. This error would occur for any script that contains both references to a single hosted file.
There are two work arounds for this:
- Upgrade your single user license to a volume license. You can contact FileMaker directly to upgrade your license if you have 5 or more copies.
- Paste the script and let FileMaker close due the licensing error. The script will save appropriately, and should work correctly.
If you require assistance on this issue, please let us know.
I keep getting a 102 'Field is missing' error, but I know my Sync layouts have all the right fields, what's wrong with it?
Make sure that the sync layouts are matching in both the offline and server copy. If you have repeating fields, make sure the fields show the maximum number of repetitions that are defined in the field definitions. In other words, if a field can repeat 5 times, make sure the repetitions in the inspector show 5.
Sync fails every time with error: Last sync failed: java.sql.SQLException: Error in table <Sync Layout Name>: Parse Error During Update 1: -1
This issue arises if a syncing solution has more than 998 syncing fields. The limit for the number of variables Filemaker will allow in a Let statement is 1000. During our Server <-> Client transactions, we require two variables for internal use, so the maximum syncing field limit per table is 998.