Welcome to the advanced section of the MirrorSync 7 documentation! This guide will help you customize MirrorSync to suit your development needs, and provide detailed instructions for various situations. If you would like a good general overview of the software, please read through the basic setup documentation first!
MirrorSync uses the word 'hub' to refer to your central database that everybody syncs with. MirrorSync supports many different types of hub databases, including FileMaker Server, Salesforce, Oracle, MySQL, SQL Server, or any other JDBC database.
A 'spoke' is another database that syncs with the hub. A spoke could be a copy of the hub database running on a laptop, iPhone, or iPad with FileMaker Pro or Go. It could also be another database server.
In addition to the supported hub database types, MirrorSync supports spoke servers running DynamoDB and AWS RedShift. MirrorSync even supports FileMaker spokes with a non-FileMaker hub: You could have a fleet of iPads syncing with a MySQL, PostgreSQL, or Salesforce database, for instance. See supported configuration types for a full list of supported types.
For any configuration, there is exactly one hub, and one or more spokes. If you're setting up a server-to-server sync between two servers, and you're not sure which one to use as the hub, ask "If I were going to delete one of these and replace it with a copy of the other?" The one that you'd keep is the hub, and the one you'd delete is the spoke. The hub database is where the data starts when configuring MirrorSync, and the spoke database(s) will be set up to match the hub during the initial sync.
If you encounter any problems during syncing, PLEASE use the 'send problem report' link on the MirrorSync launch page (you can access this using the same URL that you use in your config client, e.g. http(s)://yourServer/MirrorSync). This is much more useful than emailing us. If you do email us, we will probably ask you to re-send the report using that link, because that sends us the detailed sync log file, which is necessary to solve most tech support issues.
The spoke database can be any of the above hub types, as well as:
FileMaker Pro/Go
DynamoDB
Here are a few examples of supported configurations:
FileMaker Server (hub) syncing with FileMaker Pro or Go (spoke)
FileMaker Server syncing with one or more FileMaker Servers
MySQL / Oracle / SQL Server / JDBC database / Salesforce / RedShift (hub) syncing with FileMaker Pro or Go (spoke). FileMaker Server is needed during sync setup, but not for actual syncing
FileMaker Server / MySQL / Oracle / SQL Server / JDBC database / Salesforce / RedShift syncing with one or more database servers, including DynamoDB
MirrorSync does NOT support syncing FileMaker Pro/Go directly with other FileMaker Pro/Go devices without using some database such as FileMaker Server as the hub. It also does not support different types of spoke databases in a single configuration, such as syncing a FileMaker Server hub with a MySQL spoke and an Oracle spoke. This can be done by setting up multiple configurations that share the same hub database.
If you need to sync with a database or API that is not on this list, contact us and we'll get you a quote to support it. We've created custom implementations for state governments that use MirrorSync to sync with Web XML API's. We've also synced FileMaker Server with iTunes libraries for music companies. We're always open to a new challenge!
Does MirrorSync work with runtime versions of FileMaker Pro?
MirrorSync is untested and unsupported for use in this configuration. In addition, the legal licensing agreement for creating runtime versions of FileMaker specifically disallows any automated transfer of data between the runtime version and FileMaker Server. This restriction means that no sync process can legally be used to transfer data between the runtime edition and FileMaker Server, whether that is from a 3rd party or a home-grown automation process. Contact your FileMaker Business Account Manager for volume pricing on FileMaker Pro licenses.
Yes, it does. However, with SuperContainer the approach is very different. You will need to have a field on your sync layout that contains the SuperContainer URL that the web viewer is pointing to. That way only the URLs are synchronized, not the actual files - they remain on the SuperContainer server. The advantage is that syncing is very fast - there is no binary data being transferred. The disadvantage is that you'll only have access to the files stored in SuperContainer when you have working network access from your computer or iOS device.
To install multiple instances of MirrorSync, choose the Hosting provider option in the installer. This is exactly the same as the regular installation process, except that it will allow you to rename the instance of MirrorSync. You can continue running the installer as many times as you like, once per client, renaming each instance to something unique. These copies can then be managed via the 360Admin utility, which is found either in your Program Files or Applications folder. When installing additional instances of the application, only a single Tomcat process will be installed which is shared by all the MirrorSync instances. Log files are separated by the MirrorSync application name. This is important for hosting providers to be able to see each customer's log files separately, and makes troubleshooting with 360Works easier.
Hosting providers who do not provide FMS Admin credentials to their customers should read the next item 'MirrorSync configuration without FMS admin credentials'
Hosting providers may choose to share a dedicated computer running MirrorSync with multiple hosting customers. Be sure to read the 'split server deployments' section below. We recommend customizing your server.xml file and setting the startStopThreads attribute to at least as many instances as you're hosting, to make Tomcat start and stop much faster. Here's an example:
For installation on Mac or Windows, we recommend you use the installer with the 'hosting provider' option selected.
If you prefer to skip the installer and manually deploy using your own instance of Tomcat that you've set up, read these notes:
Create a folder at /Library/360Works on Mac, C:\Program Files\360Works on Windows, or /var/lib/360works on Linux. Make sure that it is readable and writeable to the process that Tomcat is running as. This is where MirrorSync stores its private data including the internal sync database and the stored configurations. You can use the '360directory' system property in Tomcat to point to some other path.
If you're running a single instance of Tomcat shared by multiple customers (which is how we host MirrorSync for our customers), rename the MirrorSync.war file to whatever name you'd like the instance to run as for your hosting customer, i.e. MyCustomerSync.war If you decide to run multiple instances of Tomcat, you'll need to set '360directory' separately for each Tomcat instance, otherwise those multiple instances will overwrite each other's private data.
Drop that .war file into the webapps directory in your Tomcat instance.
Modify the MirrorSync.xml context descriptor and set the administrative username and password for MirrorSync. The file is located in your Tomcat home directory, in the Calina/locahost/MirrorSync.xml. The .xml filename will match the .war filename.
If necessary for your configuration, set up URL forwarding from IIS / Apache / nginx to your Tomcat connectors, using ProxyPass or something similar. See the Tomcat documentation on how to do this.
Regardless of whether you use the installer or not, if you install enough instances of MirrorSync, you may start to run out of memory. If the memory error messages you get talk about PermGenSpace, then the memory problem is related to loading the MirrorSync classes, not the memory used during actual sync operations. In this case, modify the setenv file for your MirrorSync version (see the file locations under Increasing MirrorSync memory allocation) and add this to the list of launch arguments:
You can run the Tomcat Linux installer without any user interaction. This is useful for those who wish to automate the installation process. See the Silent Installer for details.
To uninstall on Mac, run the 'Mac Uninstaller.pkg' that comes with the MirrorSync. On Windows, run C:\Program Files\360Works\Uninstall.exe.
On either platform, you can also run the '360Works Admin.jar' application (/Applications/360Works Admin.jar or C:\Program Files\360Works Admin.jar) and remove the MirrorSync instance. However, this does not remove the 360Works service, which is shared by other 360Works applications such as Zulu. You must use the uninstaller to completely remove the 360Works service.
We will implement a Linux uninstaller in the future, but currently the user must manually remove the MirrorSync files and directories, and Tomcat if desired.
MirrorSync configuration without FMS admin credentials
In the MirrorSync.xml file, customize the fmsAdminUsername and fmsAdminPassword properties. Insert the full administrative user account information here, and MirrorSync will use that when downloading files or generating download links without prompting the users. In this way, your MirrorSync users can download files without the person setting up MirrorSync ever needing full FileMaker Server admin credentials.
Many people run MirrorSync on their FileMaker Server. However, you can run MirrorSync on its own separate server, which can reduce load on your database server. If you run MirrorSync on a separate server, then it is also recommended that you install MirrorSync on the FileMaker Server. This will enable download links to work. However, you shouldn't enter a license key on non-syncing copies of MirrorSync, just on your syncing server.
Be sure that when you are configuring MirrorSync, you specify that MirrorSync and the Database Server are on different computers. This option is on the first screen in the configuration process.
If using a Mac, no additional steps should be required. However, if you are running Windows or Linux, you'll want to re-run the installer for MirrorSync after redeploying FileMaker Server.
I am running an older version of MirrorSync. What do I need to do to upgrade to the latest version?
Be sure to read the 'IMPORTANT update notes' file that comes with the MirrorSync download. If there are specific migration issues you need to know about for the version you are installing, they will be listed there.
MirrorSync is compatible with older spoke files - no need to reset sync data or recreate configurations (although some new features, like password saving, will not function until the script is updated).
Although you can use older spoke copies with newer versions of MirrorSync, we highly recommend getting your users a new spoke copy with the latest script version as soon as it is convenient.
Updating MirrorSync only requires you to double-click the installer and follow the instructions. You do not need to uninstall the previous version. In fact, if you do, you will lose any customizations you have made to the MirrorSync.xml file.
The MirrorSync installer normally needs a network connection during the installation process. It uses this to download a copy of Apache Tomcat from the Apache website, as well as doing a network license check to validate the license key. If you are in an environment that has no outbound network capabilities, you can still install MirrorSync, but you'll need to follow these extra steps:
Contact support@360works.com and ask for a customized build of MirrorSync that does not need a network connection for license checking. Be sure to include your license key and registration information.
Manually create a new folder at /Library/360Works (on Mac) or C:\Program Files\360Works (on Windows).
Now you should be able to run the installer without any network connection.
How do I migrate MirrorSync from one server to another?
Moving from one platform to another (e.g. Mac to Windows, or Windows to Linux) is untested.
Start by determining which scenario you fall into:
My MirrorSync address is a DNS name (not an IP address which will need to change), and my hub database server is either running on the same computer, or is also a DNS name.
My MirrorSync address is an IP address which will need to change, AND I'm running a client-to-server sync.
My hub database server is an IP address which will need to change, and I'm running a server-to-server sync.
If you're in scenario #1, you're in luck. The steps are pretty simple:
Stop the 360Works Service on the old server by using the utility at C:\Program Files\360Works\ or /Applications/360Work Admin. Click Stop at the top of the window to stop the service. If you're on Linux, stop the Tomcat service.
Install MirrorSync on the new computer.
Once the installation is complete on the new computer, stop the 360Works service as described in step 1.
Copy the sync data folder on the old server from C:\Program Files\360Works\SyncData4_MirrorSync, /Library/360Works/SyncData4_MirrorSync, or /var/lib/360works to the same location on the new server, overwrite the existing folder. Make sure you have matched the file permissions from the previous computer.
Start MirrorSync back up.
Once the new instance is up and running and syncing, you can delete your old installation on the old server
For scenario #2, you're out of luck. You can export and import the configuration, so you don't have to create the configuration. However, none of your spoke users will be able to sync until you edit the MirrorSync configuration, modify your MirrorSync script as outlined in the configuration process, and distribute new spoke copies to your users. Next time use a hostname instead of an IP address.
For scenario #3, the steps are similar to #1, but you'll need to add a folder re-name step:
Between steps 4 and 5, look in the sync data directory for a subdirectory with the old IP address of your server. Re-name the folder to the new IP address of your server.
If something happens to your hub database, you might want to replace it with a spoke copy (either from a client or a server in a server-to-server sync) that has the same data as the hub.
There are two basic approaches: 1) Reset the sync data, or 2) Don't reset the sync data.
For either option, start with these steps:
Make sure the spoke database is as up-to-date as possible. If possible, run one final sync to bring any recent changes from the hub to the spoke.
Make sure you have backup copies of both the hub and spoke database.
Stop any auto-sync that is currently running
Info:
The following instructions reference different types of MirrorSync records. This refers to the value of the MirrorSync::type field; you can view this value in the "Internal" tab of the MirrorSync layout.
These instructions will also reference the sync4 field in the MirrorSync table. You can see where to find both of these values in the following screenshot:
MirrorSync Layout Internal Tab
Resetting the sync data:
This is the most reliable option. If there are MirrorSync-managed keys in any table, this is the ONLY option. If there are multiple spokes, they should all be replaced after this.
Here are the steps to take:
Delete the 'hub' and 'client' records from the spoke MirrorSync table.
Move the spoke database to the hub. If there are external containers, it should be OK to skip them.
Reset the sync data (by right-clicking the name of the configuration)
Run initial sync - there shouldn't be a dialog asking what to do about records in the spoke. If there is, you may still have a 'client' record in the spoke MirrorSync table, indicating that you either forgot to delete it before copying over from the hub, or you didn't copy over from the hub.
After that finishes, run a few incremental syncs, then enable auto-sync if appropriate.
Not resetting the sync data:
This may be an OK option if it's a big problem to distribute new spoke copies to users. It requires all tables to be developer-managed. One big downside of this approach is that if the record order does not match between the old hub database and the replacement spoke database, all deletion scans will be "slow" instead of "fast" until the sync data is reset.
Move the spoke database to the hub. If there are external containers, it should be OK to skip them.
On the hub copy only, delete the 'client' record from MirrorSync table
Run a sync to test that all is working. If so, enable auto-sync if appropriate.
Integrating MirrorSync with External Volumes on macOS
Mac OS X Catalina and later implement more restrictive security measures when it comes to running Java applications. A possible scenario where these security measures may inhibit MirrorSync is if you want to utilize the feature to automatically deploy updated versions of the hub database to the spoke server in a FileMaker Server-to-FileMaker Server sync, and the databases are hosted on a separate volume from the boot drive.
To allow MirrorSync to communicate with external volumes on the Mac server where it is installed, you will need to enable Full Disk Access permissions to Java.
While on the Mac server where MirrorSync is installed, perform the following tasks:
Open System Preferences
Click on the Security & Privacy module
Navigate to the Privacy tab
Find and select 'Full Disk Access' from the category menu on the left
Click the lock and enter your user password to authorize system changes
Check the box next to 'java'
This will let the OS know to allow Java applications, such as MirrorSync, access to external volumes. See the image for a visual representation:
Can MirrorSync send encrypted data? What about SSL enabled FMS?
Yes, it can. There are two network channels for encryption: The connection from the syncing client to MirrorSync, and the connection from MirrorSync to FileMaker Server. SSL encryption can be enabled or disabled for each channel separately. In most cases, the connection from the syncing client to MirrorSync is more important for encrypting, since the connection between MirrorSync and FileMaker Server is usually on the same computer or the same LAN.
To encrypt data between the syncing client and MirrorSync, make sure that you have a valid SSL connection installed on the web server where MirrorSync is installed. You can test this by going to https://yourServerAddress/MirrorSync. Once that is working, make sure that the $$MIRRORSYNC_URL in the MirrorSync script is set to use https://, instead of http://
Data between MirrorSync and FileMaker Server is encrypted by default, unless they are installed on the same server, in which case encryption adds no security. If MirrorSync and FileMaker Server are on separate servers, and you'd prefer to not use encryption, manually change the FileMaker Server host address to have an http:// at the beginning.
I need my database to be HIPAA-compliant, what should I do with MirrorSync?
HIPAA compliance depends on a large number of requirements. However, one of those requirements is to transmit only encrypted data. See the question above if you experience difficulties. Please keep in mind that encryption is one piece of HIPAA compliance, so consult the Department of Health and Human Services for more information.
What ports are required for MirrorSync? Can I change them?
Only port 443 is required to use MirrorSync with FileMaker Server.
Yes. MirrorSync runs over standard HTTP protocols (which are VPN compatible).
Is MirrorSync compatible with Network Address Translation (NAT)?
Yes. When configuring MirrorSync, select the option that says 'My internal and external IP addresses are different.' This will write the sync script so that if it detects it's running on the same LAN as MirrorSync, it will use the internal IP address, and if it's running outside the MirrorSync LAN, it will automatically switch to the external IP address.
Sometimes, it is impossible for the MirrorSync script to tell whether it is on the same LAN as the MirrorSync server. For instance, if MirrorSync is configured on a network where the MirrorSync server is running at IP address 192.168.1.5, and then a user tries to sync from some other network with the same IP address range (192.168.x.x and 10.1.1.x are common IP address ranges), the MirrorSync script will think that it's on the same LAN as the server, and will try the internal IP address, which will fail. If this happens, it will try for 60 seconds and then automatically switch to the external network. If the external network succeeds, this will be remembered for as long as the file is open, so future syncs will try the external network first.
If you know that the user is on an external network (for example, if you present a dialog asking the user whether they are inside the LAN or not), you can force MirrorSync to try external before internal or vice-versa. To do this, set the global variable $$MIRRORSYNC_CONNECTION to the word 'internal' or 'external' before running the MirrorSync script. This will allow you to override the normal search order.
MirrorSync-managed vs. Developer-managed primary keys
Before answering this question, it's first necessary to explain why primary keys are a complex issue with synchronization. With traditional incrementing serial numbers, one database might have records numbered 1 through 10. Another database might have records 1 through 50. If we create a new record on the first database, it will get assigned the next number in the sequence (11). However, if we try to write that record to the second database, it will conflict with record 11 that already exists there. Here are several approaches to solving that problem:
Use Universally Unique Identifiers (UUIDs) as primary keys. These are typically long, 36-character strings that look like this: "D2EF9F69-5DEA-4FE3-9095-162C77F76FBF". They are sufficiently random to statistically eliminate the possibility of duplicate values. This makes them ideal for syncing databases - you can safely write records from one database to another without worrying about conflicting IDs. MirrorSync (and most other sync frameworks) supports UUIDs.
Combine a traditional serial number with some delimiter, such as each user's initials or a file ID. This way, user 1 would have primary keys "1.1", "1.2", "1.3", and so on. User 2 would create primary keys "2.1", "2.2", "2.3", etc. This has the advantage of being shorter and more readable than UUIDs, but the added management consideration of assigning unique identifiers to each user. MirrorSync can help to configure this for you, as explained later.
A variation on the second solution would be to assign each user a particular numeric range, so that user 1 generates primary keys in the range of 1-10,000; user 2 generates primary keys 10,001-20,000; and so on. This has the advantage of pure numeric values (instead of text), but the disadvantage of the possibility of conflicts if a user exceeds the expected number of records. MirrorSync can help with this approach as well.
A very different approach is to allow conflicting primary keys to exist on each separate database, without ever writing those primary keys to other databases. When record #11 is written from the first database to the second (in our original example), instead of being written with primary key 11, it is written with primary 51 (the next number in the sequence on the second database). This has the advantage of the shortest possible primary keys, which are pure numeric values, with no possibility of conflicts. MirrorSync supports this method (and is the only sync framework that does, to our knowledge). It creates an internal table to translate between the primary keys on all database that are syncing, so that when record #11 is later updated, MirrorSync knows to change record #51 in the second database. MirrorSync also re-writes foreign keys when they are written from one database to another, so that foreign keys that contained '11' on the first database will be re-written with '51' in the second database.
Options 1, 2, and 3 are called 'Developer-managed', because MirrorSync writes the primary keys unmodified between the databases being synced. It is the developer's responsibility to pick some scheme that ensures that the same primary key is never used for different records on different databases. There are other variations on the same theme (i.e. one database gets odd numbers and the other gets even numbers), but they all are treated the same by MirrorSync.
Option 4 is called 'MirrorSync-managed.' The developer is not responsible for the uniqueness of primary keys across different databases; MirrorSync takes care of this for you.
If you have no preference for serial numbers or UUIDs, we generally recommend UUIDs. The advantage to UUIDs is that they can easily be merged manually in the event that the sync does not work for some reason. However, serial numbers are fully supported, and if it's difficult to switch to UUIDs, stick with what you have. If you need 'user-friendly' numbers for things like invoice numbers, job numbers, or check numbers, read the next section about user-friendly serial numbers before making a decision.
We highly recommend AGAINST using any meaningful data in your primary key. This applies to data modeling in general, not just MirrorSync. This means that primary keys should not be based on a name, initials, social security, invoice number, office location, company, username, etc. However, if you must use a schema like this, you should select 'Developer-managed' primary keys. Remember that when using developer-managed keys, YOU are responsible for making sure that the same primary key is never created in the same table on more than one device.
These are some common mistakes made with primary keys when syncing:
If your database is currently designed with serial numbers for primary keys, do not try to short-cut the solution by simply creating a new UUID field and try to trick MirrorSync into using that as your primary key. MirrorSync won't re-number your foreign keys, which means that the wrong children records will point to the wrong parent records when the records are synced with the server.
If you use a calculated value for your primary key, for instance by combining other fields or be adding a prefix, select 'Developer Managed' primary keys in the MirrorSync configuration. You are then responsible for making sure that the primary keys are unique across all devices. Only use 'MirrorSync Managed' primary keys with a single text or number field, such as a serial number.
The problem with MirrorSync-managed serial numbers is that they are not suitable for user-visible numbers, such as invoice numbers. That's because the primary key will be different on one device than another (read section above for explanation). If you're using the primary key serial number as your invoice number, it is clearly a problem if your invoice number is different between your laptop and the server!
There are several solutions to this problem. One of the first things to establish is whether your database uses a single field as both the primary key and the user-visible value (we'll refer to it as the 'invoice number'). It is always preferable (even when not syncing) to have these be separate fields from each other. Invoice numbers / job numbers / user visible numbers should NOT be what you use as a primary key in your database. Primary keys are internal database identifiers, and should not do double-duty as a user-readable value.
If the fields are separate, the problem becomes simpler to solve, because you have the flexibility to change the value of the invoice number without breaking relationships. See the section below on write-back values for the recommended approach.
If the same field is being used for the primary key and the invoice number, and if it is feasible to split this into two separate fields (one for relationships, and one for display/searching), you should do so. Unfortunately, this can sometimes be a very big job - creating the new field is easy, but then you either need to find every place that that field is used in the user interface and point it to the newly created invoice number field, or else you need to find every place that it is used in the relationship graph and re-point that to the newly created primary key field. Once you've split the single primary key field into two fields (a primary key and an invoice number field), follow the instructions below for configuring the invoice number field as a write-back value. If you'd like, 360Works can quote a price to do this development work for you.
Let's say that you have a UUID as your primary key field, as well as an invoice number field. Using the write-back approach, the invoice number field will be blank when a record is created in a spoke file. When that invoice record is synced to the server, it will get assigned the next invoice number, and that number will be written back to the invoice number field on the spoke file. The invoice number will end up the same on both the hub and spoke databases. This approach does not work unless the invoice number field is separate from the primary key.
Advantages of this approach:
Invoice numbers are in a simple, short numeric sequence, just like they would be without syncing.
Disadvantages:
The spoke file does not get an invoice number until the record is synced for the first time. If this does not work in your use case, see 'MIRRORSYNC_DATABASEID' below for an alternative approach.
To use this approach, follow these steps:
If you do not already have a serial number field in your table, create one. If you have a serial number which is used as a primary key, that's usable. We'll call that field 'serialNumber' in our example.
Define your invoiceNumber field as a text or number field with this auto-enter calculation: If( Get( MultiUserState ) = 2; serialNumber; "" ). Be sure to leave the box checked that says 'Do not replace existing value of field'. This will leave the invoice number blank when working offline, but fill it in when connected directly to the server.
In the primary key selection screen in MirrorSync, select invoiceNumber as the write-back field. This will cause MirrorSync to write invoiceNumber from the server back to the spoke file when the record is first synced.
Your choice of MirrorSync-managed or Developer-managed is unaffected by this setting; pick appropriately depending on whether you are using serial numbers or UUIDs as your primary key.
Let's say that you have a single serial number as your primary key, which is doing double-duty as an invoice number field. You can use the $$MIRRORSYNC_DATABASEID global variable to help. Using this approach, MirrorSync ensures that each file being synced gets assigned a unique sequential number, starting from 1. You can use this number in conjunction with a traditional serial number to create a unique number than can be used as a user-visible field as well as a primary key. The serial numbers will either be text (i.e. "1.1", "1.2", "1.3"…) or numeric (10,001 for user 1, 20,001 for user 2, 30,001 for user 3…).
Advantages of this approach:
Reasonably short user-visible numbers
IDs are assigned immediately upon record creation, without needing to wait for syncing
Disadvantages:
spoke users must do an initial sync before they can create records.
Requires a startup script to run, which means it won't work with custom web publishing applications.
IDs are not sequential, so you can't assume that an ID with a higher value was created before or after another ID.
If you are using numeric version, limits users to creating a fixed number of records before they start conflicting.
If you are using text version, you must switch all primary key and foreign fields from 'number' type to 'text.'
Setup is a bit more complex than write-back fields.
To use this approach, follow these steps:
Set the 'MirrorSync setup' script as your startup script. If you already have a startup script, call the 'MirrorSync setup' script from your existing script. This will set the $$MIRRORSYNC_DATABASEID global variable. This clientId field is stored in the MirrorSync table, and is assigned when the user does their first sync. Each synced database will get a unique, sequential clientId. Manually run the script now to set that global variable before proceeding.
We assume you have a serial field called 'serialField', and that your relationship graph uses this field as a key field. Start off by duplicating this field, to create a new field called 'serialField copy.' Do NOT place 'serialField copy' onto the sync layout for the table.
Change serialField to an auto-enter calc. Uncheck the box that says 'Do not replace existing value'. Also make sure that the 'Prohibit modification of value during data entry' is unchecked. Set the formula to one of the following:
For text keys, use this formula. This will result in primary keys that look like "1.1", "1.2", "1.3" from device 1, and "2.1", "2.2", "2.3" from device 2. Feel free to modify as desired. Be sure to change the field type from number to text. Also change all foreign keys that relate to it to text.
$$MIRRORSYNC_DATABASEID & "." & serialField copy
For numeric keys, use this formula. This will result in primary keys that look like "10001", "10002", "10003" from device 1, and "20001", "20002", "20003" from device 2. Replace whatever number you want instead of 10,000 to give a wider or narrower range of numbers.
For primary key configuration in MirrorSync, select Developer-managed, and set 'serialField' as your primary key.
Write-back values for SalesForce-to-FileMaker configurations
This section pertains to users developing a SalesForce-to-FileMaker configuration. If you're setting up a MirrorSync configuration with SalesForce as the hub and FileMaker as the spoke, you can write data generated by Salesforce back to the corresponding FileMaker record during a sync---like the auto-generated SalesForce primary key. To do this, follow the steps below:
Step through the sync configuration to the page where you specify the primary keys and timestamp fields
Select the SalesForce field you want to write to FileMaker as the hub-side write-back value
Select the FileMaker field receiving the SalesForce field value as the spoke-side write-back value
How do I configure and use two foreign keys as a primary key?
This is a common configuration for join tables in many-to-many relationships. In the primary key configuration screen, use the drop-down menus to select the two foreign keys.
Please keep in mind primary keys need to be unique. If using two foreign keys, there may be an instance where an error occurs about a duplicate node ID, since FileMaker is not validating the fields together and making sure they are always unique in a table. To avoid this, make sure:
Ensure that the combination of two foreign keys only ever occurs once in the database.
If you need to be able to have multiple join table records with the same foreign keys, add a regular serial number field to the join table and use that as the primary key, instead of the compound foreign keys.
The foreign key fields must both have the 'Not empty' validation in order to appear in the pull-down menu of eligible fields. If your join table needs to have foreign keys that may be empty, you should add a traditional single primary key to the table instead of using compound foreign keys as an identifier for MirrorSync.
Why does MirrorSync need to know about foreign keys?
MirrorSync uses foreign keys for several purposes:
If you are using MirrorSync-managed primary keys, then it needs to know about the foreign keys because it needs to rewrite them when it writes between databases see the Primary Key section above.
MirrorSync needs to know about relationships so that it can insert/update parent tables before children tables. This allows validation rules that check referential integrity to work correctly. MirrorSync deletes records in reverse order from insertions (delete children, then parents) in order to avoid triggering cascading deletions, which are less efficient.
MirrorSync will write foreign keys before other fields, so that if the foreign keys trigger default lookup values, those values will be overwritten with the custom field values from the other database. This won't work unless MirrorSync knows which fields are the foreign keys so that it can write them first.
Does MirrorSync need the databases being synced to be identical?
No, MirrorSync is able to sync between databases with completely different tables and fields from each other. In addition, it can sync just a subset of records, fields, and tables. If you are syncing different databases, for example a dedicated mobile file with a much larger server database, select the option 'Sync with a separate mobile file' (for FileMaker Go) or 'Sync with a separate server file' (for server-to-server sync). With this option selected, MirrorSync will allow you to match up your layout and field names with a simple drag and drop interface.
One important rule is that while table names and field names don't need to match, there must be a 1-to-1 correlation between the tables to sync. In other words, if you select 5 tables to sync with in the hub database, they must match up to 5 tables (not 4, or 6) tables in the spoke database.
MirrorSync will work fine with multi-file solutions. You need to make sure that one of the files in your solution has a reference to every table that you want to sync. When using the MirrorSync download feature, be sure to select the multi-file option so that your spoke users will receive all the files for the solution.
If you are using the separation model with a UI and data file, configure MirrorSync to use the UI file. After the setup is complete, copy the MirrorSync table from the UI file to the data file, update the relationship graph to point to the table in the data file, and then delete the table from the UI file. This way, you'll have all the scripts and layouts in the UI file, while the MirrorSync internal metadata will be stored in your data file.
MirrorSync supports ESS tables, but it's very slow compared to using MirrorSync to simply sync the database from the external SQL database into your FileMaker Server. For this reason, we do not recommend ESS unless syncing is not an option for some reason.
If you would like to use ESS tables, there are additional steps. First, set up sync layouts and primary keys, modification timestamps, and creation timestamps. Next, make a duplicate of your file and host it on FileMaker Server. Make the following changes to that duplicated file ONLY. We'll need to remove the ESS references to allow the user to access the data offline by turning the shadow tables into regular tables.
First, delete the data source. This is vital to preventing duplicate records and other artifacts of using ESS. Open File > Manage > External Data Sources and delete the OBDC data source.
Next, open the File > Manage > Database and navigate to the Tables tab. Copy any ESS tables. Then delete them. Make sure NOT to delete the table occurrences. After all the italicized tables are gone, paste the tables back in. This ensures they are local FileMaker files, not references to external data.
Make sure the primary keys, modification timestamps, and creation timestamps are correct. The primary key needs to be an auto-entered serial number or generated UUID.
Open the relationships tab in the File > Manage > Database and relink the missing tables.
Now you're ready to configure MirrorSync! Start the configuration process, and select "Sync with a separate mobile file" on the third screen. Choose the file that was duplicated and modified, and continue with configuration as normal.
Before configuring MirrorSync to sync between FileMaker Server and some SQL database, such as MySQL, you should consider whether using FileMaker's External SQL Source (ESS) is a better option for you. ESS allows you to access SQL tables directly from within FileMaker, as if they were local FileMaker tables.
Some advantages of using ESS:
It is built into FileMaker and FileMaker Server, therefore there is no cost, although there is often a cost for the ODBC drivers to access the SQL database.
There is somewhat less setup for ESS than MirrorSync, although you still need to install ODBC drivers and configure the ODBC DSN on the server. Using MirrorSync's ability to create FileMaker tables (this is done in the last page of the MirrorSync configuration process), the difference in setup time is not that significant.
With ESS, there is less work when fields are added, deleted, or changed; go into field definitions and click the 'sync' button. With MirrorSync, you need to walk through the configuration process again so MirrorSync can get an updated list of fields.
Using MirrorSync, the SQL data is synced to local tables in your FileMaker Server. This gives you your own copy of the data that always matches what is in the SQL database. Advantages of this approach:
It is much, much faster than using ESS. For doing reports on large tables, it is literally hundreds of times faster. For instance, we used to use ESS for pulling sales reports from our MySQL online store database. It took 40 minutes to pull a report for the quarter. By switching to MirrorSync and pulling the same reports with local FileMaker tables, it takes less than 10 seconds (240 times faster).
Having local tables allows you to access and create field indexes, which is necessary if you want to use value lists, unique validation, or the 'paste from index' feature. It is also necessary for relationships that are not based on primary keys.
Sorting records on local FileMaker tables is much faster than sorted records via ESS. Sorting more than a few thousand records from ESS becomes unusably slow.
You can add your own additional fields to local tables (with ESS, you can add unstored calculations and summary fields, but nothing else)
You can access data in your local tables regardless of whether your internet connection is working or not.
You can rename the tables and fields in your local copy and use MirrorSync's mapping feature to match them up with the ESS database.
ESS does not work with existing FileMaker tables. MirrorSync allows you to sync existing tables with the SQL database.
Unless you configure an ODBC DSN on everybody's laptop (not an option for iOS), ESS will not work if you make a local copy of the database to your own laptop. With MirrorSync, you can copy the database from the server to your own computers and iOS devices and keep all the data.
MirrorSync fully supports binary data in BLOB / Container fields. ESS does not.
Local tables fully support multi-user record locking and broadcast notifications when records are changed. ESS does not.
MirrorSync allows a developer to write custom SQL queries to only retrieve certain records. ESS does not (although a DBA could create an SQL view to solve this problem).
MirrorSync allows custom SQL statements for generating primary keys for new records inserted into SQL. ESS does not.
In summary, ESS may be a good choice when you have a small amount of data which is accessed infrequently with simple list and detail views.
MirrorSync is a better choice for data that is large, mission-critical, and accessed often.
Can I have different sync scripts to sync different tables?
Yes. MirrorSync has the ability to have multiple configurations per database. For instance, you could have a 'full sync' which syncs all tables in both directions, and then a 'save work order' sync which would only sync a handful of tables one-way from the spoke device to the server. You could then attach the 'save work order' script, which would be very fast, to a script trigger that runs whenever a work order is completed. If you set up multiple configurations syncing to the same hub database, be sure to use the same server address for both configurations (i.e. don't use an IP address for one of them and a hostname for the other).
Yes, you can. There are three different settings related to this.
Use the admin e-mail address, which is configured in the MirrorSync setup process. By entering an e-mail address here and specify a severity from the pull-down menu, you can receive an e-mail message when certain events occur. You can select to receive e-mails every time a sync is done, only when there is a warning, only when there is a fatal error, and you can also specify whether you'd like to receive detail information with the affected primary keys.
Modify the MirrorSync.xml file. By modifying this file with a text editor, you can set the 'supportEmailAddress' field. When this is set, any bugs reported by users will be cc'd to this e-mail address, and tech support responses from 360Works will go to this e-mail address instead of back to the user. This allows your users to report any problems they encounter, while letting you (the developer) make any necessary technical adjustments.
Set up advanced notifications (described in the next section), and configure AWS CloudWatch to notify you based on conditions you select.
Can I see performance statistics for sync activity?
Use the advanced notifications feature to accomplish this. The advanced notification button appears on the same configuration page where you set the administrator email address for notifications. When this is enabled, MirrorSync uses Amazon Web Services (AWS) to send notifications of sync statistics or failures.
AWS Access key / AWS secret key: This is needed for all advanced notifications. If you don't have an AWS account, you can create a new one for free at this link. If you have an AWS account and you're not sure how to get an access key or secret key, follow the documentation at this link to authenticate as yourself. For better security, you can create a separate user with a custom policy that only allows SNS Publish and CloudWatch PutMetricData.
AWS Region: Pick whichever geographic location is physically closest to your MirrorSync server, unless you have a reason to pick a specific region. When you log into AWS to see and configure notifications, you'll need to navigate to that region in the AWS console.
Sync failure SNS topic: This is not a required field. If you're familiar with how to use SNS (Simple Notification Service), you can create an SNS topic and enter the Amazon Resource Name (ARN) in this field to receive a message whenever a sync fails. SNS topics can do whatever e-mail can, but they can also send SMS messages, or initiate custom code that you write yourself using AWS Lambda.
Enable CloudWatch monitoring: This lets you send very detailed statistics to AWS CloudWatch. Check the boxes for all the stats that you're interested in tracking. Each checkbox will incur a monthly charge of $0.30 - $0.50. Once you've started running syncs with this option enabled, you can use AWS CloudWatch to view this data. You can also set up dashboards and alarms to easily visualize patterns in sync data, or get notifications if certain stats are above or below a threshold.
See the next section, Reporting sync activity with FileMaker
If you're running FileMaker as the hub and spoke, MirrorSync can create a record after every sync in a reporting database hosted on your FileMaker Server. To enable this, look for the Reporting_MirrorSync database included with the MirrorSync download. Rename it to <Your hub database name>_MirrorSync, upload it to your host server, open it up (using admin with no password), and change the full access credentials to match your hosted hub database. Now, after every sync, MirrorSync will call the "Store SyncHistory" script with JSON data. The script parses certain fields from the JSON data and creates a record, which you can use for your own searching and reporting. You may also choose to modify this script any way you want, such as receiving notifications for failures, deletions, etc.
There is a 'Nightly Summary Email' script in this file. If you'd like to receive a summary email each night of sync activity, modify the appropriate fields in the SystemSettings layout, then schedule this script to run each night.
Can I receive notifications if my auto-scheduled sync stops running?
Yes. Refer to the previous question on performance statistics and configure CloudWatch. Then, set up an alarm in CloudWatch based on some metric that you're reporting, such as total duration. It doesn't matter very much which metric you pick, or what your alarm settings are. The important part is to configure CloudWatch to notify you if the alarm goes into an INSUFFICIENT_DATA state, which is what will happen if the auto-sync stops running.
Yes, repeating fields are supported. However, performance is not as efficient if the total number of fields (including repetitions) exceeds 1,000 items.
Short Answer:Yes, MirrorSync fully supports time zone differences during sync operations. Read on, however.
However, MirrorSync cannot account for incorrect modification timestamps that are stored when FileMaker Pro or Go clients modify records directly on the server while using a different time zone.
If users that are directly connected to your hosted hub database will ever be in a different time zone than the server's time zone, manual configuration is required.
What do you mean, "incorrect modification timestamps"?
When users make edits directly to the hosted database using FileMaker Pro or Go, the modification timestamp is recorded using the client machine’s local time, not the server's.
MirrorSync, however, detects changes using the server’s time zone. This mismatch can result in recent changes being skipped during sync, because the timestamp appears older than it actually is.
Your FileMaker Server is located in New York (Eastern Time).
A scheduled sync gets run at 12:30 PM Eastern.
A user in San Francisco (Pacific Time) modifies a record on your server-hosted hub database at 10:00 AM Pacific / 1:00 PM Eastern.
The FileMaker client writes the mod timestamp on the hub record as 10:00 AM, based on its local time zone.
Another scheduled sync gets run thirty minutes later, at 1:30 PM Eastern.
To detect changes since the last sync, MirrorSync looks for records that have modification timestamp values after 12:30 PM Eastern.
The San Fran user's changed record is not detected, even though it happened after the last sync, because 10:00 AM < 12:30 PM.
The Solution: Create a custom Modification Timestamp Field
Create a new timestamp field that always uses the server’s clock, rather than the client’s. Do not delete your old modification timestamp field, as the new field will use it to trigger its updates.
Add a new timestamp field to the table, and name it something like HostModificationTimestamp.
Set it as an auto-enter calculation with the following formula:
Let ( x = zzModificationTimestamp ; Get ( CurrentHostTimestamp ) )
Replace zzModificationTimestamp with the actual name of your existing modification timestamp field.
Uncheck the following field option:
Do not replace existing value of field (if any)
Make sure this box is unchecked. This ensures the timestamp is updated on every modification.
Update your MirrorSync configuration:
If you're only syncing the fields on the sync layout, add the HostModificationTimestamp field to your sync layout and remove the old modification timestamp field from the layout.
Log into your MirrorSync server, edit your configuration, and set your table(s) to use HostModificationTimestamp as the modification timestamp.
Now, even if a user in a different time zone modifies a record, the timestamp will reflect the server’s time, and MirrorSync will detect the change correctly.
Example Host Mod Timestamp Field Def and Calculation
No. MirrorSync only uses the modification timestamp to detect recent changes. Old records don’t need a value in this field, and will get a new value when the record is modified.
If all users on that device log in with the same username, then it's probably OK, as long as you make sure that you are not using record level access privileges on the spoke device that would grant different record access to different users.
If you want each user who shares a device to log in with different usernames, then the solution is to put a separate copy of the file on the device, one for each user. Make sure that each user is only able to log in to the file that is assigned to them.
MirrorSync prevents multiple users with different usernames from sharing the same spoke file, because if record level access privileges or script filters were configured differently for each user, that would lead to the sync process incorrectly deleting the wrong records on the server, which would be very bad.
Yes, MirrorSync works with FileMaker container fields. Container fields do take longer than other field types to synchronize, so remove them from the sync (by specifying 'sync only fields on layout' and removing them from the sync layout) if you don't need them to be synced.
MirrorSync supports embedded (not external) container fields as well as external container fields (both Secure storage and Open storage). Read the next section for notes on external container fields. It does not support referenced container fields (When the user checks the box 'Store only a reference to this file' when attaching the file)
When a record is modified, MirrorSync will check each container field individually to see if it that field has changed from the previous sync. It will skip writing any container fields that match what was previously synced for that record.
Yes, MirrorSync works with external container fields, assuming your FMS folder settings are configured correctly (see the warning below for details). There are no extra steps to take when syncing with FileMaker Pro or FileMaker Server. If you are using external container fields on an iOS device AND if you are syncing with a full copy of the database (as opposed to an empty clone), there are a few steps after configuration that are required.
Download a copy of the database from MirrorSync or FileMaker Admin
Open the copy on a computer, and navigate to File > Save a copy as...
Under the type drop down, choose "self-contained copy (single file)"
This ensures the copy of the file embeds the containers into the file for easy use with FileMaker Go and offline functionality.
Warning:
Certain FMS folder settings will keep the FileMaker backup process from including external container data with your backups, which means that your external container data will not be bundled with your MirrorSync client database download (or will not be pushed to the spoke server, in the case of a server-to-server sync). For more details, read "My container data isn't getting downloaded"
Why doesn't my iPhone / iPad show a progress bar when I click the download link? Is anything happening?
When you create a download link to share with users, it creates a URL that users can click on to download a copy to their own device. For large databases and/or slow networks, this can take a very long time. Unfortunately, iOS doesn't show a download progress indicator, so it may look like nothing is happening after you click the download link. There's nothing that can be done to fix this: we suggest filing a feedback suggestion with Apple directly or using a different browser (most other iOS browsers show a progress bar). If you're really convinced that the downloads are not working, and you've waited a long time (sometimes up to an hour for very large files), send us a problem report using the link on the MirrorSync launch page. Be sure to tell us what time the download started so that we can find it in the log.
Does MirrorSync require users to sync with full access accounts?
No. MirrorSync requires a full access account only once during the initial database setup to paste in the table, scripts, and layout. When users run the sync, they will use their own user account privileges, with whatever access that grants them. See the 'Customizing MirrorSync' section below for tips on restricting user accounts.
The MirrorSync worker script is set to run with full access privileges. This fixes an edge case that can lead to unwanted deletions of records on the hub, IF the sync was set to use an account with full access when running on the server, but limited access when running on the client.
If you want to limit which users can access which records, you should customize the MirrorSync script. Since the script is set to run with fully access, you should not rely on Record Level Access (RLA) privileges to limit which records are synced. RLA permissions will apply on the spoke device outside of the sync, so if records are synced to a device that a user is not supposed to be able to see or edit, those restrictions will still apply. You may choose to disable full access on the MirrorSync Worker script, but the sync will run much slower, and creates the possibility of deleting records if RLA restrictions allow records to be synced, but then filters them out after that.
Yes, MirrorSync has robust conflict detection and field-level merging. It will detect when the same record(s) are modified on both database and take action. There are two settings in the configuration client that will control how this works:
Merge the changes together
If MirrorSync sees that all the changes made in the databases were to different fields (i.e. firstName on the server and lastName on the client), it will automatically merge the changes together. It will not tell the user that a conflict occurred, it will just write the merged record to both the server and the client. If any changes were made to the same field, it is treated as a conflict and not merged.
Note:
Conflict merging will only work on records that MirrorSync has previously written from one database to another since the last time the audit log SyncData was reset. If a record exists on both devices because it was copied over during the initial sync, and it has never been read/written by MirrorSync, then conflict merging will not work on that record. Once that record has been modified once and written by MirrorSync to the other database, merging can be done on additional changes in the future.
Flag the edit as a conflict
MirrorSync will treat all modifications to the same record as a conflict, regardless of whether they were made to the same fields or different fields.
If a conflict occurs, then MirrorSync will resolve it based on how that table is configured to resolve conflicts. There are four options for conflict resolution:
User picks
MirrorSync will present a web-based interface to the user, summarizing all conflicts and showing them exactly which fields were changed on both databases. It color-codes the actual sections of text that were changed, making it easy for even inexperienced users to make the best choice. The user is able to pick one entire record over another, pick individual fields from each record, and even manually edit the result to combine changes from both records. They also have the option to automatically select the most recent change (see next option).
Most recent change wins
MirrorSync will pick whichever record was last modified as the winner of the conflict. If Joe changes a record on his iPad at 3:00 PM, Kate changes the same record on her iPhone at 3:30 PM, and Tom changes the record on the server at 4:00 PM, then when Joe or Kate sync their database, Tom's record will be selected as the winner, regardless of when Joe and Kate do their sync.
Hub always wins
MirrorSync will always pick the value on the hub (typically FileMaker Server) as the conflict winner, regardless of other considerations. This is not generally recommended, because it is essentially first-sync wins behavior: Whoever syncs to the hub first will win conflicts with other spoke users who modify the same record and sync. This setting can be useful if changes made by users directly connected to the hub should be given priority over spoke users, or if the sync is one-way from hub to spoke.
Email administrator
This is very similar to 'User picks', except that instead of the user who is doing the sync deciding, an e-mail is sent to the administrator email address specified in the configuration. The administrator can resolve the conflict using the same web-based user interface described above, after which the user will be able to sync normally.
Is MirrorSync transactional? What happens if the connection is lost while syncing?
Although MirrorSync may be configured to use the the 'Open Transaction' FileMaker script step for speed purposes, MirrorSync is intentionally not 'transactional' in the strictest sense. If the connection is dropped during a sync, it is possible for records from one table to be written to the server, while records from another related table may not be written. However, MirrorSync keeps track of which records have been written and which ones have not, and it will resume from where it was on the next sync and complete everything as if the first sync had finished. If a record on the server cannot be written because it is locked (being edited), the spoke user will get a warning error message telling them this. MirrorSync will continue to retry that edit operation each time the sync script is run.
Can I use external authentication with MirrorSync?
For FileMaker Server, MirrorSync supports externally authenticated accounts with no modifications. However, FileMaker Pro and Go do not support this, so if you plan on syncing that way, you'll need to come up with some solution for those users to open the database on their device. The sync will work as long as the username on the local copy matches the username on the server, and the user knows the external authentication password (which may be different than the non-external account on the user's device).
Does MirrorSync respect validation settings in FileMaker?
Yes, IF they are set to 'always' (as opposed to 'Only during data entry', which is the default). The 'Allow user to override during data entry' has no effect on whether MirrorSync respects validation. If you are using UUIDs for your primary keys, we recommend setting a 'Unique value' constraint on them with the 'Always' choice selected. This protects against duplicate records, which can happen if MirrorSync is mis-configured.
Does MirrorSync retain formatting on text fields in FileMaker?
No, MirrorSync will not retain text formatting when syncing. This means that text color, style, and font information will be stripped out and converted to plain text during the sync. Some customers have come up with their own workarounds using the GetAsCSS function and custom functions to translate formatted text to plain text and then back again, but this would require custom development.
What if the sync takes longer than the auto-sync frequency?
The auto-sync frequency is the delay between the end of one sync and the start of the next sync. This means that if you sync generally takes 120 seconds, and your auto-sync frequency is set to 60 seconds, then the sync will run every 180 seconds. It will not be interrupted or canceled or run non-stop. The auto-sync feature is only available for server-to-server syncs.
Setting up a SalesForce to FileMaker Server Configuration
Setting up a MirrorSync configuration with SalesForce and FileMaker Server is mostly a straightforward process, and is very similar to a standard FMS to FMS configuration. However, there are some caveats you should be aware of, and some recommended approaches to help you navigate this process as Salesforce has unique behaviors regarding data access and entry.
You will need your Salesforce API token. If you do not have one, you can retrieve one through the Salesforce Settings page via the Reset My Security Token option.
Salesforce API Key generation
Although the names for the fields in Salesforce may not be the exact same between the web interface, the documentation, and the API call return values, they are similar enough as to be identifiable(ex. table 'Products' is called 'Product2' in the documentation, and called 'Product' in the list of available standard objects returned by the API). The easiest approach may be to create fields in FileMaker based on the field definitions provided in the Salesforce developer documentation. A link to their table and field definition documentation is provided below.
MirrorSync communicates with Salesforce via the SOAP API, so make sure to reference the appropriate Salesforce API documentation. You may also want to match your FileMaker db schema with that of Salesforce so that relationships are properly recognized in the sync. A link to docs on Salesforce schemas is provided below:
We recommend setting up a sync configuration with Salesforce as the hub. This is because Salesforce has a lot of restrictions regarding field access and mutability. Most of the restrictions are already applied by MirrorSync in the configuration setup process, and are more transparent using that approach.
In general, if the field is editable in Salesforce(Classic or Lightning interface), MirrorSync is able to update/write to the field, and for fields not editable in Salesforce, MirrorSync likely will not be able to update/write to the field. There are fields accessible by MirrorSync that can be written from Salesforce to FileMaker, and only from Salesforce to FileMaker. You can still select a table to employ a bidirectional sync, and individual fields within the table can be set to sync one-way. For many fields in Salesforce, a one-way sync setting will be applied automatically by MirrorSync as it can detect which fields are read-only.
If you matched the field definitions in FileMaker as they are specified in the Salesforce documentation, you can just step through the MirrorSync configuration and map the tables and fields accordingly.
You will need separate keys for FM and Salesforce, as Salesforce writes its own keys. This also means you must employ MirrorSync-managed keys in your sync configuration. We recommend setting up write-back fields in FileMaker to receive the Salesforce pk. They will be of type Text.
There are a few niche, special cases that you should be aware of during the setup process. Some of these cases you should look out for include:
By default, system fields like CreatedById and CreatedDate are not writeable. You can follow Salesforce documentation to enable that capability.
Salesforce requires that there always exist a Pricebook record, Standard. This record cannot from Salesforce so if you set up a one-way FileMaker-to-Salesforce sync config, you will not be able to 'delete the spoke' if syncing this table.
The Salesforce object Asset requires values in either or both of the following foreign key fields: AccountId, ContactId.
Video Demonstration of SalesForce to FMS Configuration
(Coming Soon) If you have any specific SalesForce to FMS Configuration questions, please send in an email to support@360Works.com, and we will try to answer them promptly.
For the most part, the process for setting up server-to-client sync is almost identical to server-to-server sync. If your spoke server will be a copy of the hub database, there is no need to copy the database between the servers yourself - MirrorSync will do that for you on the initial sync.
There are five ways that server-to-server syncs can be triggered:
By using the 'Sync Now' button in the MirrorSync configuration client.
By checking the 'Auto Sync' checkbox in the MirrorSync configuration client, and selecting a desired frequency in seconds. This will continue to run as a background service, even if the MirrorSync configuration client isn't running. (see important note below)
By running the MirrorSync 6 script as a FileMaker Server scheduled script. Make sure to make the frequency long enough that the next one will not occur during the previous sync.
By running the MirrorSync script in FileMaker Pro / Go which is connected as a guest of FileMaker Server.
You should never use the auto-sync option for initial syncs. Most errors will occur on an initial sync, and if an error occurs during an initial sync it's almost always faster and safer to solve the problem in the error, replace the spoke with a copy of the hub again, reset the sync data using the config client, and restart the sync from the beginning. It's recommended that you right click your config and scan for invalid data before running an initial sync, to reduce issues that could force you to restart.
How can I be notified of a problem that occurs during server-to-server sync?
There is an administrative e-mail address that is set during MirrorSync configuration. If you set this, you will receive e-mail notifications whenever the sync fails (or succeeds, depending on what granularity you set admin e-mails to). These e-mails are sent via Amazon Web Services, and thus do not require you to configure an SMTP server. You can send these emails to multiple addresses by submitting a comma-separated list.
Can I sync the same database server-to-server, and also server-to-client?
Yes, MirrorSync has the ability to configure multiple sync configurations in a database, so you could have one configuration for syncing FileMaker Server with iPads, and another configuration for syncing the same database on FileMaker Server with MySQL, for instance.
How can I trigger a Server-to-Server sync by URL?
So if your HUB server address is example.com and your configuration is named 'Inspection System', the URL you would need to initiate a Server-to-Server sync would be:
Loading a JDBC Driver for use with an unsupported database type
MirrorSync natively supports a number of common databases (FileMaker, MySQL, SQL Server, DynamoDB, etc.). If you need to sync with a different type of database, MirrorSync can often connect to it using a JDBC driver.
Because these databases are not officially supported, configuration may require more trial and error than when working with the predefined options. Whenever possible, we recommend choosing a supported database type. If that's not an option, this method provides a flexible path for advanced developers.
A JDBC driver is a Java library (.jar file) that implements the database's protocol in a way Java applications can understand. MirrorSync communicates through JDBC, so the driver is required to translate queries and results between MirrorSync and your database. Usually, each database vendor supplies its own driver, and compatibility between driver version and database version is important.
For example, the PostgreSQL JDBC driver must match the version of PostgreSQL you are running.
Once downloaded, the driver .jar file needs to be placed in MirrorSync's classpath (see below). MirrorSync loads additional libraries from its WEB-INF/lib directory inside the deployed application.
After copying the .jar file to this location, restart the MirrorSync Tomcat service to ensure the driver is loaded.
Once Tomcat is running again, configure your MirrorSync database connection by selecting JDBC as the database type. You will need the correct JDBC connection URL format for your database/driver combination (consult the driver's documentation).
If you encounter issues during configuration, contact us with the database type and driver details so we can help troubleshoot. Keep in mind that because MirrorSync wasn't designed to sync with these out of the box, extended support related to these configurations may not be free. Feel free to write in to our support team, and if we've reached the limit of free support we'll let you know.
Where should I customize MirrorSync for FileMaker?
The main MirrorSync script named "MirrorSync 6( yourConfigName )", is the only FileMaker script you should customize. We do not recommend making changes to the MirrorSync worker script. There are two sections of the script that you should customize: Global variables at the top, and scripting hooks at the bottom.
Global variables mostly start with the name $$MIRRORSYNC_, and are described in more detail in the following answer.
Scripting hooks start near line 146 of the main script with the comment #CUSTOMIZATION INSTRUCTIONS. They control custom behavior before, during, and after the synchronization process. If you have customized the DidInsert, DidUpdate, or WillUpdate sections of the script, they will not be called by default. Review the instructions in the script for $$MIRRORSYNC_DID_INSERT, $$MIRRORSYNC_DID_UPDATE, and $$MIRRORSYNC_WILL_UPDATE to enable them.
$$MIRRORSYNC_USERTOKEN: This can be set to any arbitrary text value you want on the spoke (it is ignored on the hub). It will then be available in the customization sections of the script when running on the hub. You might use this to store a username, a list of parent record UUIDs, a client name, etc., and then you could use this when filtering records, for instance, so that you only sync records for that username, those parent records, or that client.
$$MIRRORSYNC_USERNAME: If you want the sync to run with a different account than the FileMaker account used to open this spoke copy, enter it here. Otherwise, leave the value empty.
$$MIRRORSYNC_PASSWORD: If this is left blank, then the user will be prompted for their password on the first sync, or you can set a value here. If you want all users to sync with the same username and password, a better way to accomplish this (without embedding a password in your script) is by selecting the option 'Using credentials from hub database configured in previous screen' in the MirrorSync configuration client.
$$useExternalIp: The MirrorSync script can communicate with the MirrorSync server using the internal or external IP address (if a separate external IP address is configured). It will try one, and if that does not work, it will switch to the other. Usually MirrorSync can tell which one to try first by examining the IP address where FileMaker client is running, but in some cases, that can incorrectly cause MirrorSync to try the internal address first even when it is not on the same LAN as the MirrorSync Server. Setting this value to 1 will force MirrorSync to always try the external IP address first.
$$detailedLogging: Setting this value to 1 makes the sync slower, but can be helpful when troubleshooting with the 360Works support staff.
$$MIRRORSYNC_CONFIG_NAME: The MirrorSync script should be named 'MirrorSync 6 (ConfigurationName)', where ConfigurationName is the exact same case-sensitive as the MirrorSync configuration name. This parses out the value between the parentheses as the configuration name. For custom configurations with vertical market solutions, you might want to use the same script with different configurations. In that case, customize this step and set $$MIRRORSYNC_CONFIG_NAME as appropriate.
$$MIRRORSYNC_HIDE_WINDOW: Setting this to 1 will hide the MirrorSync window from popping up while a sync runs. This only works on Mac / Windows; it has no effect in iOS, which cannot hide windows off-screen.
$$MIRRORSYNC_SILENT_MODE: Change this to 1 to suppress informational dialogs that occur during sync. Error messages and anything requiring user input will still be displayed. Read how to completely suppress all dialogs.
$$MIRRORSYNC_ALLOW_INSECURE_INSERTS: If this is 0, and a spoke user attempts to insert a new record with the same primary key as a record on the hub which is filtered, the insert will fail with a security error. Change this to 1 to allow the insert to instead update the existing record, which will allow any user who knows the primary key of a record on the hub to change it, even if it is filtered out for them.
Layout-based customizations: These variables can accept a list of layouts. This can be an empty value, meaning no layouts (this is the default). It can also be the word "all", meaning apply this customization to all sync layouts. It can also be a comma-separated list of layouts, e.g. "sync_accounts,sync_contacts". Do not separate these with spaces.
$$MIRRORSYNC_DID_INSERT: The DidInsert section of the customization script is used to run custom actions after a record is inserted. By default, it is not enabled. Enable this for any layouts that need this. This disables certain performance optimizations, such as deferred flushing, so don't enable it if it's not needed.
$$MIRRORSYNC_WILL_UPDATE: The WillUpdate section of the customization script is used to run custom actions before a record is updated. By default, it is not enabled. Enable this for any layouts that need this. This slows down update operations, so don't enable it if it's not needed.
$$MIRRORSYNC_DID_UPDATE: The DidUpdate section of the customization script is used to run custom actions after a record is updated. By default, it is not enabled. Enable this for any layouts that need this. This disables certain performance optimizations, such as deferred flushing, so don't enable it if it's not needed.
$$MIRRORSYNC_WILL_DELETE: The WillDelete section of the customization script is used to run custom actions before a record is deleted. By default, it is not enabled. Enable this for any layouts that need this.
$$MIRRORSYNC_DEFERRED_FLUSH: Deferred flushing can significantly speed up insert and update operations to FileMaker, especially for solutions with a complex relationship graph. It is automatically enabled if 1) FileMaker version is 20.3 or later, 2) the spoke database is a copy of the hub (not a separate mobile file), 3) MirrorSync is configured to sync all fields in a table, instead of just fields on the sync layout, and 4) customizations are not enabled for didInsert or didUpdate. If #2, #3, or #4 are not true in your case, but you would like to use deferred flushing anyway, enable this customization for the appropriate layouts. The danger in doing this is that using deferred flushing can lead to incorrect values for auto-entered calculations that depend on values looked up from related tables, so be sure to test thoroughly.
$$MIRRORSYNC_HUB_TRANSACTIONS: By default, MirrorSync does not use the Open Transaction script step on the hub. If you want to use transactions, change this to a comma-separated list of layouts that need this, or put the word "all" for all layouts. Our benchmarking has shown that transactions can make the sync faster or slower depending on your database; try both ways and see what works best for your solution. See transactions for more information.
$$MIRRORSYNC_SPOKE_TRANSACTIONS: This is the same as $$MIRRORSYNC_HUB_TRANSACTIONS, but for spoke databases. Transactional speed may be different for databases hosted by FileMaker Server than spoke copies, which is why it can be set separately for hub and spoke.
$$CUSTOM_FILE_NAME: For databases downloaded using the version update feature, use "" to keep the filename from the server. Use Get( FileName ) to preserve the local filename. Use a hard-coded value to override the filename to whatever you want. DO NOT include the .fmp12 file extension
I don't want to sync my entire database to my spoke users. Can I just sync certain tables, fields, or records?
That's a three-part question with a three-part answer.
Syncing certain tables is very easy: When you're configuring the sync using the MirrorSync configuration utility, only include layouts corresponding to the tables that you want to sync.
Syncing certain fields is also very easy: Select the MirrorSync option to only sync fields on the layout, and don't include any fields on your sync layouts except the ones that you want to sync. Fields that are outside of the layout margins are still treated as if they are on the layout, and will sync. Note - you can uncheck the 'same for all tables' checkbox to change this setting for some layouts and not others.
Syncing certain records is only slightly less easy: Edit the 'MirrorSync Customization' script. Go to the 'hub' section (not the 'spoke', that should almost never be modified for record filtering) and follow the instructions in the 'FindChanges' documentation there to constrain the found set to just the records you want to sync for the current user. Remember that the script will be running as the user doing the sync, so you can use the Get( AccountName ) function to determine which records are available. $$globalVariables (with two dollar signs) will last for the duration of the sync, so sometimes it can be helpful to set these once in the WillSync section, and then you can refer to them anywhere in the sync.
Any records that are excluded by your search criteria will not be synced to the user. In addition, if those records were previously synced, they will be deleted from the user's device. This makes it very easy to create check-in / check-out workflows with MirrorSync, where jobs are synced to iPads, completed, and then deleted from the device.
To test your customization script, copy and paste your customizations into a new script. Go to that sync layout, show all records, and run your test script. If you don't get the results you expect, use the Script Debugger to tweak it.
If you decide to limit the synced records, you may choose to distribute an empty clone to your users instead of a full copy of the database. That way, the initial sync will only pull the records into their spoke file that they have access to on the database. Keep in mind that if they have access to many thousands of records, that could make the initial sync slower than if you distributed a full copy of the database that already contains all the record data, and let the initial sync delete the extra records.
Note:
If you use the MirrorSync customization script to filter records, MirrorSync will delete the excluded records from the spoke (client) database if they are already present. This can present a problem if there are cascading deletions that will trigger to delete other related records, because MirrorSync won't be "expecting" those records to be deleted in the spoke. If this occurs, the cascading deletions will cause those same related records to be deleted on the hub.
To resolve this without simply disabling the cascading deletions, you just need to use the MirrorSync customization script to exclude those related records from the sync.
How can I filter records on a per-user basis if everybody is logging in with the same FileMaker account?
Use the $$MIRRORSYNC_USERTOKEN to accomplish this. This is a special global variable that MirrorSync passes from the client to the server when a sync runs. You can set this global variable to whatever you want on the client, and it will be set on the server so that the MirrorSync customization script can use it when searching for records.
You can use SQL queries to filter records if your hub is a SQL Server, Oracle, MySQL, or similar database. In the screen where you select your primary keys, check the box titled 'Custom SQL qualifier'. This will pop open a dialog where you can write any arbitrary SQL qualifier to restrict which records the user will get. Note that you can use the {MIRRORSYNC_USERNAME} and {MIRRORSYNC_USERTOKEN} (see previous tip) substitution variable to customize the search based on the account name that they used to log into FileMaker Go/Pro with - even if you are using a single shared account for logging into the hub database.
You can test your SQL qualifier in this screen to make sure that there are no syntax errors. However, this will only work when editing a configuration, not when creating a new one, so complete the configuration without filtering and then edit it to test the SQL filtering.
MirrorSync will append your SQL qualifier to the auto-generated SQL ending with the table name. For instance, if you used the SQL qualifier "WHERE (category='active')", then when MirrorSync generates the complete SQL, it would look like this:
SELECT "column1", "column2" FROM "Your Table Name" WHERE (category='active') AND modificationTimestamp={TS '2016-07-15 9:00:00'}
It's a good idea to put your search terms in parentheses, because MirrorSync will append an AND condition with the timestamp. For instance, if you used this search qualifier: "WHERE status='active' OR status='pending'", that will be re-written as "WHERE status='active' OR status='pending' AND modificationTimestamp={TS '2016-07-15 9:00:00'}", which would give incorrect results. However, if you wrote your query as "WHERE (status='active' OR status='pending')", then it would be rewritten as "WHERE (status='active' OR status='pending') AND modificationTimestamp={TS '2016-07-15 9:00:00'}", which would give the desired results.
If you want to write qualifiers that use related field data, you can do that using join statements, like this. Note that you can use table aliases for the additional tables in the query, but NOT the base table:
INNER JOIN secondTable T2 ON ("YourTableName".PK=T2.FK) WHERE ("Your Table Name".category='active' AND T2.someOtherFlag=1)
This would then only retrieve items from the base table where the base table has the category set to 'active' and the second related table has someOtherFlag=1.
If you want to filter records based on the current user's account name, you can use the {MIRRORSYNC_USERNAME} substitution, like this:
WHERE assignedAccountRep = '{MIRRORSYNC_USERNAME}'
If you want to filter records based on some user selection, then set the $$MIRRORSYNC_USERTOKEN global variable on the client before running the sync. This will then be passed to the MirrorSync server, where it is available using the {MIRRORSYNC_USERTOKEN} substitution. For instance, you could pop up a custom dialog asking the user for a job number. Their selection would be stored to the $$MIRRORSYNC_USERTOKEN global variable, and could then be filtered on the server by using this custom SQL qualifier:
WHERE jobNumber = '{MIRRORSYNC_USERTOKEN}'
You can combine all these qualifiers using AND statements. For instance, let's say that you only wanted to sync active jobs, where the related user in the staffMembers table matches the current username, due in either 30, 60, or 90 days, depending on a user selection. The SQL qualifier would look like this:
INNER JOIN staffMembers T2 ON ("Your Table Name".userId = T2.primaryKey) WHERE T2.accountName = '{MIRRORSYNC_USERNAME}' AND "Your Table Name".jobStatus = 'active' AND "Your Table Name".dueDate <= (CURRENT_DATE + {MIRRORSYNC_USERTOKEN})
How can I tell if a record created in FileMaker Go has ever been synced with the server?
During configuration, select any field as a write-back value (see write-back values in the primary key section, above). Make sure that value is only set to auto-enter a value on the server, by checking that Get( MultiuserState ) = 2`. If the field is blank, the record has not yet been synchronized. If the field contains a value, then it was either created on the server, or created in FileMaker Go and synced with the server.
See the following example with a field named "tag_synced":
Tag_synced Field Definition and Calc
Defining tag_synced as a writeback field
Another approach you can take is by modifying the didInsert section of the customization script (see next question), but the write-back approach is significantly faster when syncing.
How can I run a script on the server when a record is synchronized?
Maybe you want to send an e-mail to an administrator when a user completes a delivery, creates a request for a quote, or deletes a job in production. To do these types of operations, modify the MirrorSync customization script. There is a section in the script for 'DidUpdate', 'DidInsert', and 'WillDelete'. Make sure you modify those in the 'Hub' section - it's very rare that these should be modified on the spoke. In the relevant section of the script, check the current layout, and if it's the one where you want to take action, go ahead and do whatever you want in this section. You can freely go to other layouts, do searches / inserts / updates / deletions, send e-mails, or take any other action, as long as you finish the script on the same layout and record where you started.
If MirrorSync detects that the record was modified as a result of a 'DidInsert' or 'DidUpdate' script, it will re-do the client search, taking into account any scripted record restrictions in the 'FindChanges' section. For example, if you want jobs to be removed from the user's iPad when a record on the 'SyncJob' layout has its status changed to 'Completed', take these steps:
Modify the 'FindChanges' to exclude records whose status is 'Completed' when Get( LayoutName ) = SyncJob. (To do this type of search in FileMaker, enter Find mode, Set the status field to 'Completed', Omit the current record/request, and then Constrain Found Set)
Modify the 'DidInsert' section to blank out the modification timestamp when Get( LayoutName ) = SyncJob.
Do the same thing in the 'DidUpdate' section.
Now, whenever a spoke user updates or inserts a record in the 'SyncJob' table, the script will blank out the modification timestamp. This triggers a change, which causes MirrorSync to re-do the search, which will exclude the complete job from the search, which will automatically remove it from the iPad.
You might ask, why blank out the modification timestamp? The answer is that it actually doesn't matter what you do, as long as you make some change to the record at all. Blanking out the modification timestamp is a harmless change, because FileMaker will immediately replace the blank value with a new timestamp. You could just as well set any field on the record to its current value, or make some more meaningful change, such as filling in a date field with the data that the job was completed.
I don't want users to see any dialogs when the sync runs, can I prevent them from showing?
Yes, set the global variable $$SILENT_MODE to "1" in the MirrorSync customization script. This will prevent MirrorSync from showing any dialogs unless an error occurs, and for required information like login and conflict resolution.
If you would like to also skip the login dialog, you can select the spoke option to sync with the hub database credentials. This will prevent MirrorSync from prompting the user to enter their password.
If you would like to prevent the conflict resolution dialog from displaying, pick any option other than 'user decides' in the conflict resolution section of the MirrorSync configuration process.
If you would like to either suppress or customize any dialogs from displaying, modify the "ShowMessage" action of the spoke customization section.
I don't want users to see the MirrorSync window at all when a sync runs. Can I prevent that from showing?
If you are running MirrorSync on FileMaker Pro, then yes. Set the $$MIRRORSYNC_HIDE_WINDOW variable to 1 in the MirrorSync customization script. This will move the MirrorSync window off-screen, where the user cannot see it. This is not recommended for large sync operations that will take longer than a few seconds, because without that status display, users may think that their computer is frozen.
If you are running MirrorSync on iOS, then this cannot be hidden. That is because MirrorSync needs to open a new window to manipulate the found set without losing the current selection, and new windows cannot be hidden offscreen on iOS devices. The $$MIRRORSYNC_HIDE_WINDOW variable will be ignored on iOS.
Can I block MirrorSync dialogs from being displayed to the user, or change the wording?
Yes: Set the global variable $$SILENT_MODE to "1" in the MirrorSync customization script to prevent MirrorSync from showing any dialogs unless an error occurs, and for required information like login and conflict resolution.
If you would like to also skip the login dialog, you can select the spoke option to sync with the hub database credentials. This will prevent MirrorSync from prompting the user to enter their password.
If you would like to prevent the conflict resolution dialog from displaying, pick any option other than 'user decides' in the conflict resolution section of the MirrorSync configuration process.
Using the MirrorSync customization script, all dialogs, even error messages, can be suppressed or modified. Look towards the bottom of the MirrorSync customization script in the "ShowMessage" action, and you will see a section that includes a dialog for each type of message that is displayed to users during a normal sync. You may freely customize the message displayed to the user, or the title of the buttons (but be sure that the first button still has the same meaning, and the second button still has the same meaning). If you want to hide the dialog from showing, you can disable the script step that displays it, but you must decide which option you would like to select, and exit the script with that button number. You can use this technique to display messages to your users in their preferred language.
Can MirrorSync run automatically when my users finishes a job/invoice/widget? Can it run at startup?
Yes, absolutely. If you want to run the sync on the first startup of your solution (but not subsequent startups), open the "MirrorSync 6 onFirstOpen" script and modify it to call the MirrorSync script for your configuration. Then, call this script from your startup script. This will check to see if the file is running offline, and whether it has ever been synced before. If not, it will prompt the user to do the initial sync. In addition, the MirrorSync setup script sets the $$MIRRORSYNC_DATABASEID global variable, which is necessary for some primary key strategies see the Primary Key section above.
Remember that the MirrorSync script is a regular FileMaker script that can be set to run at any time by the developer. For example, if there is a critical section of your application that requires users to see what the latest value is before they make a change, trigger the sync when the user navigates to that screen, and again when they are finished editing in that screen. You could make it so that checking a box, clicking a button, or any other condition that you choose can trigger the sync to happen.
Also keep in mind that you can create multiple sync configurations - so you could sync just a certain table, maybe with a different sync direction than usual, and attach that configuration to the button for the fastest possible performance.
Can MirrorSync run automatically every x seconds/minutes?
Yes, you can use an On Timer script to trigger this. However, keep in mind that when the script runs, it commits the current record and pops open a new window that could be open for a few seconds or longer. This could get annoying if it happens in the middle of something that the user is working on. For this reason, we recommend the approach in the previous question of linking your script sync operations to some user-initiated action.
Let's say that you have two users that independently create new records for the same client. On your server, you want to automatically delete any records that get inserted with the same client name as a record that already exists. The place to put the de-duping script would be the 'DidInsert' section of the MirrorSync customization script. Your script should get the client name (or whatever field(s) you want to use to de-dupe) of the newly inserted record and do a search for it. If only one record comes up, then there is nothing else to do. If multiple records come up, then delete whichever ones you want to get rid of. That will automatically trigger a re-sync to happen after the sync finishes. You could of course do more sophisticated operations such as combining values from the second record into the first, or simply mark the records as potential duplicates for an administrator to clean up later rather than actually deleting them.
Can I build my own front-end for the download link URL feature?
Yes, you can trigger the download link URL from anywhere in order to get a copy of the current file on the server. If you want to do this from FileMaker itself (with the 'Insert from URL' command), or anything else that is not a browser, you will want to disable the JavaScript progress pages that inform the user that the download is being generated. To do this, add a '?status=0' onto the end of the URL, like this:
There is an excellent article by Stephen Dolenski, posted on FMForums, that shows how you can notify users when a new version is available. You can read that here.
MirrorSync has in-database logging, but it's not on by default. 360Works employees may ask for this logging from time to time to help with diagnosing a problem with the MirrorSync scripts.
Here's how to enable and fetch these logs:
Turn on $$detailedLogging in the MirrorSync 6 script, like this:
This will turn on detailed logging for this database only, NOT for all databases in the sync.
Run a sync that reproduces the problem you're trying to log.
Now we need to fetch the transcript. Go to the MirrorSync layout and click the "Internal" tab.
Cycle through the records. If you're on your hub database, the transcript will be in the "hub" record. If you're in a spoke database, it will be in the "client" record.
Once you find the correct record, click the "Debugging" tab.
Copy the contents of the "Last Sync Transcript" field. If a 360Works employee requested the log, send it off to them as a reply to their email.
Note:
Turning $$detailedLogging on will impact performance on this database's syncs. Turn it off later if performance is a major concern for your sync.
Container data is often irreplaceable, and some MirrorSync users may wish to prevent MirrorSync from syncing container data deletions from one database to another. Starting in MirrorSync 7.03, we have exposed an option to prevent this.
Once MirrorSync version 7.03+ is installed AND the script named MirrorSync 6 worker is replaced with the newest version, you can do this by modifying the MirrorSync.xml file on your MirrorSync server. Look for the neverDeleteBinaryData parameter and set it to true. This parameter may not yet exist in your MirrorSync.xml file; if it doesn't you can add it alongside the other parameters by adding this text:
<!--If neverDeleteBinaryData is true, then binary data fields (like FileMaker container fields) can be inserted or changed, but if they are changed to an empty value on the source, they will be left unmodified on the destination. --> <Parameter name="neverDeleteBinaryData" value="true" />
Note that modifying this file should restart your MirrorSync server automatically, but to be certain we recommend manually restarting Tomcat.
Warning:
This change will only work with updated MirrorSync 6 worker scripts. If you have replaced the hub's worker script and not the spoke, the spoke database will not respect this change, and container data that is deleted on the hub will cause the corresponding data to be deleted on the spoke.
The MirrorSync.xml file is a configuration file intended to give users a wide range of customization options for how this instance of MirrorSync will run for all configurations. These options are not available in the MirrorSync configuration utility, and are intended for advanced users. The effects of the parameters within this file are documented within that same file. If you have any confusion about how a parameter will function, please contact support@360works.com before attempting its use, unless another part of this documentation recommends its use.
Note:
Tomcat and MirrorSync will restart after modifying this file, so be sure not to change this file during an active sync.
The name of this file may vary if you chose "Hosting Provider Installation" and selected a different name for the app than "MirrorSync". If you're unsure of the name of the file, you can find it by looking in the conf/Catalina/localhost directory; the name of the file should match the name that was assigned to the instance on installation. If there are multiple instances of MirrorSync installed on the same server, each will have their own .xml file in this location.
Each configuration in your list has a right click menu for additional maintenance procedures (context menu pictured below). Many of these options are self-explanatory, the impact and usage of the others are documented below.
MirrorSync keeps an internal record of the sync activity, including primary keys, foreign keys, creation date, mod count etc. All this data is stored in the SyncData folder. Sometimes, usually due to running out of disk space or RAM, the SyncData may become corrupt. If this happens there are two options in order to resolve the issue. We recommend first trying to Repair SyncData and if that does not fix your issue you can Reset SyncData. Both of these operations are initiated in the MirrorSync Configuration Client.
Repairing the SyncData will try to rebuild the internal database for MirrorSync. It can take a few seconds for small databases, or hours for large databases, and no syncing can happen while this is running. Existing spoke files can continue to be used after this finishes.
If Repair SyncData did not work, your next step to take would be to reset SyncData. Resetting the SyncData will delete the SyncData on the server for the specified configuration, and any other configurations sharing the same hub database.
If your primary keys are MirrorSync-managed, resetting SyncData will require re-distributing new spoke copies.
If your primary keys are developer-managed, and you reset SyncData, we recommend that you re-distribute spoke copies again. This is the best way to avoid problems like duplicate records, conflicts, or unwanted deletions. You can try syncing existing spoke files, however, they will take longer on their next sync and may require conflict resolution for recently changed records.
If it is a Server-to-Server sync you will be prompted for how you would like to perform the initial sync. Choose 'delete spoke' to wipe out the spoke database and re-insert all the records. Pick 'rebuild SyncData' to scan the databases record-by-record since a date you specify to find changes between them. This could take a long time for large databases. This cannot be undone! Please contact 360Works with any questions before moving forward with Reset SyncData.
MirrorSync's SyncData consists of two main parts, the "main" SyncData and the "AuditLog" SyncData.
AuditLog SyncData is usually much larger, and also much less critical to sync operations. The AuditLog SyncData is used for only two things: Merging field-level conflicts, and skipping container field data that has already been synced.
If you reset the AuditLog SyncData, changes made before the audit log was reset will not be able to be merged with changes made after the audit log was reset. This is usually not a big problem, making it reasonably safe to periodically clear out your AuditLog SyncData to clear up space. It can also help to reset the audit log SyncData if you're getting sync error messages similar to "Conflict was not resolved".
This option is rarely necessary. After selecting this option, the next sync for each device will be slower than normal while MirrorSync checks all field names and types. MirrorSync usually detects these changes without any admin intervention, but this step is occasionally needed if MirrorSync 'misses' a change.
This will compress your SyncData, reducing the size on disk. In MirrorSync 4 and earlier, this would often free up significant space, but in later versions, the process for writing SyncData is much more efficient, so it's unlikely that running this operation will be beneficial. Sync operations cannot run while this operation is in progress, and it can take a long time for SyncData larger than 10 gigabytes, so while it is never harmful, we don't encourage using it.
MirrorSync normally ignores changes older than the last successful sync. There may be times, however, when you'd like to treat older records as if they were recently modified and re-sync them. For instance, if records were not written due to validation failures, and then you modified the validation rules to allow those records to sync, you could use this function to retry those old records that previously failed to write.
This option will scan your database(s) for problems that may cause a sync to fail, including:
Blank primary keys
Duplicate primary keys
Invalid timestamps
Invalid container data
Large text lengths. FileMaker's JDBC driver fails when trying to sync text fields with greater than 65,535 characters or longer. If you are getting this error you can use this option to scan all your syncing tables for fields with text greater than 65,535. You can then shrink the text to be less than 65,535 characters or remove it from the sync completely. If neither option is ideal, you can switch to XML to bypass this JDBC bug.
It's a great idea to use this function before your first sync, or any time you get an error message that MirrorSync received a '?' instead of a valid result. If MirrorSync uses the ExecuteSQL function to fetch a batch of data that contains even a single invalid timestamp field value, this will cause the ExecuteSQL function to return a '?' for the entire batch. In addition, the invalid data scan will look for duplicate or blank primary keys.
Note:
A successful scan doesn't indicate that a sync will always succeed, and while MirrorSync will identify many issues, won't correct them for you as many of these problems may have different solutions depending on your preferences. However, if you get an error message related to invalid data it's a good idea to run this scan to get a more comprehensive view of what's happening.
Following is a description of files that MirrorSync writes to disk. You'll see several references to the MirrorSync.xml file below, see the provided link for details.
If you're on a Mac, and you want log files to go on an external volume, you'll need to grant special OS permission to allow MirrorSync to do this. See 'Integrating MirrorSync with External Volumes on macOS' for more details.
The core sync data. This is critical for the operation of the sync. It grows (never shrinks) until the sync data is reset, which empties it out completely. It doesn't usually grow at an extreme rate, and its size is basically a function of the number of total records across all tables multiplied by the number of spoke copies that have been synced. The amount of data stored in each record doesn't change the size of this file. You can reset the sync data by right-clicking the name of the configuration and selecting "Reset SyncData -> All SyncData." This should ideally be accompanied by distributing new spoke files to users. It is stored on whichever volume MirrorSync is installed, although its location can be customized by editing the '360WorksDirectory' setting in the MirrorSync.xml file.
The audit log database. This also grows until sync data is reset. It can often grow faster than the main sync data, depending on how often the same record is changed and the amount of data stored in the records. It is much less disruptive to periodically manually clean out this database by right-clicking the name of the configuration and selecting "Reset SyncData -> Only Audit Log". The only downside of doing this is that MS will not be able to do field-level conflict merging until it builds the audit log back up from syncing records. Changing the '360WorksDirectory' setting in the MirrorSync.xml file will change where this data is saved.
The MirrorSync logs. These are needed primarily for bug reporting and troubleshooting. Other than that, it's safe to throw them away. They are typically 5-100 megabytes per day, but they can grow very large (like 5 gigabytes per day) if there is a very large amount of sync activity. They are kept by default for 14 days, and are kept in C:\Program Files\360Works\Applications\logs. You can reduce the number of days they are kept by modifying the 'logRetentionDays' setting in MirrorSync.xml. You can also change where they are stored by modifying the 'logDir' parameter in the MirrorSync.xml file. Also, see 'storing logs on an external volume' below
The MirrorSync audit log text files are kept in the same directory as the MirrorSync logs. Unlike the audit log database mentioned in #2, these are human-readable text files that list every write operation that MirrorSync has ever done. They are automatically cleared out after the same amount of time as the MirrorSync logs (#3 above). They do not typically get unreasonably large. If you ever wonder whether MirrorSync changed or deleted a record, you can look at these text files to see that. Like the log files in #3, you can change where they are stored by modifying the 'logDir' parameter in the MirrorSync.xml file.
Temp files. These are created temporarily when downloading spoke copies, syncing container data, and syncing large amounts of non-container data. They are usually cleared out after a sync finishes. The temp files will be kept on the same volume where MirrorSync is installed. In some cases (like making spoke copies of very large databases), there might be too much data to store on this volume. You can customize the 'tempFilesDir' in MirrorSync.xml to override where these files are stored. All files in /Library/360Works/Applications/temp are safe to delete if storage space becomes an issue.
Configuration backups - every time a MirrorSync configuration is changed, all the previous configurations are saved in a file named 'MirrorSync_configsAndSchedules-XXXX.dat', in the same directory as the core sync data. If you ever make a big mistake and need to recover your last configuration, you can stop MirrorSync, pick the appropriate backup file based on the date, and rename it to remove the -XXXX portion. Start MirrorSync back up and it will load the configurations from that file. It is safe to delete any of these old backups, although they don't take up much space.
Damaged / repair folders - when the 'repair sync data' function is used, MirrorSync leaves behind a copy of the old data which was repaired. Once you've verified that the repair was successful and that you don't need the old data anymore, you can delete any folders named 'RepairTempDirectory' or 'DeviceData_damaged'.
MirrorSync logs on a busy server can get very large. We sometimes see log files of 3 gigabytes per day. To prevent your boot drive from filling up with log files, if you have a busy server, you might want to configure logs to be stored on a separate volume. On Mac or Linux, this can be done by creating a symbolic link from /Library/360Works/Applications/logs (Mac) or /var/log/tomcat9 (Linux) to the logs directory. For instance:
ln -s /logs/tomcat /var/log/tomcat9
On Windows, you cannot use aliases for this, but you can use a Directory Junction:
What is the performance like when multiple clients are syncing simultaneously?
Very good. We have achieved satisfactory results with 250 simultaneous syncing clients in testing.
How well does MirrorSync perform on slow networks?
MirrorSync is very efficient over slow networks, because it sends very little data other than the actual record changes, and it transmits this information in large chunks, which is more efficient.
Before talking about ways to make MirrorSync faster, let's first talk about what type of operations take longest:
When syncing with a database copy (i.e. iPads syncing with FileMaker Server), the first initial sync for the first device is always the slowest, because MirrorSync needs to download metadata (primary key, modification timestamp, creation timestamp, record ID, record modification count) for every record in every table in the database. After that, any initial sync on any device will be much faster. If you reset the sync data (by right-clicking the configuration), or delete and re-create the configuration, MirrorSync will need to re-scan every record again.
Full copy initial syncs (as opposed to empty clones) from a recent server copy are much faster than initial syncs from a copy made a long time ago.
Of course, incremental syncs are faster than initial syncs.
Inserts are generally faster than updates, especially for SQL databases like MySQL.
MirrorSync can tell whether records are deleted. Checking to see if anything was deleted takes no extra time. However, checking to see which records were deleted does take longer, as MirrorSync needs to scan the primary keys to find which record(s) were deleted.
Container fields take much longer to sync than non-container data.
The number of tables affects the speed of syncing: A good general rule is about .5 seconds of overhead per table, plus the time to actually sync changes.
MirrorSync is pretty fast without any special tweaking, but faster is always better. Here are some tips for squeezing out the best possible sync speed from MirrorSync:
SSD drives: MirrorSync does many thousands of tiny read and write operations to its internal database when syncing large record batches. This type of usage will benefit greatly from the ultra-low latency of SSD drives. You will see a noticeable change in sync speed by installing MirrorSync on an SSD drive. You don't need to use an SSD for your entire hard drive - you can just put MirrorSync on the SSD by using a symbolic link in OS X or changing the installation location on Windows. The SSD performance benefits will improve all sync operations, but it will be most noticeable when doing very small syncs with few or no changes, and initial syncs for large database with hundreds of thousands of records. Expensive RAID systems with huge throughput benchmarks and spinning disks will actually be much slower for MirrorSync than even the cheapest SSD storage device.
Tables with hundreds of fields will lead to slow syncs. Whenever a record is modified, MirrorSync sends all the fields for that record over the network (not just the ones that were changed). Splitting these 'wide' tables into several 'narrow' tables will make your syncs faster, and will also make a big difference when connecting directly as a guest of FileMaker Server (when not using MirrorSync).
Use the latest version of FileMaker Pro, Go, and Server. MirrorSync is constantly being tuned to use new features in FileMaker for better performance. For instance, see the following tip on deferred flushing, which requires FileMaker 20.3 or later.
Deferred Flush: MirrorSync will use the deferred flush loop mode to improve write performance. This improvement is particularly noticeable for complex relationship graphs. This is only enabled for FileMaker 20.3 or later. The following conditions must also be true, OR the $$MIRRORSYNC_DEFERRED_FLUSH customization must be enabled for the layout:
Syncing all fields for the table, not just fields on the layout
Syncing with an identical copy, not a separate mobile file
$$MIRRORSYNC_DID_UPDATE and $$MIRRORSYNC_DID_INSERT are not enabled for this table (they are not enabled for any tables by default)
Be careful about forcing this behavior with $$MIRRORSYNC_DEFERRED_FLUSH. In our testing, using the Deferred Flush mode may result in incorrect values with auto-enter calcs that are based on related data. This is safe when all of the above conditions are true, because MirrorSync will set every writeable field in the table, making it irrelevant whether the auto-enter calcs work or not. If some writeable fields are not set by MirrorSync, and you want to enable this performance option, carefully review any fields that have an auto-enter calculation which are not being set by MirrorSync, and test that they work correctly during syncs.
MirrorSync supports transactional batch write operations, but this is disabled by default. Our benchmarks have shown that sometimes there is a speedup when enabling this, but sometimes it can also slow things down. Highly complex relationship graphs are likely to benefit more from transaction batches. To enable this, open the MirrorSync script and experiment with the settings for $$MIRRORSYNC_HUB_TRANSACTIONS and $$MIRRORSYNC_SPOKE_TRANSACTIONS near the top of that script.
When MirrorSync fetches changed records from FileMaker Server, it requests all changes since 10 minutes prior to the last sync. This 10 minute overlap compensates for users whose clocks are slow. If you're using the Host modification timestamp trick outlined in the timezone section above, then the user clock doesn't matter, and you don't need this 10 minute overlap. You can get rid of it by editing the MirrorSync.xml file and change the 'fmServerOverlapSeconds' parameter from 600 (10 minutes) to 5 (5 seconds).
Container fields hold lots of data and are sent individually instead of in batches like non-container data. If there are containers that you don't need to sync, specify the configuration option to only sync fields on the layout, and remove container fields from the sync layouts.
If you want to just sync a subset of records, instead of the entire database, use scripted filters (see the customization section below for more details) instead of record level access privileges (with the MirrorSync worker script set to not run with full access). Scripted filters are much, much faster than record level access privilege restrictions.
Flag records instead of deleting them: MirrorSync has an efficient algorithm for detecting deleted records. However, it's always faster to sync edits and inserts than to scan for deletions, especially in large tables with more than 100,000 records. If you want to optimize performance, it might make more sense to flag records as being deleted instead of actually deleting them. That makes it into an edit operation, which is not affected by the number of records in the table.
Avoid unnecessary record modifications in scripts handling large amounts of records. Rather than using Replace Field Contents (which will change the modification time stamp on all affected records) or changing every record in a found set in a loop, either constrain the found set to the appropriate records before you modify them, or loop through the records and check whether the new value is actually different from the old value before making the change.
If you are starting off with an empty clone, and the initial sync is taking a very long time, you can pre-populate the empty clone table. See this section for details.
Try to avoid syncing very large text fields, such as audit logs, JSON data, or BASE64-encoded text. If you need to sync very large amounts of text, consider storing them as container fields instead, which are more efficient for large data. Do not use BASE64-encoding as a way to sync binary container data; it's much more efficient to just sync the container data directly.
360Works can review your sync logs, which include very detailed timing information, to see which tables are slowing down, and we can offer recommendations for speeding things up. Go to 360works.com/contact to get help.
Can I pre-populate my spoke database to speed up initial sync?
Yes. There are several approaches to this:
If you want the spoke user to get a complete copy of the file from the server, and then just delete certain records based on filtering criteria, then you do not need to do anything special. Just select the option in the spoke database to 'Sync with an empty clone or complete copy of the main file' when you select the spoke database in the MirrorSync configuration. When you download the database or create a download link, specify the option for a full copy. The initial sync will run very quickly in this configuration, and will take care of deleting records that the user does not have access to based on filtering criteria.
If you need to selectively delete certain records prior to them downloading a copy of the file, but you need to customize this for each user:
This is pretty simple - just make a spoke copy for that user, delete whatever data you don't want the user to have, and then send it to the user. Keep in mind that if the user has access to all the data, and if you haven't set up any filters in your MirrorSync customization script, then the initial sync will put those deleted records right back into the spoke file, so make sure you only delete the records that the user won't get during the sync.
If you are syncing with a separate mobile file that you want to pre-populate, here are the steps to follow:
Select the spoke configuration option to use a separate mobile database, instead of a copy of the hub database
Import the records you want from the main file into the mobile file. Be sure to uncheck the box that says "Perform auto-enter options while importing". The primary keys, modification timestamps, and creation timestamps need to match between the mobile file and the server file. Make sure that you import most or all the records that the user will have access to during the sync.
If you do not import all the records in a table, be sure to modify your MirrorSync customization script to exclude the records that you do not import. If you fail to do this correctly, it will lead to records on the server being deleted during the initial sync (see more details below).
After the import, open your separate hosted mobile solution. Go to the MirrorSync layout and click the internal tab, find the record with type 'Server' (create new record if necessary and set type to 'Server'). Modify the sync4 field (you'll need to switch to layout mode to see which one that is), and set it to the timestamp when you ran the import, using the format 'YYYY-MM-DD hh:mm:ss' (in 24 hour time). Most customers do not use the correct timestamp format. Be sure to check that your format matches this.
Make sure that your spoke users run a sync before making any database changes. Any new records that they create in the mobile file will be deleted during the initial sync. It's OK for them to add new records normally after the initial sync has successfully completed.
It's easy to make a mistake in this process. Don't hesitate to contact 360Works support if you'd like help with this.
If you do an initial sync with a pre-populated database, and it causes records to be deleted on the server, it's because the records being deleted were not imported into the spoke files, and were not excluded by the MirrorSync customization script. In this case, MirrorSync sees that these records are missing from the spoke file, and also that they exist on the server with an old modification timestamp, so it assumes that they were deleted on the spoke file and proceeds to delete them on the server.
What should I be aware of for very large database synchronization?
For very large databases, here are a few tips:
There is a limit of 256 gigabyte for the internal sync database. This limit can be increased by modifying the 'maxSyncDatabaseSize' in the MirrorSync.xml configuration file. This setting will only apply to NEW sync databases, so changing it will have no effect until you reset the sync data.
Although MirrorSync can use serial numbers or UUIDs for primary keys, we recommend UUIDs for very large databases. If anything becomes damaged in the internal sync database, it can be almost impossible to bring a very large databases back into sync with serial number primary keys.
Conflict merging can become slow when the databases are very large. If you do not need to merge field changes, then disable field-level merging in the MirrorSync configuration.
The bigger the database gets, the more speed benefit you will get by switching from spinning disk hard drives to SSD storage.
Deletion scanning in very large tables can take a long time to execute. In these tables, you can speed up your sync significantly by never allowing record deletion and just flagging records as deleted.
How do I make all of my users log in with the same username and password to the hub database?
In the 'spoke' section of the MirrorSync configuration process, select the option to log in with 'Hub database credentials configured in previous screen'. This will use whatever username and password you use for configuring MirrorSync as the username and password for running the actual syncs, regardless of the username / password used to open the FileMaker Go/Pro database. If you select this option, users will not be prompted for a password when they run the sync. For FileMaker Server, you may filter the records as described in the 'How can I filter records on a per-user basis if everybody is logging in with the same FileMaker account?' section below. For non-FileMaker databases such as SQL Server, you can filter records as described in 'How Can I Filter Records Using SQL?' section below.
If you select the option to sync with the hub database credentials, then whatever username and password you enter during the MirrorSync configuration for the hub will also be used for users to sync with it. This is convenient, as users will not be prompted to enter a password, but it is potentially less secure, since all users will be treated as the same user by the hub databases. It also makes it possible for a malicious user who is able to open the FileMaker client database to sync with the server, without knowing any passwords, so be sure to put a strong password on the client file to prevent this. The hub password is not embedded or stored in the client file, it is only stored on the MirrorSync server.
The default option, which is to use FileMaker client credentials, is a more secure option. The user will be prompted for their password when they first sync. The sync will then use their username and password for all requests to the hub database server. This then makes it easy to filter records on a per-user basis, and the user's account name will be used for any actions taken on the FileMaker Server.
How does licensing and pricing work for MirrorSync?
MirrorSync is completely free to use, with no limitations on features, for syncing FileMaker Server with a single copy of FileMaker Pro or FileMaker Go. You can optionally purchase additional server configurations or devices.
The supported configuration types are:
FileMaker Server with FileMaker clients (Go or Pro)
FileMaker Server with FileMaker Server
FileMaker Server with any SQL database (Oracle, MySQL, MS SQL Server, Salesforce, and Amazon RedShift are fully supported. You may also sync with any database with a JDBC driver).
SQL database with SQL database
SQL database with FileMaker clients (Go or Pro)
Device (Pro/Go Client) prices range from $95 for one device to $40 each for 100+ devices, depending on the total quantity including the devices about to be purchased, as well as the free device that comes with MirrorSync. The pricing tiers are at 1, 5, 10, 40, 60, and 100.
Please note that the devices on a license are not linked to a single solution on your license, and can be distributed amongst all license solutions.
Go to our online store to see all pricing details.
Is MirrorSync included in the 360Works Portfolio Bundle?
Yes. MirrorSync is free for one device, but Portfolio Bundle subscribers will receive an additional 10 device pack (for a total of 11 devices). Server-to-server configurations are not included in the Portfolio License, and must be purchased separately. There is also a special licensing program for Claris Partners; contact us at support@360works.com for details.
Is there a vertical market license type for MirrorSync?
No, but we'll be happy to explain how the existing licensing pricing works for vertical market solutions. Contact 360Works support for details.
MirrorSync has a REST API available to do almost anything that can be done from the MirrorSync Config Client. This is not fully documented yet; contact 360Works Support for specific use case questions. Common operations will be documented here based on customer requests.
POST http://yourserver/MirrorSync/api/v6/login?version=6.900
The version attribute is required; if you omit it you will receive an error message on future requests. It should be equal to or higher than the version of MirrorSync you have installed. Include BASIC authentication credentials for username and password, using the same credentials you would use with the config client. This will return a JSESSIONID cookie. Pass this same cookie in for future requests.
GET http://yourserver/MirrorSync/api/v6/getConfigurations
This gets a list of all MirrorSync configurations on the server. This includes all the low-level details of each configuration, as if using the 'export' button on the config client. Pay special attention to the 'id' attribute, as it is used for other API calls. The 'name' attribute contains whatever name you assigned to the configuration. Note - this can return a very large amount of data, if you have very complex configurations, or many configurations.
POST http://yourserver/MirrorSync/api/v6/doSync { "configId": "dc6c8826-7a9a-429a-946e-637898109e8d", "initialSyncMode": null, "initialSyncParam": null }
configId is the 'id' attribute from the configuration. initialSyncMode is null for incremental syncs. initialSyncParam is treated as the $$MIRRORSYNC_USERTOKEN variable.
POST http://yourserver/MirrorSync/api/v6/setSchedule { "configId": "dc6c8826-7a9a-429a-946e-637898109e8d", "frequency": 60000 }
configId is the 'id' attribute from the configurations that you fetch with getConfigurations. frequency is the desired auto-sync frequency, in milliseconds.
Is MirrorSync localized into any non-English languages? Does it work in other languages?
At this time, MirrorSync localized to English, Japanese, and French. If you would like to offer help with translation, please contact support@360works.com to let us know. You should still be able to add MirrorSync to your solution in other languages. See the previous section on customizing dialogs for tips on localizing your solution.
If your sync was previously working, and the databases are now out of sync or unable to sync, odds are decent that resetting the sync data will solve the issue. Ideally, this should be accompanied by throwing away all spoke database copies, re-distributing new spoke copies, and re-running the initial sync for each spoke.
MirrorSync keeps an internal SQL database of all primary keys, modification timestamps, and internal record IDs for every device being synced. Sometimes this internal database can get out of date, especially if primary keys are modified in the database. If you suspect that this is the problem, you can reset this internal data, which essentially starts MirrorSync over again as if it had never synced anything. Read the reset SyncData section for details.
If the MirrorSync installer hangs during installation, please force quit the installer and then follow the instructions below to recover. This typically occurs when you've had a previous installation go bad for whatever reason and the 360Works service is in a bad state.
Mac
Launch the Terminal and run the following commands:
If you're running into trouble with the installer, please try doing the following:
Run as Administrator
Temporarily disable firewall
Temporarily disable anti-virus
The ProscTomcatDaemon.jar file could not be started.
Error Message: Could not connect to 360Works Tomcat Daemon: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.io.EOFException
If you receive this message during installation, and you have Kerio VPN client installed, you will either need to remove Kerio in order to install MirrorSync, or you will need to install MirrorSync manually. The Kerio VPN client, by default, uses the same port that we use to communicate with the ProscTomcatDaemon.
I'm not seeing my databases from the Choose database button. What's happening?
Make sure the fmrest extended privilege is enabled for the account you are using in MirrorSync. Also be sure the database is accessible from the machine you are installing MirrorSync on.
I See a Filtered List of Databases, or No Databases
By default, FileMaker Server has the 'Filter databases in Client Application' setting enabled; this will filter out databases that FileMaker deems the user doesn't have access to; this can limit the databases that the user can select from the configuration wizard. Make sure to use credentials in MirrorSync that allow access to the database you're using.
When a sync does not complete successfully, the next sync with this device will run in recovery mode. Recovery mode does extra checks and runs more slowly, to ensure that data is not lost.
What is different in recovery mode?
Insertions are first tried as an update, then switched to an insert only if the update fails. This takes longer, but prevents duplication of records caused by inserting the same record twice.
When a record has changed since the last sync, MirrorSync normally skips syncing that record if the contents match the last known value. In recovery mode, all recently modified records are synced, even if their contents appear to match.
Deletions on device A would normally also be deleted on device B. In recovery mode, the record from device B will be used to re-insert the record on device A. If the last sync failed midway, it's impossible for MirrorSync to tell whether a record that's only present in one of the databases was inserted on one database or deleted on the other. If this causes a major problem, flag records as deleted, instead of actually deleting them. Usually you'd do this by creating a field in each table called "deleted" and setting it to true when you'd like to treat those records as deleted.
Some typical situations that might cause this if they occur during an ongoing sync:
A network error that severs the connection between MirrorSync and the hub/spoke (lost wi-fi connection, network outage, etc.)
An unexpected shutdown or stoppage of one of the resources that are critical to the sync (i.e. a user kills the syncing script or exits the db, the server hosting the hub or MirrorSync crashes, the device the spoke db is on shuts down unexpectedly, etc.)
To minimize the frequency of these issues:
Make sure your users have stable, strong internet connections while syncing so the risk of a disconnection is small.
Train your users not to interrupt a sync if possible. If they have to, they should use the "Abort Sync" button instead of exiting FileMaker, stopping the script, or shutting down their device.
Sync frequently. This will minimize the impact of recovery mode when it's triggered, as MirrorSync will only have to consider the changes since the last sync.
Ensure that the server(s) and services hosting your hub database and MirrorSync instance are very stable to prevent server failures from contributing to this.
Getting an error 502 during sync or configuration
This is caused by a configuration in IIS (on Windows) or Apache (on Mac OS X) that times out after when waiting for a request. Sometimes, very complex configurations or operations may take longer than this timeout period to complete, causing the web server to send a 502 error and stop the request.
Testing Timeout
You can test whether the timeout setting is the issue by triggering this URL (replace your server address in the URL), which should wait 3 minutes and then return a successful test result. If you get an error instead, then you definitely need to change the timeout settings as detailed in the rest of this section:
To increase this timeout on the Apache webserver that's being used by your FileMaker Server, open up this file in a text editor of your choosing: /Library/FileMaker Server/HTTPServer/conf/extra/httpd-proxy-fmiwebd.conf
timeout=600 denotes a 10-minute timeout. Adjust as necessary, where 1200 is 20 minutes, 3600 is an hour, etc.
Now restart FileMaker's built in Apache webserver using the following command in Terminal:
fmsadmin restart httpserver
If for some reason you get your timeout up to 7200 (2 hours) and still fail in your testing, subsequent timeout updates need to be changed in an additional file.
Keep in mind the /MirrorSync portion is dependant upon your installation method. If you installed using a hosting provider install, this portion will be whatever name you set during the installation.
Update the timeout=7200 designation to whatever you're updating your mod_proxy.conf to.
Finally, restart the httpserver as your would in the normal process.
If you have field-level access privileges that prevent the spoke user from modifying certain fields, you will get this error when records are modified on the server. The solution is to set the MirrorSync script to run with full access on the client and server, as described in 'Does MirrorSync require users to sync with full access accounts?`'.
Record could not be written because of an error 507: Value in field failed calculation test of validation entry option
This is an error coming from FileMaker. This means one of the fields you are trying to sync does not pass a field level validation by calculation, which will cause the modifications in the set to revert and not get updated. MirrorSync cannot get around field level validation.
This can happen for several reasons, but the most common is a timeout happening in the web server. Try re-running the MirrorSync installer (you don't need to uninstall first), which configures the web server timeout.
This means that MirrorSync itself ran out of memory (not necessarily the machine it's running on). Sometimes you might see the message “Java Heap Space.” This can happen when syncing very large batches of records. It may also appear as “GC overhead limit exceeded.”
If you get this error during an incremental sync (not the initial sync), and you are not expecting a large number of changes, check your MirrorSync customization script. Make sure you are not doing a Show All Records or Perform Find that expands the found set. Use Constrain Found Set instead, to reduce the size of the result set.
MirrorSync runs inside Tomcat, which in our installations has a default memory allocation of 1,024 MB of RAM. You can increase this by editing the setenv file.
File locations:
The location of the setenv file depends on your MirrorSync version. MirrorSync 7.1 and later run on Tomcat 11; earlier versions run on Tomcat 9.
What to change:
Update or add the CATALINA_OPTS setting to include your desired heap size (e.g. 4096M for 4 GB).
If a CATALINA_OPTS line already exists in this file, it probably contains the -Xmx parameter. If it does, just modify the number to your desired allocation, for example -Xmx4096M.
If the -Xmx parameter does not exist anywhere in the file, you can add it with a new line. See below:
macOS/Linux (setenv.sh):
CATALINA_OPTS="$CATALINA_OPTS -Xmx4096M"
Windows (setenv.bat):
set CATALINA_OPTS=%CATALINA_OPTS% -Xmx4096M
Important:
The new memory settings will not take effect until Tomcat is restarted. You can restart Tomcat from the 360Works Admin, following these instructions.
Reading/Writing Incorrect Timestamp Values from MySQL
MirrorSync uses the timezone configured in the MySQL server. Most of the time, this is fine. However, some applications (such as Magento) may ignore the MySQL configured time zone and instead default to GMT/UTC timezones when writing time values. In that case, MirrorSync may not detect changes made in MySQL and/or it may write incorrect time values to other databases. To fix this, modify the MirrorSync.xml configuration file. Find this entry and change the value to "GMT".
This is an issue that we have observed on OS X, where you may get the beachball and the configuration client can lock up / freeze / hang when entering text into a dialog. So far, we have been able to trace all these occurrences to one of the following 3rd party extensions:
Rescue Time
BetterSnapTool
BetterTouchTool
Caffeine
QuicKeys
If you encounter this issue, check to see if you have any of these applications running, either by looking at the top-right menubar corner or using the Activity Monitor application. Quit them if they are running. If you are experiencing this problem and you do not have one of these applications running, go to your Users & Groups system preference, click the 'Login items' tab, and then use Activity Monitor to quit out of all applications listed as a login item. If that still does not solve the problem, please contact us at support@360works.com and we will help troubleshoot the problem.
Configuring MirrorSync creates gaps in serial numbers
This is known and expected behavior. When MirrorSync auto-detects primary keys, modification timestamps, and creation timestamps, it does so by creating a new test record, checking to see which fields are writeable and which ones have auto-enter options on them, and then deletes that test record. This will result in the next number in the serial number sequence to be used up.
Re-run the MirrorSync installer; as part of that process you can reset the password to whatever you want. Running the installer will not lose or overwrite any configurations.
I'm getting a MySQL error that starts with 'Packet for query is too large'
MySQL has a default limit of 4 megabytes for BLOBs, which are basically MySQL's version of a FileMaker container field. You can increase this limit by adding these lines (or something similar, read MySQL documentation for full details) to your MySQL configuration file:
Error from server: user lacks privilege or object not found: SYNCCLIENT in statement [INSERT INTO SyncClient(timestamp, description, device_id, clone) VALUES (?,?,?,?)]
Restarting the MirrorSync application through the 360Work Admin.jar file will resolve this issue.
There is a duplicate ID in node <spoke or hub> in <layout Name> on <device name>. Duplicate ID: <primary key of duplicate>
This error message indicates that there are one or more duplicated primary key values in the specified device and table. For MirrorSync to function, all primary key values must be unique to that table. Duplicates will need to be removed before this device can complete a sync. To find duplicate primary keys in a FileMaker table, use find mode to search the primary key field of that table for ! .
To prevent duplicates in the future, we highly recommend that you:
On your auto-enter calculation (if you use one), UNCHECK the box that says 'Do not replace existing value of field' (if any)
Change the validation on the primary key in the hub file in the specified table to "always" instead of "only during data entry".
Check the "Unique value" box
Check the "Not empty" box (this won't help with duplicates, but it's good to do it anyway to prevent empty PKs)
After doing so, we also highly recommend that you distribute new spoke files.
Example Field Options Dialog
For one-way syncs from spoke to hub, the spoke database cannot start off with records that match existing records on the hub
Imagine that you have the same contact record with the same primary key present on two different spoke databases, both syncing one-way to the hub. Now imagine that device 1 changes the record. Because the sync is one-way, MirrorSync is unable to write this data to device 2, meaning the user of that device will have the wrong data. Furthermore, every time device 2 changes any field on the record, this will cause a conflict that cannot be merged, because device 1 and 2 have no knowledge of each other, and MirrorSync can't write their changes to each other.
To prevent this scenario, MirrorSync does not allow you to have any records present on the spoke device if 1) that table is one-way from spoke to hub, and 2) any record on the spoke also exists on the hub (indicating that the same record could very well exist on other spokes).
If you get this error, you can solve it by either:
Empty out that spoke table during the initial sync, so it contains no records
Switch that table to a bidirectional sync
One or more of the spoke databases have records in them. You must select a mode for the initial sync operation.
This is normal when a server-to-server sync has previously run, but the sync data has been reset. If you want the spoke database to be wiped and replaced with the hub database contents, pick the option to 'Delete spoke'. If you're using UUID primary keys, and you want to keep the data in the spoke database, pick the 'Rebuild sync data' option. For rebuilding, MirrorSync scans all the primary keys in both databases to merge them together.
MirrorSync isn't syncing some or all modifications/updates, but is successfully syncing insertions and deletions
This is almost always a problem with the way timestamps are getting written and read. To boil it down to a single rule:
The modification timestamps that MirrorSync is configured to use must always be set to the server time of the machine the database is on.
This is because there's no other metadata that "tells" MirrorSync what time offset to use for a table's timestamps. Using server time as that metadata makes the most sense, so that's what MirrorSync always does.
Situations that could cause this:
Users directly modifying a synced FileMaker Server database from some time zone that's west of (chronologically before) the server's time zone (You can find an explanation and solution to this here). This is by far the most common situation that causes this.
The developer setting the timestamp for UTC time. This is more common in SQL setups, but it's still possible to make this mistake in FileMaker. If you're syncing a MySQL server and seeing this problem, refer to this documentation.
Misconfigured timestamps. Possibly: incorrect format, incorrect auto-enter options, timestamps that are configured to only update in certain situations, etc. This is uncommon.
The first option accounts for 99% of these problems. Try implementing host modification timestamps in a single table using this documentation, and test to see if this resolves it for that table; if it works then implement this solution across all tables.
MirrorSync is restoring deleted records on an incremental sync
Sometimes a user may report that they deleted a record and that during their next sync the record was restored to their database by MirrorSync. There are three ways this can happen:
If one user edits a record, and then another deletes the record, the edit will win conflict resolution, causing the deletion to be re-inserted to the database of the user that deleted it. If just a few records are restored intermittently and it's not happening to just a single user, this is likely what's happening. This is not a bug, it's intentional behavior to prevent new data from being deleted. If someone cared enough to edit the record it seems likely that the deletion was unintentional.
There are two different databases using the same internal identifier while syncing. Usually this is caused by a user copying another user's database and attempting to use that one concurrently with the original. If this is the case, you'll probably see deletion reversions occurring regularly with two different users; the fix is to supply one of those users with a fresh spoke database to sync from.
The previous sync from this spoke device got unexpectedly interrupted, was unable to reconnect itself and continue, and had no chance to put the spoke, hub, or itself in a normal state to prepare for the next sync. If this happens the next sync with that spoke database will restore all records that were deleted since the last successful sync. See the following section, recovery mode.
MirrorSync is restoring deleted records while initially syncing a full copy spoke
Sometimes while a new spoke database is initially syncing, hub records that were very recently deleted are restored to the hub. This can occur due to an interaction between two little-known features of MirrorSync interacting in an unintuitive way. The features:
During an initial sync MirrorSync must consider any inserted/modified records in the last 25 hours before the time the user downloaded the database to be modifications by the spoke.
When a record is deleted in one database and modified in another, MirrorSync will restore the deleted record.
See below for details about why these things occur. This means that during an initial sync, MirrorSync will restore any records that were recently modified and deleted on the hub after the spoke database was downloaded.
Explanation:
To understand why, consider this situation: A hub FMS hosting server is in Auckland, New Zealand, and a user with a spoke copy of that hub database is in Honolulu, Hawaii. The New Zealand server's timestamps will be 23 hours ahead of the Honolulu user's modification timestamps.
This means that any modifications that the user makes to the database in the first 23 hours after their download will have modification timestamps that are "in the past" when looked at from the perspective of the hub. If MirrorSync doesn't account for this issue, these modifications will never get synced. To deal with this, MirrorSync will consider all modification timestamps from up to 25 hours before the download to be modifications that were made on the spoke copy.
If the record in question hasn't changed on the hub, this is not a problem. MirrorSync will simply see that the two records are the same, won't make any changes, and the sync will proceed as normal.
However, if the record that's considered to be modified was deleted on the hub database after the user downloaded the database, but before the 25-hour limit, something else will occur: the deletion will be reverted and the record will be reinserted to the hub.
This is because while MirrorSync is evaluating record conflicts (situations where two changes to a record do not agree with one another), it always prefers to preserve information. In a situation where one database has deleted a record and another has modified it. MirrorSync assumes that there was a good reason to modify it, that it's possible that the deletion was unintentional or problematic, and will restore that record, even when the deletion was on the hub.
In most situations this is positive, however in this particular situation the modification coming from the spoke may not actually be a new modification, but MirrorSync must assume that it is for the reasons listed above.
Possible Workarounds
Advise your users to always immediately sync the spoke after it gets downloaded, so there's very little opportunity for relevant deletions to occur before their initial sync completes
In the same vein as above, setting the sync to be a server-side initial sync will likely eliminate almost all the occurrences of this issue
Instead of actually deleting records, simply flag records as deleted
Certain FMS folder settings will keep the FileMaker backup process from including external container data with your backups, which means that your external container data will not be bundled with your MirrorSync client database download (or will not be pushed to the spoke server, in the case of a server-to-server sync).
Review the following screenshot. If you have both the Additional Database Folder (1) and Container Data Folder (2) settings enabled, you'll see the "Back Up Container Data Folder" (3) option. If this is disabled, the external container data will not be included in backup processes. MirrorSync uses the FMS backup feature, so this means that the container data of databases in that additional folder will not be bundled with your MirrorSync spoke/client database download. For MirrorSync to deliver your external container data in your download, you must either enable this setting. The only workaround would be to manually retrieve the container data from the server and put it in place with your client database.
If you're configuring MirrorSync to work with a very large and complex database, you may receive an error because a MirrorSync request to the FM Data API timed out, returning an error response instead of what MirrorSync was requesting. You can increase that timeout and relevant MirrorSync timeouts by doing two things:
Increase the timeout of the Data API itself using the instructions on this page.
Install the latest version of MirrorSync. We doubled MirrorSync's internal timeout for data API requests to 12 minutes in 7.11+. If you're on the latest version, and you're still running into the 12-minute timeout, email us at support@360works.com to request a custom build of MirrorSync.
The error Failed getting modification counts from client: Client has sent an abort message or been flagged as unresponsive is likely to be fixed by this as well, but in this case you also need to apply this change:
Find the getModificationsTimeout_xxx parameter in your MirrorSync.xml file.
Uncomment it so the line looks like <Parameter name="getModificationsTimeout_xxx" value="480" />
Change the xxx in the parameter name to the name of the sync layout (not table name) you wish to increase the timeout for. This is case-sensitive.
Change the value from the default to whatever you want this timeout to be, in seconds (e.g. "720").
Duplicate the full parameter line for every layout you want to extend this timeout for, and change the name after the underscore for each.
Similar to the above, the error Request to fetch metadata for <tableName> exceeded the timeout value of <timeoutSeconds> seconds... often requires extending the FM Data API timeout as well; however extending MirrorSync's internal timeout for this also requires these steps:
Find the getTableDetailsTimeout parameter in your MirrorSync.xml file. If it doesn't exist yet, copy and paste this into the file: <Parameter name="getTableDetailsTimeout" value="240" />
Change the value from the default to whatever you want this timeout to be, in seconds (e.g. "720").
After you make these changes, if your solution is still exceeding the 12-minute internal timeout MirrorSync has for this type of request, please contact 360Works support for assistance.
MirrorSync executes all its client-side actions using "commands", which are Data API calls or responses to Insert from URLs that contain script parameters that will control what MirrorSync will do.
Sometimes, a MirrorSync problem is completely script-side, meaning we can't see what happened in the server log unless that specific value got returned to the server. In these cases, a 360Works support employee may ask you to debug the script and let them know the result.
To do this requires some one-time setup. Unfortunately, this can't be done in FM Go; if you need to debug an FM Go database's script you'll need to copy that database to a workstation where you can use FM Pro.
First, open the file that the error is occurring on using the account that MirrorSync uses to sync. Usually this is a full access account, but in FM Pro/Go client syncs it's common to run even the hub script with the client user's credentials, which may not be full access. The 360Works employee directing you should let you know which database to debug.
Then, go to the MirrorSync layout, and click on the "Debugging" tab, and open layout mode:
Double-click the "Go" button in the bottom right, and choose the MirrorSync 6 worker script:
Edit the script parameter, and change $$MIRRORSYNC_SCRIPTNAME = "<Put your MirrorSync script name here, e.g. 'MirrorSync 6 (My configuration)'>" to:
$$MIRRORSYNC_SCRIPTNAME = "MirrorSync 6 (<configName>)". Check out this example, where my main script's name is MirrorSync 6 (basicClientSync)
Now, MirrorSync is set up for simulation mode, we just need to give it a command.
The input next to the go button, labeled "Send command", accepts a MirrorSync command; these are always in the form of a FileMaker calc Let statement. For example, this command will perform the "tableCatalog" action on your database: Let([$command="tableCatalog";$sessionId="anythingCanGoHere";$layoutFields="";$colDelim="~}zRA,Yn~";$rowDelim="|f!8&rW1|";$$hub="1";$$sequence=1];True)
It should be noted that some MirrorSync commands are not purely read operations. The one above is, but some commands can modify your database; don't proceed without the advice of a 360Works employee.
After putting the command in that box, you can optionally open the script debugger and click the "Go" button to run the MirrorSync script with that command. In simulation mode, most scripts will display their output in a custom dialog. Depending on the nature of the issue, the 360Works employee you're working with may be interested in that output, or they may be interested in debugging a specific part of the script; they'll let you know.
When working with 360Works support, you may be asked to locate a specific MirrorSync record on either a hub or spoke database. Follow the steps below to identify the correct record:
Open the database and navigate to the MirrorSync layout:
Click the Internal tab and look at the Type field:
Check the Type field. It will be one of the following:
Hub
Client
Server
Cycle through the records until you find the one requested by the support team.
Don't delete or modify these records unless you're explicitly told to by the support team.
