Skip duplicate entry mysql replication

August 22, 2018 MySQL-MariaDB

In a Master/Slave replication data consistency is a biggest concern and is very easy  to break your replication with  issues such as duplicate keys, missing rows , accidental writes from application connected to the slave or even by a DBA connected to the slave 🙁 “yes things happen” .

1. read_only

When the read_only system variable is enabled, the server permits no client updates except from users who have theSUPER privilege. This variable is disabled by default.

It’s a good practice then to set read_only=1 on slave servers to prevent any (accidental) writes.

Unfortunately users with the SUPER privilege can override the setting and could still run DML queries.

  2. super_read_only

However, since Percona Server 5.6.21 and MySQL 5.7.8, you can use the super_read_only feature to extend the read_only  option and apply it to users with SUPER privileges. If the   super_read_only=1 the server prohibits client updates even from users who have SUPER.

3. temporary use of slave-skip-errors

With slave-skip-errors you can make the replication slave ignore certain error codes (you can find the list of MySQL error codes here: Server Error Codes and Messages)

Example : the purpose of the example below is to temporary skip error

• 1062 : Duplicate entry ‘%s’ for key %d
• 1054 : Unknown column ‘%s’ in ‘%s’

Caution : Do not use this option unless you fully understand why you are getting errors. If there are no bugs in your replication setup and client programs, and no bugs in MySQL itself, an error that stops replication should never occur. Indiscriminate use of this option results in slaves becoming hopelessly out of sync with the master, with you having no idea why this has occurred.

Here are a couple bugs to consider on Percona Server 5.6:

• set –read_only on when –super_read_only is used as a command line option
  https://bugs.launchpad.net/percona-server/+bug/1389935
  Fixed on Percona Server 5.6.26-74.0
• MySQL aborts when you enable super_read_only with read_only
  https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1483956
  Fixed on Percona Server 5.6.26-74.0
• super_read_only mode breaks replication with some queries
  https://bugs.launchpad.net/percona-server/+bug/1441259
  Fixed on Percona Server 5.6.28-76.1

For more information, please check following links:

• https://www.percona.com/doc/percona-server/5.6/management/super_read_only.html
• https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_super_read_only

Normally MySQL replication will stop whenever there is an error running a query on the slave. This happens in order for us to be able to identify the problem and fix it, and keep the data consistent with the mater that has sent the query. You can skip such errors, even if this is not recommended, as long as you know really well what are those queries and why they are failing, etc.

For example you can skip just one query that is hanging the slave using:

mysql>SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; START SLAVE;

“1062 | Error ‘Duplicate entry ‘xyz’ for key 1′ on query. Default database: ‘db’. Query: ‘INSERT INTO …”There might be cases where you will want to skip more queries. For example you might want to skip all duplicate errors you might be getting (output from show slave status;):

If you are sure that skipping those errors will not bring your slave inconsistent and you want to skip them ALL, you would add to your my.cnf:

slave-skip-errors = 1062

As shown above in my example 1062 is the error you would want to skip, and from here we have: _ Error: 1062 SQLSTATE: 23000 (ER_DUP_ENTRY) Message: Duplicate entry ‘%s’ for key %d_

You can skip also other type of errors, but again don’t do this unless you understand very well what those queries are and what impact they have on your data:

slave-skip-errors=[err_code1,err_code2,...|all]

and for the error codes you can see them all here … (you will see them in your _show slave statu_s; also).